EV MSMQ folder growing beyond explanation.

[xgtdec]

Hi Folks

I just wanted to share a problem we have with EV at the moment that somebody may be able to help out with, or perhaps at least offer some insight to.

I arrived to work last Thursday to complaints of archiving being unavailable, nothing new there..im used to EV flaking out on me!!

We have 4 EV servers in total and evserver01 and evserver04 take care of the email, they have 2 exchange tasks each.

So after troubleshooting what happened was that the MSMQ folder had filled and evserver04 had fallen over. 

I followed this technote http://www.symantec.com/docs/TECH48588 as the services were dead and although the msmq queues had items in them..everything was unresponsive.

So Thursday evening we were back functional with an empty msmq folder(these are on a SAN drive) I also up'd the limit from 8 gig to 16gig.

On Friday morning the evserver04 seemed ok...but the exact same issue had hit evserver01, and i had to run through the same fix to restore functionality.

So as it stands the msmqs are empty after the nightly archiving run....but the msmq folder size are

evserver01: 5.65gig
evserver04: 10.4gig

So after a weekend evserver04 has gone past the original limit of 8gig!!

Some troubleshooting lead me to the msmq folder and the possibility of AV scans. Now this only cropped last Thursday so im just presuming the exclusions were in place, and of course security guys arent being helpful but the tell me that the F\:MSMQ locations are excluded.

So basically im keeping the show on the road by increase the space available to the MSMQ folder because it seems to grow way beyond what i think it should do, and as a horrilbe aftershock i now have a lot of users with thousands of pending items and an inability to archive manually(case logged for this)

Has anyone any ideas or advice on where to start??

Declan

[TonySterling]

So could you tell what folders in the MSMQ directory had items?  It sounds as though some enabled the MSMQ Journaling feature.

What are the OS's of your EV Servers?

[GertjanA]

If necessary and possible, set your archiving tasks to report mode, and restart them. this should clear out storage queue for sure. tony's remark on journaling is also a good one. This will fill quickly!

[xgtdec]

Thanks Lads

The Os is win 2008 R2 across all 4 ev servers, where would i see the msmq journaling feature?

I didnt have the foresight to look in the directory!!!

The storage queues are actually empty...its the msmq directory that filling!

[TonySterling]

If you look on the properties of MSMQ there should be a check box.

To enable or disable Message Queuing target journaling

1. Click Start, point to Run, type compmgmt.msc, and press ENTER to display the Computer Management MMC console.

2. In the console tree, right-click the applicable queue.

   Where?

   • Computer Management/Services and Applications/Message Queuing/YourQueueFolder (Public Queues or Private Queues)/YourQueue
      

3. Click Properties.

4. On the General page, in Journal, select the Enabled check box. Or, to disable target journaling for the queue, clear the Enabled check box.

[xgtdec]

Thanks Tony

I wouldnt have expected to see them checked, i certainly would have done it and i can only presume that since those queues have been recreated then journaling is set to off by default.

So yesterday the figures were

evserver01: 5.65gig
evserver04: 10.4gig

and today the figures are

evserver01: 58Mb
evserver04: 82Mb

Ive been racking my brains trying to come up with what could cause such growth and then such decline, so far ive come up with this!

I rebooted EVSEVER01 and EVSERVER04 as i recreated queues and it was the only way to become operational again. Since i did that the helpdesk has received calls..not many but enough for me to wonder....where users can no longer archive manually. I got a list of about 10 users and they all share the same VSG, and that VSG is controlled by EVSERVER02. SO the task for these users resides on EVSERVER04, but they would need the message queues of EVSERVER02 available to manual archiving.

I checked the queues on EVSERVER02 and they were all empty....so before i did anything i rebooted evserver02...and lo and behold over 60000 items had appeared in the storage archive queue and started to process. I left it alone and all went through a full run tonight, and then this morning..all is good msmq directories are tiny and archiving fully funtional.

SO..my question now....is it a design flaw on our part to have a task on EVSERVER04 for mailboxes where the same users are also relient on EVSERVER02 for manual archiving??

[GertjanA]

ah...

This is probably what is called 'misalligned' archives.

In other words. The archive resides on server04, but the archivnig task for this user is on server02. On server 02, you probably see some outgoing queues to server 04.

You can run the below query to find the archives that have this. Run against Directory database.

--query to find mailboxes using a storage service on a different computer.
select A.ArchiveName, CE.ComputerName as 'Comp entry for task', CE2.ComputerName as 'Comp entry for storage' from ExchangeMailboxEntry EME
inner join exchangemailboxstore EMS on EME.mbxstoreidentity  = EMS.mbxstoreidentity
inner join exchangeserverentry ESE on ESE.ExchangeServerIdentity = EMS.ExchangeServerIdentity
inner join archivingretrievaltask ART on ART.ExchangeServerEntryid = ESE.ExchangeServerEntryid
inner join task T on T.TaskEntryID = ART.TaskEntryID
inner join computerentry CE on CE.ComputerEntryID = T.ComputerEntryID
inner join root R on R.VaultEntryID = EME.DefaultVaultID
inner join archive A on A.rootidentity = R.RootIdentity
inner join vaultstoreentry VSE on VSE.VaultStoreEntryID = A.VaultStoreEntryID
inner join storageserviceentry SSE on SSE.ServiceEntryID = VSE.StorageServiceEntryID
inner join computerentry CE2 on CE2.ComputerEntryID = SSE.ComputerEntryID
where CE.ComputerEntryID != CE2.ComputerEntryID

[xgtdec · Accepted solution]

Ok, i cant see us being able to change the config with any sort of ease, i think its better to just keep in mind that all servers should be checked in the event of users not being able to archive, the issue is sometimes related to the server hosting the VSG rather than the server hosting the mailbox task!!

Thanks for all the input guys...very much appreciated!!!