MSMQ behaviour

[Sortid]

Hi,  I'm using EV10 SP3, Exchange 2010.  We have 2 EV servers, one journaling one for users mailboxes, with separate SQL 2008 server.  Last night we enabled 100 people in EV, the first of our pilot groups.  We forced an archive run at 4pm which finished at 12:30am.  Over that period it archived 416,000 items.  Looking at the SAN utilisation, everything went along ok until 8pm-12am when for some reason MSMQ went crazy and spiked for 4 hours (see pic).  The other luns continued on at around 50-100 I/O throughout the period, but MSMQ went to 900.  What could cause this behaviour? 

(The pic shows MSMQ in red, index (blue), partition (green), and SQL database (dark blue) and Logs (magenta) volumes)

Thanks.

[Arjun_Shelke · Accepted solution]

Where have you placed the MSMQ storage location? Is it default location? We recommend moving the MSMQ location to different drive than the system drive.

I found following information in Performance Guide.

Archiving generates IOs on local disks. The primary causes of these are
 The creation of temporary files used when archiving and conversion
 IOs generated by MSMQ
 IOs to the Enterprise Vault cache location.
To isolate the IOs that MSMQ and Enterprise Vault cache cause, place the MSMQ files and the Enterprise Vault cache on fast local disks separate from the system disk and from each other. MSMQ is used during Exchange archiving and journaling but not for File System Archiving, Domino journaling and Domino mailbox archiving, PST migration, or SMTP archiving.
Blade servers generally have slow local disks that are not suitable for high IO loads.

[Arjun_Shelke]

Refer Symantec Enterprise Vault 10.0 Performance Guide
http://www.symantec.com/docs/DOC4553

[Sortid]

Sorry, should have said...The MSMQ drive is a standalone RAID 1 SAN lun.  The only thing on there is MSMQ.  Cache and system are located elsewhere.  I'm curious as to why it spiked 4 hours after the archive started running...It's not like MSMQ loads up with a heap of data and then does a flush after a period of time, it's use should fairly consistent across the archive period.

[EV-TSE]

That's not strictly true, MSMQ can as you put it "load up with a heap of data". 

It all depends on how your vault stores have been configured.  If you have elected to set the Safety Copy setting to "After Backup" then you would see a progressive increase in the amount of storage used by MSMQ because items are kept in there until post processing  has completed.

[JesusWept3]

That's not true at all

the item goes from A2 to storage archive, once it's been archived it goes to SQL in to watch file and scans for backup, after backup is detected it posts the item to A1 and then goes ahead and turns the item in to a shortcut

but you could be right that it was a backup that occurred and it posts 490k items to A1

what I would suggest is setup perfmon and add the EV msmsq's and find out exactly which queue is causing the io

[John_Santana]

What is the recommended MSMQ LUN config for the best practice with EVJournal option ?

[JesusWept3]

MSMQ recommendations are all the same regardless of whether its mailbox archiving, journal archiving or public folder archiving

[John_Santana]

Cool, thanks Jesus.

[Sortid]

Interesting. Thanks guys.  The backups don't run until after 1am.  I'll set up some monitoring I think and see if this behaviour repeats itself when we add more people to EV.