Exchange 2010 journaling performace

[Rob_MN]

Hi – new to EV and to this community. I've recently hired a contactor to upgarde my EV solution to 10.0.3 & Exchange 2010 SP3. When running on Exchange 2007 it smokin fast, but when I change the journal mailboxes to new ones residing on E2010, the ingestion rate is extremely slow.

All systems are new hardware running on Windows 2008 R2, 32 procs, 32GB RAM. The EV service account is on E2010 & the throttling policy is applied. Each of the 4 journal mailboxes is running on its own non-dag Exchange server with CAS role installed. Indexing is disabled on the DB & the DB is pointed to use the local CAS instance.

I've adjusted the journal task to use 20 concurrent connections to Exchange & 10K messages per pass.There are no errors or obvious concerns that my contractor is seeing.

Any recommendations on additonal settings or things to check?

Thanks!

Rob

[Ben_Watts]

Hi Rob,

First thing I would do would be to drop those figures back down to default again rather than 20 concurrent connections and 10k messages per pass as that isn't very likely to help the situation to be honest.

When you say 'slow' how are you gauging the performance (what is happening in the Mailbox)?

If you can give us some more information it would be easier to suggest a possible cause or resolution.

[Ben_Watts]

Not that you should need to as Envelope Journaling was in 2007 as well as 2010 but the JournalDelay regkey can help in situations such as this  -  http://www.symantec.com/docs/HOWTO77164

This is why it can slow archiving down, but like I said this was introduced in Exchange 2007 so not sure this will apply to your situation  - http://www.symantec.com/docs/TECH61066
 

[GabeV]

Since you hired a contractor, he/she should have optimized the new servers. Take a look at this technote and make sure that all the recommended settings were applied correctly to the EV and SQL servers:

Recommended steps to optimize performance on Enterprise Vault (EV), Compliance Accelerator (CA), Discovery Accelerator (DA), and SQL Servers in an EV environment
http://www.symantec.com/docs/TECH56172

Do you have the archiving rates before and after the upgrade?

[Rob_MN]

Ben & GabeV - thanks for the suggestions. I've reviewed these settings and they are not all set as recommended, but the solution is running like a champ using E2K7 journal mailboxes.

Is there any uber-sensitivity for E2010 (e.g. indexing to CIFS location) that would create an extreme bottleneck for ingestion when the ONLY system change is the platform on which the journal mailbox resides?

[PeterWendell]

Never mind.

[GertjanA]

Hello Rob, If you monitor tje Journal mailboxes themselves, are they having many items? Large items? Do you perhaps have forefront scanning on that Exchange server? Can you, or the contractor, reverify all settings have been done properly? Regards. Gertjan

[Rob_MN]

My contractor is recommending that I install Exchange 2007 onto the EV servers and stop trying to make it work with E2010.

I've flipped a few of the Exchange databases to journal to E2010 mailboxes. What I'm seeing is that it takes EV a LONG time to process through the "Message Class: Message" queue & then I don't see the J2 queue grow over 15 messages as it process the items. What I believe is happeing is that that the "Message Class: Message(Enterprise Vault PendingArchive Part)" queue doesn't cycle through all the messages whose modified time is over 5 minutes - thus resulting in a huge backlog of messages in the mailbox. It almost seems that the EV throttling policy isn't effective, but yet I have confirmed that it is applied & that it has the correct settings.

It might be time for a support case...

[JesusWept3]

installing Exchange 2007 on the server is an extremely bad idea
Do you have Exchange 2010 and Exchange 2007 Journaling tasks using the same server?
What version of Outlook have you got installed on the server?

Also is the vault store located on the same EV server?
Or you do have the Archive on EVServer1 and the task on EVServer2?

[Rob_MN]

I do have both E2K7 and E2010 journaling tasks on the same EV serverd with Outlook 2007 SP3. I have 4 EV servers with this configuration. So when I'm making the change to E2010, the only thing that I"m doing is changing the journal mailbox that the Exchange DB is pointed to (change is made in the Exchange console). All other variables stay the same & I restart all of the EV services after making the change.

I have 4 Exchange 2010 servers with only 1 non-dag DB that holds only the EV system mailbox and 2 journal mailboxes (1 active & 1 passive).

The vault stores, archives, MSMQ location, indexes, and Exchange DBs are all remote running on high speed NetApps storage. The EV servers are only doing the processing.

[Rob_MN]

I have also observed that restarting the EV services on Server A seems to be affecting the processing/ingestion capabilites of Server B. Is this expected behaviour or should each EV server process independently of the others?

[JesusWept3]

Hmmm, when that happens, can you have a look at things like memory usage and such on the processes before you restart?

I remember working with AndrewB and he discovered that the MonitoringMiddleTier had a slow leak and journaling was performing painfully slow, but when restarting the Admin Service, it killed and restarted the monitoringmiddletier and suddenly everything was faster again

[Rob_MN]

I'm keeping an eye on the MonitoringMiddleTier and the CPU is zero and the memory is 11-12MB - doesn't really seem to be an issue.

I opened a case with our VAR on Friday (required before being escalated to Symantec). They had me re-crate the journal task on Friday afternoon, but no luck. The current queues in my two journal mailboxes on E2010 are 92K and 130K while the two on E2K7 are both around 3200. 99% of the messages are in the (EnterpriseVault PendingArchive Part) queue - so EV has started to process them. Based on my limited knowledge of this product, the root cause seems to be with either EV or MSMQ.

Is there any performance tweaking that can be done for MSMQ or is it pretty much all defaults? Whem messages are moved to the J2 queue, is that an SMTP transfer or a file copy?

[JesusWept3]

So an item goes from IPM.Note to IPM.Note.EnterpriseVault.PendingArchive.Part item
After 5 minute elapses and it no longer needs to wait for a journal report, it changes it to a Pending message class and posts to J2

The Journal task then connects and takes the item via MAPI and places the actual message in to Storage Archive in 1.5MB chunks (so a 9MB message, you'd expect to to see Six seperate messages in Storage Archive queue)

Then the EV StorageArchive.exe would pick up that item from Storage Archive, and then create the DVS files and check to see if the item has any shareable parts and whether its already shared, and adds the relevant details to the database.... after storage has been succesful, it then passes a pointer to the item in to the J1 queue, and then the Journaling Task picks that up and finds the message in the mailbox and performs a hard delete against the item.

So if you see items just stuck waiting in the .PART section, then it wouldn't even have reached MSMQ at this point, unless it physically cannot post to the MSMQ, usually you see that with the MSMQ 1GB quota being breached, and upping to 8GB resolves it (8GB used to the be the default, but it was bought down by MS to 1GB)

Sometimes people enable Journaling on MSMQ, meaning every copy of a message that goes in to an MSMQ has a copy made, and that counts towards quota, its useful for troubleshooting sometimes, but people forget to disable it, and then it fills up the quota and nothing more can be done.

There are also throttling mechanisms that EV uses, for instance if the Storage Archive queue gets really backed up, then it won't post until it clears down, in order to avoid making whatever situation worse.

Major slow downs like this though, i've sometimes seen by different filters.
So i know of one company that uses Orchestria as a filter, and for whatever reason, DL Expansion takes 30 seconds, so EV just sits waiting for a response from Orchestria to perform its custom tagging and what not before it can move on to the next item.

Compliance Accelerator also installs a custom filter for journaling as well for random sampling. and you may have your own selective journaling or custom filter set in the registry for different purposes (maybe deleting items from certain users, or adding custom index properties based on key words etc)

[Rob_MN · Accepted solution]

I think I may have fixed it by adjusting the Exchange server power management setting to "maximum performaince" (previously set as Balanced). My two E2010 servers caught up from a 30K/50K mailbox queue to running as just as the E2K7 servers.

Thanks to all for your suggestions & I'll update this post if I end up needing additional changes.

Rob

[JesusWept3]

Interesting, the Balanced should have just been about shutting down disks after 15 mins idle and such and shouldn't have had any impact to the best of my knowledge, but can't argue with the increase in performance!

[Rob_Wilcox1]

Glad you have it sorted out.