EV9 poor (journal) archiving performance

[goatboy]

Hi

Environment in EV9.0.2 and Exchange 2003.

We have ~2300 users being journaled to a single mailbox, this mailbox is targeted by journaling archiving in EV - single journaling task.

I ran the numbers over 3 months and we're getting an average of 6891 items being archived per hour by this task, obviously a way off the best practise numbers quoted in the EV9 performance guide (20k-60k). Servers are beefy, plenty of RAM, CPU, storage are (new) Centeras with collections enabled.

The mailbox targeted by the journal task always has a big backlog, we can't seem to archive quickly enough.

Questions:

1. Is it a good idea to up the "number of concurrent connections to exchange server" from 5 to 10 as per the performance guide? What about the "Maximum number of items per target per pass"  (currently set to 1000)

2. Is it beneficial to set up, say, an additional 4 journaling tasks, and split the 2300 users so they journal into 4 mailboxes instead of 1? Any downsides to doing this? I can't find any recommendation in the performance guide around number of journal tasks vs users.

3. Can you have separate journal tasks all archiving to the same archive, or are new archives needed for the new journal tasks?

Thanks

[JesusWept3]

concurrent connections isnt going to make much of a difference
neither will messages per pass really

You can have different tasks and targets going to the same archive

You can split up the mailboxes if you want, that should give you better performance
But honestly, something is desperately wrong here, the numbers are really bad

I know we had a similar situation where the throughput was worthless, and we found that the EV Monitoring Agent had a significant handle leak and was consuming a LOT of resources, we killed that and then journaling flew

So somethings you may want to check
Handles and memory counts of all processes on the machine
Have a look at disk queue lengths on the EV Server
Have a look at disk queue lengths on the exchange server
Have a look at disk queue lengths on the SQL Server

Also on the SQL Server, check and see if there are any processes being blocked or deadlocked, and that it isnt SQL causing the slow down.

And if you're on Exchange 2003, it may be worth using Outlook 2003 instead of 2007 as 2007 is needed for Exchange 2010, its recommended to use outlook 2003 for Exchange 2003 and 2007 due to the fact that MAPI is more performant on Outlook 2003

Also do a cursory check of Application and System Logs on EV/Exchange/SQL to make sure we're not getting any issues from hardware.

And its always a good idea to look an make sure that the NIC's and Network are in good shape, that you're not dropping packets or anything like that

Also you can do a dtrace of JournalTask and StorageArchive and see if there are any real slow deltas
 

[goatboy]

Already on Outlook 2003.

Monitoring agent is a big consumer of memory, and has ~15K handles.

Is it safe to kill? What is it and does it need to be disabled permanently?

[JesusWept3]

thats the monitoring agent, it writes all the statistics to the EV Monitoring Database that you need to view some of the SQL Reports or up to date information in the Enterprise Vault Operations Manager (EVOM)

Also what are your backlogs like in journaling at the moment?

[JesusWept3]

oh and end task it, EV will start it back up

[goatboy]

 238552 backlogged items in the journal mailbox. 

I ran the Dtrace, what should I look for? I am seeing these sorts of numbers, not sure what is (ab)normal:

800762 13:15:04.259  [4172] (JournalTask) <4160> EV:H CArchivingAgent::ProcessItemInternal Time Taken to Process Item - C026EF85FDA3FAC6F343BBF6D443F171: 1.200357 secs

844408 13:15:22.963  [5100] (StorageArchive) <496> EV:H SavesetReceive Time taken to store item - C03D7F08D0CD7C3E1F8FDCA4083829F1: 0.241217 secs

840189 13:15:22.556  [3736] (StorageArchive) <5052> EV:H SavesetReceive Time taken to store item - C03D788ECB1A023B41C8B119C42E79B1: 0.185905 secs

789694 13:15:02.947  [4172] (JournalTask) <5580> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter]: [0.010175] seconds

825483 13:15:08.072  [1200] (JournalTask) <8968> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [5.138734] seconds

812493 13:15:05.963  [4172] (JournalTask) <3636> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [1.715702] seconds

796078 13:15:03.916  [1200] (JournalTask) <4632> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [22.294424] seconds

For the 22 second external filter event, it's taking 22 seconds to perform a small part of the whole process to archive a single email? That is painful. A few of these and that would explain such poor archiving performance?

Our custom filtering does a DL lookup to tag messages with custom retentions based on DL membership.

Thanks

[goatboy]

is 15K a significant number of handles? Will try this anyway...

[JesusWept3]

Wow,
Well kill the MonitoringAgent then do another dtrace to see if that helps any with the custom filter
but you have hit the nail right on the head, if it doesn't get any better then you may need to call support to find out why its taking so long

Is that your own custom filter or is it something like orchestria or another third party piece that is applying the filters?

[goatboy]

It's our own filter:

<RULE NAME="10 Years RECIPIENT" RETENTION="10 Years">
      <RECIPIENTS>
           <DL>Retention 10 Years</DL>
      </RECIPIENTS>
</RULE>

<RULE NAME="10 Years AUTHOR" RETENTION="10 Years">
      <AUTHOR>
           <DL>Retention 10 Years</DL>
      </AUTHOR>
</RULE>

There's an additional 18 rules covering other DLs (9 AUTHOR, 9 RECIPIENT). Then 2 more rules at the end that do SUBJECT matching.

6891 items archived on average per hour, that's 116 per minute, and ~2 per second. I'll get some numbers around how long the custom filter is taking on average shortly.

[JesusWept3]

Thing about averages, you have to account for weekends, holidays , downtime due to backup modes etc, so the number can change dramatically

But still, 22 seconds for one of those messages is *really* bad

[TonySterling]

Do you have a lot of Internal SMTP Domains?

This issue is resolve in EV 9 SP4:

Environments with a very large number of internal SMTP domains experienced delays in journal archiving [Ref 13024, E2678932]

In some environments, delays occurred during journal archiving. This happened when custom filtering was enabled, and a very large number of internal SMTP domains was specified in the InternalSMTPDomains string value under the following registry key:

HKEY_LOCAL_MACHINE
\SOFTWARE
 \KVS
  \Enterprise Vault
   \Agents

This has been fixed.

[goatboy]

InternalSMTPDomains - this value is not present.

Backup mode - generally about 1.5 hrs a night for 5 nights when incrementals run, longer for fulls.

I pulled the "Time Taken to Process External Filter" values from an ~10 minutes DTRACE before and after killing the monitoring agent process.

Before - 3530 entries, average time 1.42 seconds

After (this DTRACE was shorter) - 2466 entries, 1.38 seconds.

Here are the top 20 numbers.

Before:

CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	103	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	82	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	33	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	33	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	33	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	32	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	32	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	29	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	29	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	27	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	27	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	26	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	23	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	22	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	22	seconds

After:

CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	165	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	163	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	106	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	72	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	30	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	28	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	27	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	26	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	26	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	24	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	22	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	22	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	22	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter	21	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	21	seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter	21	seconds

[goatboy]

This looks like the issue we're having: http://www.symantec.com/business/support/index?page=content&id=TECH191331

Problem

When using several Custom Filter rules based upon large distribution lists, the Exchange Journaling task is slow to process items.

But the fix is for 9.0.3 only!

[JesusWept3]

Then really your best bet is to to upgrade to 9 SP4+

[Nathan_Clark_2]

just a fyi the handle consumption is significant and should be fixed in the next available release.

[goatboy]

So we had some nested DLs in the DLs that were being looked up by the custom filter, that wasn't helping things. But removing them didn't help throughout that much. Created 4 new journal mailboxes and finally the backlog is going down, albeit slowly. Question, is Exchange server hardware a big factor in journal archiving rates? The Exchange server hosting the journal mailboxes is pretty old.

[JesusWept3 · Accepted solution]

It all really depends on how old the hardware is, and how many items per day you are journaling
The biggest issue you are going to face is disk queue lengths

But really the issue you are seeing is all AD lookups and thats where the slow down is coming from
Considering you're on 9 SP2 and there are significant number of custom filter improvements, your best bet really is to upgrade, may have made all the journal splits and multiple targets unnecessary

[goatboy]

Thanks. Are you talking disk queue lengths on EV server or Exchange? If Exchange, the disks containing the information stores or smtp queues or both? The numbers are good on the new journal mailboxes so I am pleased with the change. Agree with upgrade, except we have CA and DA version dependencies, so upgrade will need to cover these as well...

[JesusWept3]

Avg Disk Queue Length on the Exchange servers that host the Information stores
If it gets too backed up then you will see a rapid decrease in performance

Typically you will see this when users getting the yellow balloon popups on their desktop stating that items are attempting to be downloaded, and you could see that in outlook when connected to the journal mailbox on the EV Server.

Anywho! EV9 SP4 server is compatible with all CA and DA versions of 9
So like DA9 SP2 can work with EV9 SP4, or EV9 SP2 with DA 9 SP4
You just have to ensure that the EV API installed on those servers (or the EV Binaries) match the SP version of the main EV Server

So if you have EV9 SP4 server and DA9 SP2, you should install EV 9 SP4 binaries on the DA server so that the EV binaries match the server

You can check the compatibility chart here on page 87
http://www.symantec.com/business/support/index?page=content&id=TECH38537

[TonySterling]

Minor correction, the version of DA must be equal or greater than the version of EV. From the compatibility doc.

If the major version of Compliance Accelerator or Discovery Accelerator is the same as the version of Enterprise Vault, the minor version (Service Pack) of Compliance Accelerator or Discovery Accelerator must be the same as, or later than, the minor version of Enterprise Vault.

For example:

You can run Compliance Accelerator 10.0 with Enterprise Vault 9.0.2 servers, but not Compliance Accelerator 9.0.2 with Enterprise Vault 10.0.

You can run Discovery Accelerator 9.0.2 with Enterprise Vault 9.0.1 servers, but not Discovery Accelerator 9.0.1 with Enterprise Vault 9.0.2.