EV9 poor (journal) archiving performance

concurrent connections isnt going to make much of a difference
neither will messages per pass really

You can have different tasks and targets going to the same archive

You can split up the mailboxes if you want, that should give you better performance
But honestly, something is desperately wrong here, the numbers are really bad

I know we had a similar situation where the throughput was worthless, and we found that the EV Monitoring Agent had a significant handle leak and was consuming a LOT of resources, we killed that and then journaling flew

So somethings you may want to check
Handles and memory counts of all processes on the machine
Have a look at disk queue lengths on the EV Server
Have a look at disk queue lengths on the exchange server
Have a look at disk queue lengths on the SQL Server

Also on the SQL Server, check and see if there are any processes being blocked or deadlocked, and that it isnt SQL causing the slow down.

And if you're on Exchange 2003, it may be worth using Outlook 2003 instead of 2007 as 2007 is needed for Exchange 2010, its recommended to use outlook 2003 for Exchange 2003 and 2007 due to the fact that MAPI is more performant on Outlook 2003

Also do a cursory check of Application and System Logs on EV/Exchange/SQL to make sure we're not getting any issues from hardware.

And its always a good idea to look an make sure that the NIC's and Network are in good shape, that you're not dropping packets or anything like that

Also you can do a dtrace of JournalTask and StorageArchive and see if there are any real slow deltas
 

Already on Outlook 2003.

Monitoring agent is a big consumer of memory, and has ~15K handles.

Is it safe to kill? What is it and does it need to be disabled permanently?

thats the monitoring agent, it writes all the statistics to the EV Monitoring Database that you need to view some of the SQL Reports or up to date information in the Enterprise Vault Operations Manager (EVOM)

Also what are your backlogs like in journaling at the moment?

oh and end task it, EV will start it back up

 238552 backlogged items in the journal mailbox. 

I ran the Dtrace, what should I look for? I am seeing these sorts of numbers, not sure what is (ab)normal:

800762 13:15:04.259  [4172] (JournalTask) <4160> EV:H CArchivingAgent::ProcessItemInternal Time Taken to Process Item - C026EF85FDA3FAC6F343BBF6D443F171: 1.200357 secs

844408 13:15:22.963  [5100] (StorageArchive) <496> EV:H SavesetReceive Time taken to store item - C03D7F08D0CD7C3E1F8FDCA4083829F1: 0.241217 secs

840189 13:15:22.556  [3736] (StorageArchive) <5052> EV:H SavesetReceive Time taken to store item - C03D788ECB1A023B41C8B119C42E79B1: 0.185905 secs

789694 13:15:02.947  [4172] (JournalTask) <5580> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter]: [0.010175] seconds

825483 13:15:08.072  [1200] (JournalTask) <8968> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [5.138734] seconds

812493 13:15:05.963  [4172] (JournalTask) <3636> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [1.715702] seconds

796078 13:15:03.916  [1200] (JournalTask) <4632> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [22.294424] seconds

For the 22 second external filter event, it's taking 22 seconds to perform a small part of the whole process to archive a single email? That is painful. A few of these and that would explain such poor archiving performance?

Our custom filtering does a DL lookup to tag messages with custom retentions based on DL membership.

Thanks

is 15K a significant number of handles? Will try this anyway...

Wow,
Well kill the MonitoringAgent then do another dtrace to see if that helps any with the custom filter
but you have hit the nail right on the head, if it doesn't get any better then you may need to call support to find out why its taking so long

Is that your own custom filter or is it something like orchestria or another third party piece that is applying the filters?

It's our own filter:

<RULE NAME="10 Years RECIPIENT" RETENTION="10 Years">
      <RECIPIENTS>
           <DL>Retention 10 Years</DL>
      </RECIPIENTS>
</RULE>

<RULE NAME="10 Years AUTHOR" RETENTION="10 Years">
      <AUTHOR>
           <DL>Retention 10 Years</DL>
      </AUTHOR>
</RULE>

There's an additional 18 rules covering other DLs (9 AUTHOR, 9 RECIPIENT). Then 2 more rules at the end that do SUBJECT matching.

6891 items archived on average per hour, that's 116 per minute, and ~2 per second. I'll get some numbers around how long the custom filter is taking on average shortly.

Thing about averages, you have to account for weekends, holidays , downtime due to backup modes etc, so the number can change dramatically

But still, 22 seconds for one of those messages is *really* bad

Do you have a lot of Internal SMTP Domains?

This issue is resolve in EV 9 SP4:

Environments with a very large number of internal SMTP domains experienced delays in journal archiving [Ref 13024, E2678932]

In some environments, delays occurred during journal archiving. This happened when custom filtering was enabled, and a very large number of internal SMTP domains was specified in the InternalSMTPDomains string value under the following registry key:

HKEY_LOCAL_MACHINE
\SOFTWARE
 \KVS
  \Enterprise Vault
   \Agents

This has been fixed.