Highlighted

EV 11.0.1 CH3 - Journal Tasks Falling Behind

Hi Symantec Community,

I am running 4 VMWare VMs with EV 11.0.1 CHF3 all with Windows Server 2012 (not R2).  We are running Archiving on 8 Exchange 2010 SP3 Mailbox servers and Journaling 1 Exchange 2010 SP3 mailbox server.  Two of the EV servers are dedicated to Exchange Mailbox Archiving (4 tasks - 1 per Exchange server - staggered timing), and two of them are dedicated to Journaling (3 tasks - all same Exchange server).  The issue is with our Exchange Journal mailboxes getting hammered with load that the EV Journaling servers cannot keep up.

Servers' Specs:

Archiving/Directory Server1 - 16 cores/16 GB RAM

Archving Server2 - 8 cores/16 GB RAM

Journaling Server3 - 8 cores/16 GB RAM - 3 mailbox targets (2 active and 1 inactive)

Journaling Server4 - 8 cores/16 GB RAM - 2 mailbox targets (1 active & 1 inactive)

Targets:

Journal-Inactive (not being Journaled via Exchange, just trying to catch up)

Journal-Inactive-2 (not being Journaled via Exchange, just trying to catch up)

Journal-Active (Journaled via Exchange databases)

Journal-Active-2 (Journaled via Exchange databases)

The inactive mailboxes are no longer being Journaled via Exchange databases, but the active ones are via Exchange database level journaling.

We have approximately 300,000 emails journaled per day, and we've allocated two dedicated servers to balance this load.  We have two Journal Vault Stores writing to the same EMC Centerea.  After much troubleshooting with Symantec and our vendor, we have NOT reached a solution on why our Journaling stalls out.  My tasks are set to 10 concurrent connections with 10,000 items per pass for the inactive mailboxes, and the active ones are set to 5 concurrent connections with 5000 items per pass. 

Each Vault Store has it's own SQL database on a beefy physical server.  The physical SQL box has 20 cores and 256 GB of RAM.  I regularly perform offline maintenance on the affected databases, per the TECHNOTE from Symantec, but this only has a temporary effect on the speed/frequency of archiving via the Journaling tasks.

I have noticed that we encounter many DCOM errors in the EV logs, and there is also a number of C-Clip errors regarding missing or failed savesets.

The MSMQ queues are within reasonable limits, but I've noticed that the Storage Queue Limits are being met (250,000, which is the limit on the server level within Admin Console).

Does anyone know if there is a precedent for this issue, and how I may be able to resolve it?  Symatec support has had very limited success with troubleshooting, so I have come here for some additional support.

Thanks in advance!

Josh

1 Reply

A couple questions: 1.  What

A couple questions:

1.  What is the hourly archiving performance on the Jounal vault stores.  Given the specs you've provided I'd expect to see 35-50K items archived per hour.

2.  You're using VM servers for EV, are the resources dedicated?  EV requires dedicated resources on the VM servers.  Waiting for resources on the EV server from the host will kill performance.

3.  What is the resource load and specs of the VM Host server(s).

4.  There are tuning parameters on teh Centera to increase the number of threads for processing the data flow.  The default setting is not enough.  I don't recall the exact setting at this point, but I recall having to increase (double) them in my environement when I was using a Centera.