Journal archiving not working after upgrading to EV11

[woelki]

Hey guys,

in the last days we recognized that our journal archiving is not running properly. The mailbox size increased up to 400GB. Unfortunately EV never told us that something is wrong.

Short details about environment:

Exchange 2010 DAG SP3 RU14

EV11 CHF5

- ARC01 (FSA)
- ARC02 (Index)
- ARC03 (MBX and Journal Archiving)

Before upgrading to EV11 (from EV10) we were able to archive the regular mailboxes (4 MBX server), archiving for one journal mailbox and do a migration from a foreign vault site and this at average 60% CPU utilization. Regular archiving now still work but journal archiving seems to be very sensible. Currently because of the high backlog we created a new journal mailbox and a new task in EV.

We already created a case at Veritas and our supporter from India is quite good, but now after the second follow up call we are still fighting the symptoms, not the root cause.

This is what we are doing to get the journal archiving working again:

In the mailbox we find 3 message classes

- IPM.Note
- PendingArchive
- PendingArchive Part

We stop the task controller service and convert the EV classes back to IPM.Note.
The we purge the messages in the journal queues of MSMQ. Afterwards restart single storage service or sometimes the whole MSMQ. The journal archiving is working, but I extremely wonder about 100% CPU utilization.

Unfortunately it seems that the journal archiving is very sensible at this moment. For example it runs until the backup mode starts. After releasing the backup mode it is not willing to continue. Then we have to execute the steps above again.

In most cases the journal task is able to convert the messages to PendingArchive Part, but only a small amount of items can be queued to the MSMQ, with small I mean about 50 messages, but it should be more about 50.000

So the problems seems to be that the messages cannot be queued.

Have you got any further ideas, what I can look for?

Kind regards,

Christian

[WiTSend]

1.  What is the size of our MSMQ storage?  

2.  CPU at 100% is not unusual

3.  How many storage threads do you have. 

You can increase the size of the MSMQ to give more headroom for moving the data.  If you storage device can handle more load and your memory usage is reasonable on the EV server you can also increase the number of threads in the Storage service.  

Also, when you say "during backup", what are you doing for backup?  Are you putting both the vault store and Indexes in backup mode?  You should also consider stopping the Journal archive task so that additional emails are not queued up in MSMQ during backup.

[woelki]

Hi there,

MSMQ:

Free space: 25GB
Queue total size: 20GB
Queue total length: 89903
Queue pending length: 50002

Yes, we are putting index and and vault stores in backup mode. I know that the messages get queued in the journal mailbox during this time, but as the backup mode is running after COB, not that much messages are stored in it... and like I said, some weeks ago it was no problem at all.

Regards,

Christian

[CConsult]

Did you change the schedule of any tasks?

What does your Archiving Report say?

rightclick on corresponding Archiving Task >>Reporting>>View Reports>>select Newest Report>>More Details (next to Mailbox processed)

[JRuss]

Hi Christian,

If your journal mailbox is 400GB then EV will be slow to archive it due to the amount of time it takes for exchange to respond to MAPI requests.  Journal Grouping is most likely the bottle neck here, I would suggest turning off grouping (setting journal delay to 0) until the backlog clears, then set journal delay back to its original value.  The only negative to this is that you will have more items stored than normal, but it does allow you to ingest the backlog quickly.