EV 8.0 archiving very slow

[Elliot_G]

Recently we had our EV 7.0 installation (2 EV servers on a 3 node Microsoft cluster) upgraded to EV 8.0 (SP3). The installation went to plan and everything has continued to work. Except for a matter of speed.

Previously we could, in our 5 hour archiving Window, make a full pass through all the mailboxes that each server needed to process. Since the upgrade, we are lucky to get through half the mailboxes each night. On Sunday we have a 10 hour Window and this seems to work. So our archiving rate has halved.

At the time the fact that the archiving run now includes a new element – shortcut processing – by which moved shortcut locations are updated in the archive database, might be what was slowing everything down. So we switched this off (from Mailbox Policy) but it made no noticeable difference.

It is not the archiving itself that is holding things up – the archive queue used to be in the hundreds in the middle of a run. It is now generally in single figures.

Has anyone else had this problem, or is unique to our environment? It is causing something of a backlog of items waiting to be archived to build up. I ran a manual archive in report mode and it showed this developing – as well as taking about 8 hours to run!

[patrickkuah]

Hi,

Is this Exchange Mailbox archiving? If yes, log in a support case. Tell them this is an UPGRADE and for EXCHANGE MAILBOX ARCHIVING. they will verfiy if what you are seeing is actually a bug.
Thanks.

patrickkuah

[JesusWept3]

Lets not jump to the "its a bug!" route just yet.
So firstly the few things that make EV8 slow (for the moment) is

 - Auditing
 - Update Shortcut Location
 - Monitoring

so the questions would be
1. How are you determining archiving is slow?
2. Do you have an idea of what the throughput was before and after upgrading to EV8?
3. Do you have anything such as "RestoreShortcutBody" or Delete Empty Folders enabled in your registry key?
4. How many Storage Server threads have you set in the properties of the storage service?
5. How many concurrent MAPI Connections are set on the tasks?
6. How many tasks do you have per server?
7. What is the MSMQ quota and are you sure you haven't breached it?
     (Default in 2003 is 1GB, check the properties of your physical MSMQ location to check, as many dead letters or MSMQ journaling may be filling it up)
8. Are you using Vault Store Groups and sharing between them? (OSIS)
9. Are you running Outlook 2003 or Outlook 2007 on the server?
10. Are you seeing any excessive Locking or Blocking on the SQL Server?


Step 10 is the really important one to determine if you are seeing any infinite loops being generated by the Archive Folder View  due to parent folder issues
also there are issues with NULL folders in the same view also.

For those issues then a case would need to be opened with symantec to help resolve it

[GertjanA]

Apparently you upgraded 'quickly'from 7 to 8. Did you work with 7,5 a little? Same issue? Or did you do 7 to 7.5 to 8.0?

See list of JW for other necessary checkes

[Redsnake]

Also if you have replication enabled, check whether your replication is Asynch or Sync. It should be ASynch. That makes a huge difference.

To repeat the question above, How do you determine you archive rate?

[Elliot_G]

Thanks for these suggestions. i will work through them on Monday and get back to the list.

One immediate question though. I know very little about SQL Server. How do I know if  "Are you seeing any excessive Locking or Blocking on the SQL Server?" I am not sure how to detect this, or what would be excessive.

Elliot

[Elliot_G]

From the replies I gather that this is not the usual situation. So clearly something is up with our environment?

Frist of all, some background. It is indeed Exchange mailbox archiving. We have 2 EV servers (actually Microsoft cluster nodes). EVserver1 archives from 1 Exchange server, EVServer2 gets the big job of archiving from 2 and hence has twice as much work to do. But both are showing the slowdown. the upgrade was a 2 stage 7 -> 7.5 -> 8. All done the same day.

Now to answer the specific questions:

1. How are you determining archiving is slow?
By the fact that it takes twice as long (on average) to process the same number of mailboxes as before.

2. Do you have an idea of what the throughput was before and after upgrading to EV8?
On EVserver 1 - about 3500 mailboxes in 3-4 hours. Average about 45,000 items archived in a run. Since then this would take over 6 hours (but only 5 hours in a daily run, hence completed next day) . Numbers of items archived has dropped, buut not be that much.
On EVserver 2 - about 7000 mailboxes in a little over 5 hours. About 90,000 items per run. Now takes abouty 10 hours. Numbers of items archived in the 5 hour nightly run has dropped, but not be that much. (Because each mailbox now only gets archived on everage every second day)

3. Do you have anything such as "RestoreShortcutBody" or Delete Empty Folders enabled in your registry key?
Cannot find anything like that.

4. How many Storage Server threads have you set in the properties of the storage service?
10. Did try dropping this back to 5, but this made things worse

5. How many concurrent MAPI Connections are set on the tasks?
10. Once again tried reducing this, but it was slower.

6. How many tasks do you have per server?
EVserver1 - 1 archiving task. EVserver2 - 2 archiving teaks.

7. What is the MSMQ quota and are you sure you haven't breached it?
     (Default in 2003 is 1GB, check the properties of your physical MSMQ location to check, as many dead letters or MSMQ journaling may be filling it up)
Some time ago we raised the quota from 1 GB I think to 8GB. I cannot rember why, but was in relation to a support call. EVserver1 is generally lower than 90 MB.. EVserver2, for some reason is 1.16 GB, even when all queues are empty!

8. Are you using Vault Store Groups and sharing between them? (OSIS)
Yes. Two vault stores in one group.

9. Are you running Outlook 2003 or Outlook 2007 on the server?
Outlook 2003

10. Are you seeing any excessive Locking or Blocking on the SQL Server?
I have no idea how to look for this. The event logs on this server seem clean.

I hope this information will enable someone to point us in the right direction!

Elliot