Archiving very slow on new install

I'm doing a brand new install of EV 10.0.2 - 1 server (virtual) with 4 cpus and 16 GB RAM, SQL on a separate server and Exchange 2010. After the install completed, we selected the first mailbox. A Run Now in Report mode took 3 minutes to complete and told us taht there were 56K messages to archive (2496 MB). I figured this should take an hour or so. I next did a Run Now in Archiving and Shortcut mode and it worked -- but it took about 9 hours! There were no errors and the performance was fairly consistent. I ran a dtrace to see what was going on but nothing was jumping out at me. After a couple of hours, I remembered tht we hadn't configured the A/V exclusions so we fixed that, rebooted and restarted the manual archive. Performance didn't improve. CPU load was under 5% and the message queues were empty. It only archived between 4500 and 5000 messages an hour @ average size 50K/msg.

I know that the recommended minimum server configuration is now 8 cpu cores but they insisted on going with 4 and it isn't the bottleneck right now. My next guess would be that it can't read the data from Exchange fast enough, but we ran the throttling script before we started archiving and verified that the archiving task started without any errors. The partitions are on a network share but the storage queue was always at 0 while I was watching so I don't think that is the bottleneck either.

Does anyone have suggestions on how to narrow this down?

thanks,

Mark

Enterprise Vault

Information Governance

Performance

JesusWept3
13 years ago
Well the thing is te CPU is really only going to spike on storage processing and content conversion, but if its a single thread and its just handling a steady stream of email, i wouldnt expect it to go all that fast.

The thing is theres a ton of moving parts, SQL Server (its own disk IO, network bandwidth, CPU usage), and the same on the exchange server and same on the EV Server. It could be SQL Server slowing down, it could be exchange performing some kind of maintanance etc

The issue really is that MAPI isn't tremendously fast in the first place, and you might just be seeing the limit of what exchange can do for a single mailbox anyway, but still might be worth just seeing what kind of loads you are seeing on the exchange side of things especially with disk queue length

TonySterling
Moderator
13 years ago
Can you share the version of Outlook including service pack you have installed?
AndrewB
Moderator
13 years ago
are there any server monitoring applications loaded on this server? NetIQ, LANDesk, anything like that?
Mark_Tkachyk
Level 3
13 years ago
Outlook is 2007 SP2 + hotfix 2475891
AndrewB
Moderator
13 years ago
there are a lot of caveats with SP2 for EV compatibility. it is definitely recommended to install SP3 and make sure to set the registry entries described in this Microsoft Knowledge Base article http://support.microsoft.com/kb/952295.
Jeff_Shotton
Level 6
13 years ago
I think from your description of the issue, your main issue may be that you were processing a single mailbox.

Enterprise vault is a multi-threaded application, and you would have been processing a single mailbox with a single archiving thread (only journaling uses multiple threads per mailbox). If you were processing lots of mailboxes, then you would likely have seen rates 5x greater than those you experienced. By default 5 mailboxes will be processed at the same time for exchange mailbox archiving.

You also never stated if the server CPU was maxed out at any point during the archiving run (only the average), but what I expect is that it probably looked pretty quiet whilst all the work was being performed.

Try doing more mailboxes concurrently and also potentially increasing the number of concurrent connections on the task and checking if the rate improves.

Regards,

Jeff
Mark_Tkachyk
Level 3
13 years ago
I agree that it was only using 1 thread and therefore should be expected to go significantly faster with the standard 5 threads. But the throughput I was getting was about an order of magnitude less than I had expected. CPU never spiked while I was watching it -- always in the single digits. I will try Outlook SP3 and try to get them to give me several users to run concurrently.
JesusWept3
Level 6
13 years ago
Well the thing is te CPU is really only going to spike on storage processing and content conversion, but if its a single thread and its just handling a steady stream of email, i wouldnt expect it to go all that fast.

The thing is theres a ton of moving parts, SQL Server (its own disk IO, network bandwidth, CPU usage), and the same on the exchange server and same on the EV Server. It could be SQL Server slowing down, it could be exchange performing some kind of maintanance etc

The issue really is that MAPI isn't tremendously fast in the first place, and you might just be seeing the limit of what exchange can do for a single mailbox anyway, but still might be worth just seeing what kind of loads you are seeing on the exchange side of things especially with disk queue length
Jeff_Shotton
Level 6
13 years ago
I would have hoped for maybe double, but no more than that through a single thread. If you look at the performance guide for EV10, 50k per hour for 8 cores would give you 6250/hour per core. Obviously you cant run a single thread on more than one core.

As Alex said, check out those other things, but also see if the archiving rate increases linearly with the number of concurrent mailboxes. If it does, then you arent maxing anything out except for MAPI.

Regards,

Jeff
Mark_Tkachyk
Level 3
12 years ago
I was able to run the archiving with more mailboxes and generate some stats.   The good news is that the throughput increased linearly with 5 active threads.   The bad news is that it is about half what the performance guide would indicate.    The EV10 performance guide starts with 8 cores but the EV9 peformance guide listed 40K per hour for 4 cores (what we have deployed) and I am getting about 20K per hour.    Attached is a stats spreadsheet for the EV (EV01) and Exchange Mailbox (MB01) servers.   Both are virtual servers and they were collecting a handful of performance statistics from them.   I don't have any stats from the SQL server as yet.   The table shows a 22 hour period with continuous archiving although it probably was only running 5 threads for the first several hours.   Do these numbers look excessive to anyone or do I need different stats?

I opened a case with support and after analyzing the dtrace, they came back with the suggestion that it might be the SQL server because of these methods:

Archive Task

Function ----------------------- Avg Time

{CMAPISaveset::_Insert}      0.246

Storage Archive

Function ----------------------- Avg Time

{SavesetReceive}                  0.609

It's a new EV implementation so I can't understand how SQL could be a bottleneck at this stage.   Maybe the server is too busy with all the other databases on it.   But the throughput I've measured has been very consistent so far and I've run archiving on different days at different times.
EV and Exchange stats.xlsx13 KB
AndrewB
Moderator
12 years ago
sql is really the backbone of EV and it is always recommended not only to have SQL on a seperate server from the EV server but to have that server dedicated to EV databases. in some cases, with a very large cluster, we have seen good performance when sql had its own dedicated instance on a shared cluster. do you have any more information you can share about the sql server you're using? have you run any perf stats to see if it's overloaded during your archiving runs?