EV-server LSASS.exe causes high load on AD (up to 20Mbps)

[erik_peters]

We've just migrated 3000 users from EV9.0.4 to EV10.0.3. The new EV-cluster is now performing well, serving all users but they seem to "attack" the AD.

We enabled VC for all users and allow to build the new VC-cache with 20 concurrent treats. EV as well Exchange 2010 are performing well but we see a load of 20Mbps on our AD-infrastructure coming from the EV-cluster. Lowering the treats takes of some load but it is still way to much. Our Exchange 2010 CAS servers are only using 0.5Mbps serving +5000 localy cached mailboxes.

Question: Which component can cause such a high load on AD? I searched the documentation for requirements on AD-performance but nothing found there. Only one hit for Discovery Accelerator but we only use the Exchange-Archiving part of EV.

[JesusWept3]

is it constant? and does it happen at certain times
Typically the hardest your AD will be hit is during the provisioning task runs

[erik_peters]

It happens all the time. Actually right now it is between 5 and 10 MBps. A failover doesn't fix the problem. Rebooting a node and failback gives no better results. MS-queue's are empty.

Just stopped some services in sequence (Index>Shopping>TaskController>Storage). AD-load stays untill I stop de Storage Service.

[JesusWept3]

OK so do you do journaling at all?
also what version of EV are you on?

I suppose what you could be seeing is just the vault cache builds looking up sender/recipient information in AD. So if you look at http://yourEVServer/EnterpriseVault/VCView.aspx or http://yourEVServer/EnterpriseVault/ClientDiagnostics.aspx you can see what vault cache requests are being handled at that moment in time

[erik_peters]

No journalling.

Indeed, a lot of cache building is done. This wil be the case for months because we are migrating usersystems to Win7 and all caches need to be rebuild.

What are you trying to say: EV does for every single item that is stored in the tempory vault cache a lookup to AD to resolve recipient information? Is there any indepth documentation available about this process?

Erik.

[JesusWept3 · Accepted solution]

theres no low level tech documentation like that available.
At the moment i'm just making a best guess based on the fact that before enabling vault cache, the traffic was minimal, and now you have a lot of vault cache builds going on, its hammering your network and causing a lot of overhead on the Global Catalogs.

If stopping the storage service stops the calls, then what you might want to do is try and narrow down which part of storage is causing it, whether its StorageOnlineOpns, StorageServer, StorageCralwer etc

And get a DTrace of that process MigratorServer and see exactly what is it calling it when it accesses AD

[erik_peters]

I did a dtrace on "MigratorServer". I've got hundreds of lines from this kind:

1021143 14:29:41.391 [37252] (MigratorServer) <25300> EV:L {CClientIdentity::DetermineIdentity:#194} Converting [S-1-5-21-1009192273-2161540178-1385530778-12393] to a functioning SID, then checking if it is a user SID...

This looks like EV did found a SID that he doesn't know and tries to lookup the SID in AD (for resolving the displayname?).

I suppose during the initial build of Vault Cache this is done for all such items where Senders or Recipients in the vaulted item are deleted accounts in AD. In normal circumstances the lsass.exe process caches the SID. This is proven because there is not one single line in the log about resolving existing SID's. Only the non-existing SID's are there and they are there more than one time for the same SID. This implicates in a short time 4137 SID-lookups to AD resulting only in 278 unique SID's. Also this behavior looks normal. SID's that can't be resolved will not be cached by the local lsass-process.

So, many thanks to advise me about the dtrace-log. I hope my conclusion was the right one. Do you think I should ask Symantec Support for an official statement?