Multipe IndexServer.exe processes take all CPU after upgrade to version 8 SP3

[Hans_Peeters]

Hi,

Last week we upgraded in 2 steps from version 7 SP4 to version 8 SP3.
We archive mailboxes between 6PM & 6AM.
During office hours we do not run any task besides provisioning task (twice) & synchronise (twice)

Already for 2 days we see a huge CPU load during office hours. CPU > 90%
Looking at Task Manager we see a lot of indexserver.exe processes.

What is happening here? How to troubleshoot this? What can be the reason?

Every hour we also see some warning events in event log:
Event: 7303
Category: Index Broker
--> Timed out waiting for an IndexServer to stop

Regards
Hans Peeters

[GertjanA]

Hello Hans,

From the top of my head...

Check the policies. Especially the mbx policies. Somewhere in there, you will find a tab called 'moved items'
I believe it is active, de-activate it.

What this does is to update the archive, to reflect the Outlook-folder structure.
(ie, you archive something from Outlok, then re-arerange folders in Outlook, that will not reflect in archive explorer)

This places a load on EV and Exchange servers.

I have started this gradually (1 exchange server at the time using provisioning), and am almost near having done everyone...

I suggest you disable this first, then check to see if it helps. If nmot, dtrace indexing I guess

[Hans_Peeters]

Hello Gertjan,

This new move-shortcut-resync-archive feature was one of the reasons of our upgrade.
Some users were complaining that folder structure of archive explorer becomes out of sync with outlook if you move shortcuts or rename folders.

I thought this resync process was scheduled during archiving schedule. In my case between 6PM & 6AM.
I will disable this feature on all policies and evaluate.

Help shows below:
Updating is done as follows:

• When a scheduled run of the Exchange mailbox archiving task occurs.

• When you use Run Now with a Run mode that includes shortcut processing.

In my case it is very weird that the moved items press keeps on running after 6AM....

[Hans_Peeters]

Hello GertJan,

- For all Exchange Mailbox Policies I disabled the option "Update archive location for items moved in the mailbox" in tabsheet "Moved Items"
- Then I stopped all Tasks & services & rebooted EV server.

After reboot immediately same problem: high CPU and a lot of IndexServer.exe processes

[GertjanA]

Hello Hans,

I assume you have run provisioning again after changing the policy, and when that finished have resynched the mailboxes?
Otherwise it would not have an effect...

Is there anything else perhaps going on? Is there perhaps an index-rebuild going on?
Any errors in the upgrade logs?
Do you have any other errors in the eventlog (on the EV-servers, check also application and system, do not forget the sql-server)
Anything on the Exchange-servers?
Is there perhaps a change in virus-scanning happening you are not aware of?

What happens (if all of the above is ok) if you plave all archiving tasks in report-mode, and restart them. I seem to recall (but have not access to the update doc) that when you did not put your tasks in read-only before upgrading, this issue might occurr. Check the upgrade documentation for known issues. Perhaps something in there.

If that is not helpfull, and you are really worried, and you have a support-contract with Symantec, give support a call. They might be able to help out.

[Michael_Bilsbor]

Hi,

I agree that modifiable meta data is the likely culprit.  In actual fact modifable meta data is processed after the scheduled run to try and not impact your archiving schedule.

Though you have turned off the feature, my guess is that there are outstanding requests still queued up and so as soon as you rebooted those requests would be picked up and consume the CPU.

If you look at MSMQ you will see queues with A6 in the name, I expect these have entries on them which are causing the high CPU.  You could purge those queues and i expect that CPU would die down.

At the end of the day though you need this feature to work, so it may be that you need to tune the enterprise vault system so that it temporarily at least consumes less indexserver processes and therefore less CPU.  Speak to support, there are registry keys called something like MaxIndexServers that may be useful for you to use.

Cheers,
Mike

[JesusWept3 · Accepted solution]

The question for me would be, is this genuinely a problem that is causing other processes to become unreponsive? If you just let it ride it will clear up soon enough but it's going to take some time dependant on how much data had been archived and moved since your initial ev deployment but really if you look at the CPU specifications and memory specifications for ev, this is one of those times when you want the big beefy servers, after all what's the point in purchasing a large $10,000 server if you're only going to use 10% of it's potential at any given time?

[LanceChase]

Have you checked the number of items in the Queues for the servers? Even if you disabled everything and restarted, as soon as you did the processing on the queues would continue.

[Hans_Peeters]

High CPU load issue is solved by just waiting.
Most probably it was caused by some outstanding requests.