Enterprise Vault 10 Indexing hammers server

[BigPhil]

So I have been running Enterprise Vault 10 since it first came out (upgraded from 9.0.2) and have had a growing problem with the indexing service. When the indexing service is stopped and all other services running, task manager is showing about 80 processes. As soon as I start the indexing service the process count on the server goes up to about 375 and the processors (8 on this server with 8 gb ram) goes up to 90-100% constantly.  This did not happen with Enterprise Vault 9 and below. Obviously the change to the indexing service in v10 has a huge impact on performance.

Another thing that happens is that when I use the powershell scripts to set the backup mode for the indexing service and vault stores. The scripts usually hang when trying to set the backup mode to true/false for the indexing service. The following events are also logged when this happens:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          9/20/2011 7:55:24 AM
Event ID:      40966
Task Category: Index Admin Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EVSERVER1.domain.com
Description:
A program fault has raised an exception.

Exception: The creator of this fault did not specify a Reason.
Diagnostic:
Type: System.ServiceModel.FaultException`1[[System.Runtime.InteropServices.COMException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]
Reference: An error occurred while invoking Void <NotifyIndexLocationBackupModeChangeEx>b__4().
Error Context : Change index location backup mode to True for IndexRootPathEntryId 15AC2C3D256C4F745A97FB3B4591E40261810000DomainEV

Command Line: "e:\Program Files\Enterprise Vault\EVIndexAdminService.exe" -EntryID:18022E54F0CCF2F4283E0732E2C5B302A1710000DomainEV
Application Domain: EVIndexAdminService.exe
Process Id: 6528
Thread Id: 14988
Stack Trace:    at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.COMInvokeHelper(Object pICOMInterface, Action act, String context, Boolean rethrow, Boolean logError)
   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminCOM.NotifyIndexLocationBackupModeChangeExInternal(String indexLocationEntryId, Boolean inBackMode, Boolean wait, Boolean logEvent)
   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminCOM.<>c__DisplayClass6.<NotifyIndexLocationBackupModeChangeEx>b__4()
   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Action act, String context, Boolean rethrow, Boolean shutdownCheck)

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          9/20/2011 7:55:24 AM
Event ID:      40966
Task Category: Index Admin Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EVSERVER1.domain.com

Description:
A program fault has raised an exception.

Exception: Error HRESULT E_FAIL has been returned from a call to a COM component.
Diagnostic: HRESULT: 80004005
Type: System.Runtime.InteropServices.COMException
Reference: A COM error occurred while invoking Void <NotifyIndexLocationBackupModeChangeExInternal>b__35().
Error Context : Change index location backup mode using IndexBroker for RootPathEntryId 15AC2C3D256C4F745A97FB3B4591E40261810000DomainEV

Command Line: "e:\Program Files\Enterprise Vault\EVIndexAdminService.exe" -EntryID:18022E54F0CCF2F4283E0732E2C5B302A1710000DomainEV
Application Domain: EVIndexAdminService.exe
Process Id: 6528
Thread Id: 14988
Stack Trace:    at KVS.EnterpriseVault.Interop.IndexAdminClass.SetIndexLocationBackupMode(String indexRootPathOrIndexingServiceEntryId, Boolean backupMode)
   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminCOM.<>c__DisplayClass38.<NotifyIndexLocationBackupModeChangeExInternal>b__35()
   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.COMInvokeHelper(Object pICOMInterface, Action act, String context, Boolean rethrow, Boolean logError)

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          9/20/2011 7:55:24 AM
Event ID:      7264
Task Category: Index Broker
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EVSERVER1.domain.com

Description:
Abnormal error occurred
 
Error:     Exception occurred.  (0x80020009)
Reference: CCache::SetIndexLocationBackupMode
Index:     
Info:      EM/e
           

Needless to say, Enterprise Vault 10 is really irritating me!!! Does anyone have similar problems or any ideas on what to try? This is a single server setup with the latest EV hotfixes and Windows updates installed, about 1800 mailboxes archived and 1 jouranling target.
 

[Paul_Grimshaw]

I would log a support case ASAP so we can look into it. As you can appreciate there is not much within the internal knowledge base as the technology is so new but obviously this performance impact should not be occurring.

[BigPhil]

Paul, I did end up logging a case with support this afternoon so we'll work on it tomorrow and see what happens. v10 has been nothing but a headache for me so far, so hopefully we can work some of the issues out and get it running good again. I'll update with our findings.

[Paul_Grimshaw]

Hi Philip,

I am sorry to hear that your initial experiencde with EV10 has not been as successful as you would have liked and hopefully we can get the fixed up for you ASAP. We have seemed to have a good response to EV10 in respect that I do not see a lot of major EV10 related cases coming my way currently so hopefully trhis an environmental anomoly we can identify and fix up for you pretty quickly. As you say please inform this thread on your progress.

Paul.

[JesusWept3]

Awesome follow up, thanks!!!!

[BigPhil]

Support case #: 415287391

So we made some tuning adjustments and the performance of the server is better now. We changed the indexing admin task as well as the Exchange archiving task schedules to not overlap when they are running (basically the indexing is now running during the day and archiving during the night). We also bumped the ram up from 8gb to 16gb to help with the indexing (apparently there is some metric the software uses to throttle the indexing aggresiveness back if the memory is insufficient...also 16gb is the recommended amount per the Enterprise Vault 10 performance guide). A change was also made to the Exchange archiving task to increase its connection to the Exchange server from 5 to 10 (because the EV server has 8 processors) as well as decrease the items archived per pass from 1000 to 500 (default appears to be 200). The last thing changed was the storage service processes was increased from 5 to 10 (again because the EV server has 8 processors...also in the performance guide).

With all that said, the server is not flat on its face anymore but the powershell scripts to set EV in backup mode still hang when setting/clearing backup mode for indexing. The engineer said that because my indexes/vault stores/sql databases are on NAS storage and utilizing snapshots for backups that I may not have to put the indexes/vault stores in backup mode. He wasnt totally sure about that, so I may make another post about that and see what peoples thoughts are on that. I am on NetApp storage so I still need to use a trigger file to tell EV that its been backed up but the only way it will check is if the storage service is restarted or the vault store is put in/taken out of backup mode. anyways...it seems to be running a bit better now.

Enterprise Vault 10 performance guide: http://www.symantec.com/business/support/index?page=content&id=DOC4553

Enterprise Vault 10 indexing best practices guide: http://www.symantec.com/business/support/index?page=content&id=DOC4250

Edit: also forgot to mention...as far as the number of processes that are running on the server...apparently its the nature of the beast with this version of Enterprise Vault and can fluctuate a bit from server to server. So seeing 300 or so processes running should not be a concern at this point.

[BigPhil]

So we're back to the same old stuff...the server is just being pounded. If anyone else is running Enterprise Vault 10, how many processes does the task manager show are running when the indexing service is started and the index admin task is running? I'm hovering around 450 processes now which seems totally crazy to me.

[GertjanA]

Hi Phil,

No solution or advice. This is worrying. Is there perhaps an issue with your index-locations?

Is perhaps an update to 64bit indexes going on on all indexes?

Is perhaps a lot of rebuilds going on?

How is your MSMQ looking? Lots of archiving going on?

Please keep us updated on any progress you make with support. We might then be able to prevent this from happening.

Thanks,

Gertjan

[Nathan_Clark_2]

BigPhil, do the PS errors occur when indexing is stopped? what happens when you re-try?

Re perfomance can you send me the HTML output of a perfmon /report (System Diagnostics report) whilst experincing perfomance issue?

Thanks

nathan_clark@symantec.com

[AndrewB]

agreed. this is one of the best discussion threads i've come across in a while. Thanks to the original poster for being so thorough.

[BigPhil]

@GertjanA

There are no upgrades going on from 32bit to 64bit indexes, there are no index rebuilds going on that I can see (indexing monitoring window has zero things listed in it with no filters applied), I have stopped all tasks except provisioning task and indexing admin task, there is about 50k items in archiving a1 but I would expect that to be there because the archive task is stopped, but other than that they are clean.

@Nathan Clark 2

I'm not sure what happens when I run the scripts when the indexing service is stopped. At this point, getting the scripts to run properly again is the least of my worries ;) I'll try to get a perfmon going at some point.

Update: So last night I stopped all tasks except indexing and adjusted the indexing schedule to run 24/7 because I saw an error in the system status page indicating that there were 119705 items that needed to be indexed. I looked at the server this morning and there are still 116655 items to be indexed. Its like the indexing is just taking all resources, but doing nothing with them! I had stopped the journaling task yesterday also so now my journaling mailbox is up to 80k items and growing until we can figure out the indexing issue.

[Paul_Grimshaw]

Phil,

Get the case re-opened if it has been closed and get somebody to take a look at why your indexing rate seems to be so slow to cause the backlog.

Cheers..Paul.

[Nathan_Clark_2]

Just to follow on from Mr Grimshaw this number of processes is normal if you are ingesting (journaling and mailbox).

This should not cause any overall impact on the general functionality, as per Paul suggestions your real issue is the ingestion rate and therefore utilization (which needs to be investigated thru the support case)

Thanks

Nathan

[BigPhil]

The case is still open and has been elevated to tier 2 engineering. We have also re-install the binaries over the top of the exisiting install and install a couple of the hotfixes again. The server is no longer hammering the cpu's, but its not doing anything. It now finally showed that the latest journal index was failed which was a plus. I attempted a rebuild of the index but the rebuild failed and the report said something along the lines "unable to put the index in ReadOnly mode" and then stopped the task. I then tried to delete the task from the indexing monitor window and it wouldnt delete and just hang there. After a reboot and another attempt to delete the task I then got a message on the indexing monitor task window that said "The following subtasks could not be deleted: Reason: Indexing Service: Class not registered. I stopped the indexing service, restarted it and then was able to delete the task. I then did a search for any failed indexes and guess what...no failed indexes now!? I then attempted a synchronize against the index that had been previously marked as failed and now I get the following error, Event ID: 41314, Task Category: Index Query Server, "The 64-bit search engine is not running. The Index Query Server will attempt to start it."  Nothing appears to be happening and the Index Administration Task status just says "Processing" all the time.

I then stopped and deleted any indexing tasks/subtasks and then deleted the Index Administration Task itself and re-created it. I no longer see any warnings/errors about 64-bit search engine not started, but the indexing task still seems to just be sitting there. The schedule for the task is also set to allow it to run now.

[BigPhil · Accepted solution]

Ok, case closed! It appears that there might have been two main causes for the bad performance.

1) The cpu's going crazy could have been caused by a corrupt installation. I re-installed EV ontop of itself and this seems to have cured that problem. I'm not 100% convinced of this though. After reinstall, it was still going crazy, but then...all of a sudden it marked the latest journaling index as failed and then the cpu's stop going crazy. It was late and I didnt take any other action on it. The next morning I looked at it and the index was no longer failed but apparently self healed itself. I ran a sync task on the index and it didnt report any problems with it.

2) This was probably the biggest factor in my case. We are using a couple of older NetApp 3020 heads that was hosting the indexing data. I started on EV 7.5 and had always used cifs to connect to the index locations...bad move with EV 10, period....read the indexing performance guide a few posts up for great indexing info! The NAS is very busy and doesnt have enought IOPS for EV 10 indexing and it just couldnt keep up with the ingestion rate of messages. I have since moved the indexes to a not as busy EqualLogic SAN group we have and the performance is night and day! I'm sure I could have moved the indexes to a LUN on the NetApp, but the thing is very busy and old so I figured the better move would be to get off of it for now. We are going to upgrade to new heads and disks so I'll move the data back once thats complete but use an iSCSI connection instead.

[GertjanA]

Hi Philip

thanks for the update.

Good to keep in mind for the future..

Gertjan