Index Volume v10.0.2 keeps failing

[Victor_Geyyer_2]

Hello All,

I was wondering if you could share how to address the following problem:

Index Volume for recently (12 days ago) created 10.0.2 Journaling Vault Store started failing today.  So far it failed 3 times.

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          1/23/2013 4:18:32 PM
Event ID:      41329
Task Category: Index Volumes Processor
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MyEVServer.MyDomain.com
Description:
Index volume is marked as failed.

Archive Name: MYEVJOURNAL
Index Volume ID: 1126AEF7E8493C346B70D777F8A052E091110000XXX.MyDomain.com_1534
Reason: IndexingEngineError
Error Type: CriticalError

Description: Indexing engine exception

------------------------------------------------------------------------

2 times I was able to 'Synchronize' that index.

3rd time the 'synchronization' task is just there with status 'inactive' (why 'inactive'?), but index is down.

Currently index has 1382806 Indexed Items.

Any ideas how to proceed in my situation will be greatly appreciated.

Thank you!

[TonySterling]

Hey Victor,

you can use this to find why that volume is failing:

Getting a specific Index Volume back into a normal state (from a Failed / Rebuilding state)

Article:TECH168134

|

Created: 2011-08-26

|

Updated: 2012-07-28

|

Article URL http://www.symantec.com/docs/TECH168134

To find index volumes that have failed, use the query:

use EnterpriseVaultDirectory
Select *
from IndexVolumeView
where Failed = 1

[Victor_Geyyer_2]

Hi Tony,

Thank you so much for the respond.

Indexing works strange (to me) in the EV 10.0.2.

It has been failing through the night: the error I mentioned above was recorded in the event log: @23:25; 01:06; 02:05; 3:05 – though nobody “synchronized” indexes between failures, so I would expect failed index to stay failed.  And yet – this is not the case.

In the morning index appeared to be healthy and SQL query returned no failed indexes.

Could you shed some light on the logic of EV behavior, if possible, please?

Thank you in advance!

[Victor_Geyyer_2]

Hi Tony,

So it failed again.

The FailedVolumeReason column has 13, which is "Indexing Engine Error".

What would you suggest as the next step?

Thank you!

[TonySterling]

that error code would imply the Indexing is overloaded.  Are you currenlty updating your 32 bit indexes?  Is your Index location still local to the EV Server?

Any other events around those in the app log?

[Victor_Geyyer_2]

Thank you, Tony!

This is the brand new cluster: there were no 32bit indexes; I would assume it should not have any “updating” running.  Nothing interesting in the event logs before the index fails.

[JesusWept3]

Hmm i've seen something similar at a customer Don't suppose you have a bunch of errors being thrown in the System log do you?

[JesusWept3]

Specifically this error Event ID: 0 Task: Sharing Service An error occurred while dispatching a duplicated socket: this handle is now leaked in the process. ID: 4972 Source: System.ServiceModel.Activation.TcpWorkerProcess/29805701 Exception: System.TimeoutException: This request operation sent to <> did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

[Victor_Geyyer_2]

Thank you for the reply.

Not noticing any interesting messages in the system log.

4 min. before the index reported as failed last time it reported following informational message:

"Log Name:      System
Source:        Service Control Manager
Date:          1/24/2013 1:08:53 PM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MyServer.MyDomain.com
Description:
The Application Experience service entered the stopped state."

1 min before that I am noticing:

"Log Name:      System
Source:        Service Control Manager
Date:          1/24/2013 1:11:03 PM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MyServer.MyDomain.com
Description:
The WMI Performance Adapter service entered the running state."

And

"Log Name:      System
Source:        Service Control Manager
Date:          1/24/2013 1:11:03 PM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     MyServer.MyDomain.com
Description:
The WMI Performance Adapter service entered the stopped state."

None of these looks very "interesting" to me.

 

[Victor_Geyyer_2]

How about this extract from the dtrace around the time index gets failed: 566906 10:06:51.422 [6388] (EVIndexAdminService) <3616> EV-M {VelocityProcessHealthMonitor.CheckProcessPriority} VelocityProcessPriorityMonitor is performing it's periodic check of process priorities 566907 10:06:53.594 [5552] (EVIndexVolumesProcessor) EV-L {QueueController} Manager Queue Controller: Some outstanding Work Item waiting processing but not ready to start yet (e.g. overloaded). (return No weights, all the looping on the weights have been done and no candidate was found) 566912 10:06:58.594 [5552] (EVIndexVolumesProcessor) EV-L {QueueController} Manager Queue Controller: Some outstanding Work Item waiting processing but not ready to start yet (e.g. overloaded). (return No weights, all the looping on the weights have been done and no candidate was found) 566913 10:07:01.422 [6388] (EVIndexAdminService) <3616> EV-M {VelocityProcessHealthMonitor.CheckProcessPriority} VelocityProcessPriorityMonitor is performing it's periodic check of process priorities 566916 10:07:03.594 [5552] (EVIndexVolumesProcessor) EV-L {QueueController} Manager Queue Controller: Some outstanding Work Item waiting processing but not ready to start yet (e.g. overloaded). (return No weights, all the looping on the weights have been done and no candidate was found) 566925 10:07:08.594 [5552] (EVIndexVolumesProcessor) EV-L {QueueController} Manager Queue Controller: Some outstanding Work Item waiting processing but not ready to start yet (e.g. overloaded). (return No weights, all the looping on the weights have been done and no candidate was found) 566933 10:07:11.422 [6388] (EVIndexAdminService) <3616> EV-M {VelocityProcessHealthMonitor.CheckProcessPriority} VelocityProcessPriorityMonitor is performing it's periodic check of process priorities 566934 10:07:13.594 [5552] (EVIndexVolumesProcessor) EV-L {QueueController} Manager Queue Controller: Some outstanding Work Item waiting processing but not ready to start yet (e.g. overloaded). (return No weights, all the looping on the weights have been done and no candidate was found) 566937 10:07:16.048 [6388] (EVIndexAdminService) <3616> EV-L {ProcessManager.LeaseRenewal} Renewing the lease of the object EV Index Volumes Processor. Current state of lease: Active 566938 10:07:16.048 [6388] (EVIndexAdminService) <8064> EV-L {ProcessManager.LeaseRenewal} Renewing the lease of the object EV Index Query Server. Current state of lease: Active 566939 10:07:18.594 [5552] (EVIndexVolumesProcessor) EV-L {QueueController} Manager Queue Controller: Some outstanding Work Item waiting processing but not ready to start yet (e.g. overloaded). (return No weights, all the looping on the weights have been done and no candidate was found) 566948 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {IndexingWCFProxy`1} [62667ms] Called SearchCollectionStatus 566949 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {IndexingWCFProxy`1} SOAP FAULT: detail xml = Could not start the collection service for collection 1126AEF7E8493C346B70D777F8A052E09_1534.| 566950 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-H {IndexingWCFProxy`1} Exception: The exception [search-collection-service-could-not-start] was thrown. Info: Diag: Type:System.ServiceModel.FaultException`1[[System.String, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] ST: at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.ReThrowAsTypedFaultExceptionIfAppropriate(FaultException faultEx)| at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.Invoke[TResult](Expression`1 method)| at Symantec.EnterpriseVault.Indexing.Common.IndexingWCFProxy`1.Invoke[TResult](Expression`1 method) Inner:None 566951 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-H {VelocityProxy} Cannot get collection status for [1126AEF7E8493C346B70D777F8A052E09_1534]. The exception [search-collection-service-could-not-start] was thrown. 566952 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-H {SynchronisationWorkItem} Exception: Indexing engine exception Info:Critical error when getting the start up type. Diag: Type:Symantec.EnterpriseVault.Indexing.ContentSource.IndexingEngineException ST: at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityProxy.GetCollectionStatus(String CollectionName)| at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityIndexingEngine.GetIndexVolumeItemCount(IIndexVolumeIdentifier indexVolumeID)| at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.IndexVolumeOperations.SynchroniseIndexedItemsCount(IndexVolumeMetadata indexVolumeMetadata, IndexVolumeInternalStateTracker internalState)| at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.SynchronisationWorkItem.PrepareForProcessing() Inner:System.ServiceModel.FaultException`1[System.String]: The exception [search-collection-service-could-not-start] was thrown. (Fault Detail is equal to Could not start the collection service for collection 1126AEF7E8493C346B70D777F8A052E09_1534.| ). 566953 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-M {SynchronisationWorkItem} Error during the processing ErrorType=CriticalError Error Description=Symantec.EnterpriseVault.Indexing.ContentSource.IndexingEngineException: Indexing engine exception ---> System.ServiceModel.FaultException`1[System.String]: The exception [search-collection-service-could-not-start] was thrown. (Fault Detail is equal to Could not start the collection service for collection 1126AEF7E8493C346B70D777F8A052E09_1534.| ).| --- End of inner exception stack trace ---| at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityProxy.GetCollectionStatus(String CollectionName)| at Symantec.EnterpriseVault.Indexing.IndexingEngine.VelocityIndexingEngine.GetIndexVolumeItemCount(IIndexVolumeIdentifier indexVolumeID)| at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.IndexVolumeOperations.SynchroniseIndexedItemsCount(IndexVolumeMetadata indexVolumeMetadata, IndexVolumeInternalStateTracker internalState)| at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.SynchronisationWorkItem.PrepareForProcessing(). WorkItem 'XXXXXXXXXXXXX.MyDomain.com_1534' 566954 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {SynchronisationWorkItem} Stopped event signaled for 'XXXXXXXXXXXXX.MyDomain.com_1534' 566955 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {SynchronisationWorkItem} The processing of the work Item will carry on: Workitem 'XXXXXXXXXXXXX.MyDomain.com_1534' 566956 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {ActiveIndexVolumeAgentList} Processing completed for agent Index volume Agent For: Work Item XXXXXXXXXXXXX.MyDomain.com_1534 has reached a check point with a reason Error 566957 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {SynchronisationWorkItem} The Processing is finished with a status of 'CheckPointing'. WorkItem 'XXXXXXXXXXXXX.MyDomain.com_1534' 566958 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {IndexVolumeOperations} Synchronising IndexedItems count for [XXXXXXXXXXXXX.MyDomain.com_1534]. 566959 10:07:21.391 [5552] (EVIndexVolumesProcessor) EV-L {IndexingWCFProxy`1} Calling SearchCollectionStatus

[Rob_Wilcox1]

Have you talked to Symantec Support about this issue?

[Victor_Geyyer_2]

Yes. They had me rebuilding the index, which is going on since Friday and with this rate will be progressing to the next Friday.