Index errors after upgrade to 10.0.3

[Sarah_Seftel1]

Hi

Upgrade of EV from 9.0.2 to 10.0.3 in microsoft cluster (active passive).
Upgrade completed succesfully.

Index administration task created with upgrade to all 32bit indexes.

Upgrade is not working (stays inactive), but it looks like system is having indexing problems, as events are full with the following:

Log Name:      Symantec Enterprise Vault

Source:        Enterprise Vault 

Date:          14/03/2013 13:40:13

Event ID:      40966

Task Category: Index Admin Service 

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      Server

Description:

A program fault has raised an exception.

Exception: Failed to get subtask for subtask ID 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver

Diagnostic: 

Type: System.ServiceModel.FaultException`1[[Symantec.EnterpriseVault.Indexing.Common.CommunicationFault, EVSharedManagedInterfaces, Version=10.0.3.0, Culture=neutral, PublicKeyToken=26c5e2ccf2b9267c]]

Reference: An error occurred while invoking Void <StopSubTask>b__f().

Error Context : Stopping sub task for TaskEntryId 17DC08710A3B705409EFC2C9200093CFE1013a00vaultserver and SubTaskEntryId 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver

Command Line: "E:\Enterprise Vault\EVIndexAdminService.exe" -EntryID:1EC5115667B050B40B71EFC0504B51FD71710000vaultserver

Application Domain: EVIndexAdminService.exe

Process Id: 8836

Thread Id: 18288

Stack Trace: 

Server stack trace: 

   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.WebService.ITaskProcessorWebService.StopProcessingSubtask(String taskId, String subtaskId, StopReason reasonForStop)

   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminImpl.Stop64BitSubTasks(String taskEntryId, String subTaskEntryId, StopReason reasonForStop)

   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Object COMInterface, Action act, String context, LoggingMode loggingMode, Boolean rethrow)

-----------------------------------------------------------------------------

Log Name:      Symantec Enterprise Vault

Source:        Enterprise Vault 

Date:          14/03/2013 13:44:02

Event ID:      7182

Task Category: Web Application (WP)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      Server

Description:

Index Search failed: Unspecified error  (0x80004005)  

Index: 1EB6DC7DA00ED2447BA21981FFB2D908D1110000vaultserver/VolumeSet:866 

Internal reference: SRCI 

------------------------------------------------------------------------------

Upgrade went smooth with no special issues, all steps performed by the book, including moving the IndexMetaData to shared location.

Any ideas?

Also attached dtrace of the following:

EVIndexAdminService

EVIndexVolumesProcessor 

IndexBroker

IndexServer  

Thanks,

Sarah.

[Saswata_Basu]

I had recently Migrated and upgraded 3 sites from 9.0.3 32bit - 10.0.3 64bit on Windows 2008 R2 recently but never found any issues with Indexing. My servers were all non clustered servers hence can't conclude if this is a issue with Clustered installations.

Were there Indexing errors on 9.0.3 prior to upgrading to 10.0.3? Did you check the event log on old server?

[James_Drake]

My server isn't clustered, its about as simple as you can get. Did you have all available Windows updates installed at the time?

The indexing was working fine before the upgrade. The 32bit indexes continue to be searchable, its just they can't be upgraded and no new content is indexed.

Since this is working in our test lab, I am actually fully updating the lab server with Windows updates now, although this is a slow process because the lab is on old hardware. My test server didn't even have SP1 installed, it was the original release of Windows 2008 R2 with a few hotfixes pre SP1.

Basically, if I can break the lab server, at least I have something 'obvious' to work from.

[Sarah_Seftel1]

My server is clustered, but it has nothing to do with that, as issue occures on both nodes.

all the 32 bit index worked fine before the upgrade.

the 32 bit also working after the upgrade.

only 64bit upgrade items and new archiving index is not working.

There were no issues with the server archiving & indexing before the upgrade.

[James_Drake]

I performed a dtrace against id 32 and 36. The extract below may be a potential indicator to the problem:

If anyone has any ideas what to look for I would welcome the suggestions.

433 11:30:09.859  [5796] (EVIndexVolumesProcessor) <Status Checker Thread for 1C72CE853B0CF3C409CD7FB12BB0C6B4D1110000evserver_415:10164> EV-M {StatusChecker} No Additions acknowledgments received. Will check again in 10 seconds...
434 11:30:09.865  [5796] (EVIndexVolumesProcessor) <Fetcher Thread for 1B5447B888E723A49B5FAAC1E3B5A6BC41110000evserver_471:7512> EV-M {EVAdditionDataReaderFactory} Item could not be fetched. ISN=[138340], SSID=[201304045096491~201302030628260000~Z~B0E21A20FA23192ECC0898DDCF95D071] Error number: 0xC0041923. Going to retry after 1000 milliseconds...
435 11:30:09.876  [5796] (EVIndexVolumesProcessor) <10016> EV-M {VelocityIndex}  Did not retrieve any Acknowledgements
436 11:30:09.894  [5796] (EVIndexVolumesProcessor) <Fetcher Thread for 1EF10105583625648B16AF07EEFFC55281110000evserver_487:10172> EV-H {ItemAdditionsAllocator} Exception: Error encountered whilst trying to fetch item Sequence Number '42173' Item Identifier '201304055183215~201302032255420000~Z~B0FBFEB5593E6CBC9A6F8395500A6241' Info:Could not determine if there are additions to do Diag: Type:Symantec.EnterpriseVault.Indexing.ContentSource.IndexableItemException ST:   at Symantec.EnterpriseVault.Indexing.ContentSource.EVAdditionDataReaderFactory.GetItem(AutoReleaseComObject`1 archiveCrawler, UInt64 nextItemSequenceNumber, UInt64 maxItemSequenceNumber, UInt32 timeoutInSeconds, UInt32 maxRetries)|   at Symantec.EnterpriseVault.Indexing.ContentSource.EVAdditionDataReaderFactory.GetInstance(UInt64 nextItemSequenceNumber, UInt64 maxItemSequenceNumber, UInt32 timeoutInSeconds, UInt32 maxRetries)|   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.ItemAdditionFetcher.GetItem(UInt64 nextItemActionID, UInt64 maxItemID)|   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.ItemAdditionFetcher.GetNextItem(UInt64 nextItemID, UInt64 maxItemID)|   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.ItemAdditionsAllocator.get_HasSomeWorkToDo() Inner:None
437 11:30:09.906  [5796] (EVIndexVolumesProcessor) <Fetcher Thread for 1EF10105583625648B16AF07EEFFC55281110000evserver_487:10172> EV-M {EVAdditionDataReaderFactory} Item could not be fetched. ISN=[42173], SSID=[201304055183215~201302032255420000~Z~B0FBFEB5593E6CBC9A6F8395500A6241] Error number: 0xC0041923. Going to retry after 1000 milliseconds...
438 11:30:09.906  [5796] (EVIndexVolumesProcessor) <Status Checker Thread for 1B5447B888E723A49B5FAAC1E3B5A6BC41110000evserver_471:6120> EV-M {StatusChecker} No Additions acknowledgments received. Will check again in 10 seconds...
439 11:30:09.930  [5796] (EVIndexVolumesProcessor) <10016> EV-M {VelocityIndex}  Did not retrieve any Acknowledgements
 

[Sarah_Seftel1 · Accepted solution]

PLEASE NOTE : The registry values below are only suitable to version 10.0.3

Finally, Issue resolved. We had to add them manually and re-register all dll's

Missing registry values in FIPS folder under [HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS].

Those were not present on the servers:

[HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:88,17,10,56,cb,cc,a6,02,6f,92,7b,29,93,93,0f,42,94,d7,\
  49,48,00,fc,1c,a4,bf,d6,38,7a,9b,02,e8,1b,f6,10,0f,3a,70,a6,d6,eb,16,9e,ab,\
  97,16,3e,d0,d6,1c,5c,88,47,b0,ac,28,01,1e,c0,99,be,37,05,17,60,aa,6b,cb,93,\
  5b,7f,c8,f4,89,9a,d2,3c,78,34,11,a6,f2,07,bf,15,dc,37,5e,3e,cc,40,f4,b7,33,\
  54,ca,62,4d,6e,28,f5,49,98,8d,45,32,7d,65,f9,b4,49,42,62,09,8c,61,37,c5,39,\
  1a,0e,a8,28,f6,00,5e,00,44,75,7b,80,9d,a9,98,d3,90,58,40,1a,16,61,ff,c7,6d,\
  ea
"ManagedDLLHash"=hex:6d,a2,58,c9,f0,d0,64,53,0f,01,9e,64,33,2c,6c,6a,3f,34,7e,\
  87,13,bb,bf,76,33,43,bc,81,0e,b8,d3,ea,78,f5,86,d9,30,f8,7b,23,10,77,23,66,\
  f4,ee,3c,c6,1b,de,e8,f0,e8,d6,51,3c,30,10,1b,30,c5,46,c6,ff,c7,85,88,b1,c0,\
  4f,5a,90,af,a4,49,85,71,00,f1,1e,58,8b,f7,e5,f6,cd,92,ae,57,b1,6e,df,42,78,\
  40,2f,1c,46,c4,8a,ef,71,bc,60,eb,98,f3,c5,e6,8c,58,e2,da,42,35,32,00,a6,93,\
  e7,23,5b,47,c9,51,77,4e,b2,27,b0,67,9a,06,d8,e5,33,28,cf,8f,2d,33,66,a5,92
"InstallPath"="C:\\Program Files:\\Enterprise Vault\\x64"

 Those were present but won't work properly without the ones above…

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:f1,32,05,6b,91,44,9f,82,ad,33,9f,d8,d7,a3,f1,80,e8,fb,\
  25,95,56,5b,fb,c8,3f,3d,43,56,9d,fa,91,d1,c8,20,98,01,fb,e2,d2,50,e0,fc,fd,\
  2e,92,d3,95,70,4f,83,c4,29,18,bb,ce,84,33,fa,b3,4e,3b,5d,f5,e8,a6,4f,a8,22,\
  9f,68,53,66,57,11,67,36,64,89,45,b8,06,11,48,b4,a5,ad,38,c1,c3,49,ac,e7,05,\
  8b,b0,8a,26,f6,f7,f6,be,63,fe,ed,07,89,b7,0e,b9,8e,1b,5e,22,0c,2c,bb,31,b7,\
  2e,87,62,9b,73,85,3d,74,c3,81,4b,fa,aa,5b,a0,86,fb,6d,4f,ce,c6,8a,4b,33,2f,\
  c2
"ManagedDLLHash"=hex:3f,1f,46,d0,fc,d2,0a,a2,db,f8,d5,5c,57,9b,0d,43,ce,82,15,\
  96,82,4f,71,2c,56,83,d9,e6,cf,3b,2f,56,2e,c9,d7,ba,ae,79,d9,cd,90,eb,a3,f5,\
  cf,cb,6f,0d,e4,22,55,a6,b4,0a,c6,a8,60,92,a4,77,c3,f2,c5,c8,7c,c0,59,75,7c,\
  36,63,b1,e5,97,3c,97,13,61,8e,33,8b,d3,54,cf,a1,7d,e9,7f,f3,c3,16,74,bb,58,\
  33,d5,b4,d9,ec,ab,b8,d5,5c,89,5b,81,4c,3c,ef,fd,a4,80,74,4f,1f,79,29,33,bb,\
  ba,2c,2c,7a,34,cb,67,d5,7b,a5,cc,db,e5,78,4d,36,fb,93,81,7a,37,11,79,89,51
"InstallPath"=" C:\\Program Files:\\Enterprise Vault"

added the registry keys manually, re-registered all dll's and restarted services.

[James_Drake]

I have tried the same fix for my customer and it has also resolved the issue. Clearly there is some common area that means this key isn't created for some installations, but at least its easy to fix and is now something I will be making a standard check aganist my future installations.

[Sarah_Seftel1]

Happy to know it helped!

I will notify Symantec, so they will know it's got nothing to do with my cluster...