cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Index errors after upgrade to 10.0.3

Sarah_Seftel1
Level 6
Partner Accredited

Hi

Upgrade of EV from 9.0.2 to 10.0.3 in microsoft cluster (active passive).
Upgrade completed succesfully.

Index administration task created with upgrade to all 32bit indexes.

Upgrade is not working (stays inactive), but it looks like system is having indexing problems, as events are full with the following:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Date:          14/03/2013 13:40:13
Event ID:      40966
Task Category: Index Admin Service 
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Server
Description:
A program fault has raised an exception.
 
Exception: Failed to get subtask for subtask ID 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver
Diagnostic: 
Type: System.ServiceModel.FaultException`1[[Symantec.EnterpriseVault.Indexing.Common.CommunicationFault, EVSharedManagedInterfaces, Version=10.0.3.0, Culture=neutral, PublicKeyToken=26c5e2ccf2b9267c]]
Reference: An error occurred while invoking Void <StopSubTask>b__f().
Error Context : Stopping sub task for TaskEntryId 17DC08710A3B705409EFC2C9200093CFE1013a00vaultserver and SubTaskEntryId 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver
 
Command Line: "E:\Enterprise Vault\EVIndexAdminService.exe" -EntryID:1EC5115667B050B40B71EFC0504B51FD71710000vaultserver
Application Domain: EVIndexAdminService.exe
Process Id: 8836
Thread Id: 18288
Stack Trace: 
Server stack trace: 
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
 
Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.WebService.ITaskProcessorWebService.StopProcessingSubtask(String taskId, String subtaskId, StopReason reasonForStop)
   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminImpl.Stop64BitSubTasks(String taskEntryId, String subTaskEntryId, StopReason reasonForStop)

   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Object COMInterface, Action act, String context, LoggingMode loggingMode, Boolean rethrow)

-----------------------------------------------------------------------------

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Date:          14/03/2013 13:44:02
Event ID:      7182
Task Category: Web Application (WP)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Server
Description:
Index Search failed: Unspecified error  (0x80004005)  
Index: 1EB6DC7DA00ED2447BA21981FFB2D908D1110000vaultserver/VolumeSet:866 
Internal reference: SRCI 
------------------------------------------------------------------------------
 

Upgrade went smooth with no special issues, all steps performed by the book, including moving the IndexMetaData to shared location.

 
Any ideas?
 
Also attached dtrace of the following:
EVIndexAdminService
EVIndexVolumesProcessor 
IndexBroker
IndexServer  
 
Thanks,
 
Sarah.

 

1 ACCEPTED SOLUTION

Accepted Solutions

Sarah_Seftel1
Level 6
Partner Accredited

PLEASE NOTE : The registry values below are only suitable to version 10.0.3

Finally, Issue resolved. We had to add them manually and re-register all dll's

Missing registry values in FIPS folder under [HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS].

Those were not present on the servers:

[HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:88,17,10,56,cb,cc,a6,02,6f,92,7b,29,93,93,0f,42,94,d7,\
  49,48,00,fc,1c,a4,bf,d6,38,7a,9b,02,e8,1b,f6,10,0f,3a,70,a6,d6,eb,16,9e,ab,\
  97,16,3e,d0,d6,1c,5c,88,47,b0,ac,28,01,1e,c0,99,be,37,05,17,60,aa,6b,cb,93,\
  5b,7f,c8,f4,89,9a,d2,3c,78,34,11,a6,f2,07,bf,15,dc,37,5e,3e,cc,40,f4,b7,33,\
  54,ca,62,4d,6e,28,f5,49,98,8d,45,32,7d,65,f9,b4,49,42,62,09,8c,61,37,c5,39,\
  1a,0e,a8,28,f6,00,5e,00,44,75,7b,80,9d,a9,98,d3,90,58,40,1a,16,61,ff,c7,6d,\
  ea
"ManagedDLLHash"=hex:6d,a2,58,c9,f0,d0,64,53,0f,01,9e,64,33,2c,6c,6a,3f,34,7e,\
  87,13,bb,bf,76,33,43,bc,81,0e,b8,d3,ea,78,f5,86,d9,30,f8,7b,23,10,77,23,66,\
  f4,ee,3c,c6,1b,de,e8,f0,e8,d6,51,3c,30,10,1b,30,c5,46,c6,ff,c7,85,88,b1,c0,\
  4f,5a,90,af,a4,49,85,71,00,f1,1e,58,8b,f7,e5,f6,cd,92,ae,57,b1,6e,df,42,78,\
  40,2f,1c,46,c4,8a,ef,71,bc,60,eb,98,f3,c5,e6,8c,58,e2,da,42,35,32,00,a6,93,\
  e7,23,5b,47,c9,51,77,4e,b2,27,b0,67,9a,06,d8,e5,33,28,cf,8f,2d,33,66,a5,92
"InstallPath"="C:\\Program Files:\\Enterprise Vault\\x64"

 Those were present but won't work properly without the ones above…

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:f1,32,05,6b,91,44,9f,82,ad,33,9f,d8,d7,a3,f1,80,e8,fb,\
  25,95,56,5b,fb,c8,3f,3d,43,56,9d,fa,91,d1,c8,20,98,01,fb,e2,d2,50,e0,fc,fd,\
  2e,92,d3,95,70,4f,83,c4,29,18,bb,ce,84,33,fa,b3,4e,3b,5d,f5,e8,a6,4f,a8,22,\
  9f,68,53,66,57,11,67,36,64,89,45,b8,06,11,48,b4,a5,ad,38,c1,c3,49,ac,e7,05,\
  8b,b0,8a,26,f6,f7,f6,be,63,fe,ed,07,89,b7,0e,b9,8e,1b,5e,22,0c,2c,bb,31,b7,\
  2e,87,62,9b,73,85,3d,74,c3,81,4b,fa,aa,5b,a0,86,fb,6d,4f,ce,c6,8a,4b,33,2f,\
  c2
"ManagedDLLHash"=hex:3f,1f,46,d0,fc,d2,0a,a2,db,f8,d5,5c,57,9b,0d,43,ce,82,15,\
  96,82,4f,71,2c,56,83,d9,e6,cf,3b,2f,56,2e,c9,d7,ba,ae,79,d9,cd,90,eb,a3,f5,\
  cf,cb,6f,0d,e4,22,55,a6,b4,0a,c6,a8,60,92,a4,77,c3,f2,c5,c8,7c,c0,59,75,7c,\
  36,63,b1,e5,97,3c,97,13,61,8e,33,8b,d3,54,cf,a1,7d,e9,7f,f3,c3,16,74,bb,58,\
  33,d5,b4,d9,ec,ab,b8,d5,5c,89,5b,81,4c,3c,ef,fd,a4,80,74,4f,1f,79,29,33,bb,\
  ba,2c,2c,7a,34,cb,67,d5,7b,a5,cc,db,e5,78,4d,36,fb,93,81,7a,37,11,79,89,51
"InstallPath"=" C:\\Program Files:\\Enterprise Vault"

added the registry keys manually, re-registered all dll's and restarted services.

View solution in original post

46 REPLIES 46

Rob_Wilcox1
Level 6
Partner

I would be tempted to contact Symantec Support on this issue, and discuss with them the internal reference you see in the event log entry.  To be fair though, it does sound like 'something' cluster related.

Working for cloudficient.com

Sarah_Seftel1
Level 6
Partner Accredited

Just opened a case.

Will update with the results once we have any.

JesusWept3
Level 6
Partner Accredited Certified
Delete and recreate the index rebuild tasks would be the first easiest option
https://www.linkedin.com/in/alex-allen-turl-07370146

Sarah_Seftel1
Level 6
Partner Accredited

Hi

can't rebuild, synch or do any job on indexes, they all result with same errors like i attached.

also reinstall of binaries didn't solve the issue.

I have a feeling this will end up in a Symantec hotfix in the end, as I was working on this with Symantec yesterday for 4 hours and still have no idea on how to solve it.

JesusWept3
Level 6
Partner Accredited Certified
So just so I understand, did they delete the subtasks from the SubTask table and the index tasks from the vac?
https://www.linkedin.com/in/alex-allen-turl-07370146

Sarah_Seftel1
Level 6
Partner Accredited

I deleted subtask and tasks from the ev DB's, as after you create the task, you cannot stop it... you get an error saying : the following subtask could not be stopped [archive name] reason : 0x8013150b, followed with event 40966 Exception: Failed to get subtask for subtask ID 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver

 

Major fun smiley

James_Drake
Level 4
Partner

I was wondering if you resolved this problem? I have performed the same version upgrade as you but this is a single server and it was from a 2003 to 2008 R2 server. I have a similar issue.

My rebuild and upgrade index tasks never start and just stay inactive, however I can perform synchronisation tasks. It appears as if new indexing is working ok.

Sarah_Seftel1
Level 6
Partner Accredited
We figured out that dll are not registered correctly on both nodes. But reinstall of ev media don't seem to be helpful. It is still being tested and we are still working on the issue. Will update when we will have a solution.

JesusWept3
Level 6
Partner Accredited Certified

I take it FileReRegister.bat doesnt help any?

https://www.linkedin.com/in/alex-allen-turl-07370146

Sarah_Seftel1
Level 6
Partner Accredited

FileReRegister.bat was only suggested today, so it will be tested on Sunday when we are back to work.

Hoping that this will solve the issue...

 

Will update.

James_Drake
Level 4
Partner

Since I have the opportunity to work on this until the end of today. I tried the FileReRegister.bat, deleted the subtasks from the database, the task from the VAC, rebooted the server.

After trying a rebuild/upgrade again, the same problem persists with the same errors in the event log.

Although I originally thought new content was being indexed, this was checked against the journal archive. 706 items show in the 64bit index, but this has now ceased to increase despite the constant archiving of journal items.

Following the user archive last night, new 64bit indexes were opened but they contain no indexed content and a check from and end user perspecitve shows I cannot search for any of these newly archived items.

I don't know if this is related, but when I try to stop the index subtasks in the GUI I get a message stating thay there has been a logon failure and to check the account the task runs under has 'logon as a service' rights. I am logged on as the EV service account and it has all these and full admin rights already.

For our environment we don't have a massive amount of archived emails (approx 15 million), so I am considering just wiping all the index data and starting again, however the fact that new content now doesn't appear to be indexing doesn't fill me with confidence. At least I have some index content now, even if it is 'closed legacy 32bit'.

 

 

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

I would not delete all the index data unless it were a last resort.

You could give this a try first.

How to reset the 64-bit Indexing Engine following corruption of core data files.

Article:HOWTO59060  |  Created: 2011-08-26  |  Updated: 2012-09-13  |  Article URL http://www.symantec.com/docs/HOWTO59060

 

James_Drake
Level 4
Partner

Thanks but that didn't work. It seems it is now worse, or better depending upon which way you look at it. EV has decided that at small random amount of 64bit indexes are now officially in a failed state. However, trying a rebuild task just results in the same issue of it never starting.

We have several mailboxes that belong to users no longer with the company. I would like to test a complete index deletion and build from scratch for one of these users. What would be the best way of achieving this?

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified

Did you see these entries?

Event ID:      41372

Indexing Service has started the synchronization of index volume metadata between the indexing engine repository and the Directory database. 

Event ID:      41377

Indexing Service has completed the synchronization of index volume metadata between the indexing engine repository and the Directory database.
Total index volume(s) whose metadata was created: x
Total index volume(s) whose location was updated: y

 

James_Drake
Level 4
Partner

Yes those are present

Old_Man_Jensen
Level 4
Partner Accredited Certified

James,

Going back to that logon failure, I believe that's referring to the account that the indexing service itself is running under.

I know that in the past, the password for EV Services' accounts have gone corrupt for some odd reason, and the fix has been to just go into the properties of the service itself (via Services.msc) and re-enter the account's password from there.

May as well verify that it's using the EV service account as well, although I doubt that would've been changed.

May as well check that the EV service account does have 'Log on as a Service' rights, although the EV setup does grant this right to the service account right from the start.  Always worth covering that base, however.

Sarah_Seftel1
Level 6
Partner Accredited

FileReRegister didn't solve the issue for me too I'm afraid, on both nodes.

however - although Synch is not working, I can stop and delete synch task, upgrade task - cannot be either stopped or deleted.

Pass my files to Symantec.

 

James_Drake
Level 4
Partner

The account is used to start all EV services, so it wouldn't make any sense if it didn't have those rights. However, just to be sure I have re-entered the details against the index service and checked that the account has 'log on as a service' rights - which it does.

After leaving it over the weekend, I have now got:

 

300 32 bit indexes

127 64bit

87 failed

 

Clearly it has opened 127 64bit indexes total but there is nothing at all indexed in them. The event log continues with its continous index errors.

The latest is event 41329 (frenzyerror). The errors in general leading up to this seem to indicate that the indexer simply cannot read the items from the vault to index them. All EV services use the same service account and all other tasks bar indexing work fine. It doesn't look like this is a basic account rights issue.

Sarah_Seftel1
Level 6
Partner Accredited

OK

Just got to another customer where I upgraded to 10.0.3

64bit index not working at all.

upgrade of indexes not working.

all items archives since upgrade not indexed.

 

I have the feeling that a microsoft update is causing it.

Want to compare windows updates and installed software from my 2 customers and your server?