β03-14-2013 07:10 AM
Hi
Upgrade of EV from 9.0.2 to 10.0.3 in microsoft cluster (active passive).
Upgrade completed succesfully.
Index administration task created with upgrade to all 32bit indexes.
Upgrade is not working (stays inactive), but it looks like system is having indexing problems, as events are full with the following:
at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Object COMInterface, Action act, String context, LoggingMode loggingMode, Boolean rethrow)
-----------------------------------------------------------------------------
Upgrade went smooth with no special issues, all steps performed by the book, including moving the IndexMetaData to shared location.
Solved! Go to Solution.
β04-15-2013 01:35 AM
PLEASE NOTE : The registry values below are only suitable to version 10.0.3
Finally, Issue resolved. We had to add them manually and re-register all dll's
Missing registry values in FIPS folder under [HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS].
Those were not present on the servers:
[HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:88,17,10,56,cb,cc,a6,02,6f,92,7b,29,93,93,0f,42,94,d7,\
49,48,00,fc,1c,a4,bf,d6,38,7a,9b,02,e8,1b,f6,10,0f,3a,70,a6,d6,eb,16,9e,ab,\
97,16,3e,d0,d6,1c,5c,88,47,b0,ac,28,01,1e,c0,99,be,37,05,17,60,aa,6b,cb,93,\
5b,7f,c8,f4,89,9a,d2,3c,78,34,11,a6,f2,07,bf,15,dc,37,5e,3e,cc,40,f4,b7,33,\
54,ca,62,4d,6e,28,f5,49,98,8d,45,32,7d,65,f9,b4,49,42,62,09,8c,61,37,c5,39,\
1a,0e,a8,28,f6,00,5e,00,44,75,7b,80,9d,a9,98,d3,90,58,40,1a,16,61,ff,c7,6d,\
ea
"ManagedDLLHash"=hex:6d,a2,58,c9,f0,d0,64,53,0f,01,9e,64,33,2c,6c,6a,3f,34,7e,\
87,13,bb,bf,76,33,43,bc,81,0e,b8,d3,ea,78,f5,86,d9,30,f8,7b,23,10,77,23,66,\
f4,ee,3c,c6,1b,de,e8,f0,e8,d6,51,3c,30,10,1b,30,c5,46,c6,ff,c7,85,88,b1,c0,\
4f,5a,90,af,a4,49,85,71,00,f1,1e,58,8b,f7,e5,f6,cd,92,ae,57,b1,6e,df,42,78,\
40,2f,1c,46,c4,8a,ef,71,bc,60,eb,98,f3,c5,e6,8c,58,e2,da,42,35,32,00,a6,93,\
e7,23,5b,47,c9,51,77,4e,b2,27,b0,67,9a,06,d8,e5,33,28,cf,8f,2d,33,66,a5,92
"InstallPath"="C:\\Program Files:\\Enterprise Vault\\x64"
Those were present but won't work properly without the ones aboveβ¦
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:f1,32,05,6b,91,44,9f,82,ad,33,9f,d8,d7,a3,f1,80,e8,fb,\
25,95,56,5b,fb,c8,3f,3d,43,56,9d,fa,91,d1,c8,20,98,01,fb,e2,d2,50,e0,fc,fd,\
2e,92,d3,95,70,4f,83,c4,29,18,bb,ce,84,33,fa,b3,4e,3b,5d,f5,e8,a6,4f,a8,22,\
9f,68,53,66,57,11,67,36,64,89,45,b8,06,11,48,b4,a5,ad,38,c1,c3,49,ac,e7,05,\
8b,b0,8a,26,f6,f7,f6,be,63,fe,ed,07,89,b7,0e,b9,8e,1b,5e,22,0c,2c,bb,31,b7,\
2e,87,62,9b,73,85,3d,74,c3,81,4b,fa,aa,5b,a0,86,fb,6d,4f,ce,c6,8a,4b,33,2f,\
c2
"ManagedDLLHash"=hex:3f,1f,46,d0,fc,d2,0a,a2,db,f8,d5,5c,57,9b,0d,43,ce,82,15,\
96,82,4f,71,2c,56,83,d9,e6,cf,3b,2f,56,2e,c9,d7,ba,ae,79,d9,cd,90,eb,a3,f5,\
cf,cb,6f,0d,e4,22,55,a6,b4,0a,c6,a8,60,92,a4,77,c3,f2,c5,c8,7c,c0,59,75,7c,\
36,63,b1,e5,97,3c,97,13,61,8e,33,8b,d3,54,cf,a1,7d,e9,7f,f3,c3,16,74,bb,58,\
33,d5,b4,d9,ec,ab,b8,d5,5c,89,5b,81,4c,3c,ef,fd,a4,80,74,4f,1f,79,29,33,bb,\
ba,2c,2c,7a,34,cb,67,d5,7b,a5,cc,db,e5,78,4d,36,fb,93,81,7a,37,11,79,89,51
"InstallPath"=" C:\\Program Files:\\Enterprise Vault"
added the registry keys manually, re-registered all dll's and restarted services.
β03-14-2013 01:51 PM
I would be tempted to contact Symantec Support on this issue, and discuss with them the internal reference you see in the event log entry. To be fair though, it does sound like 'something' cluster related.
β03-17-2013 07:08 AM
Just opened a case.
Will update with the results once we have any.
β03-17-2013 08:24 AM
β03-18-2013 02:28 AM
Hi
can't rebuild, synch or do any job on indexes, they all result with same errors like i attached.
also reinstall of binaries didn't solve the issue.
I have a feeling this will end up in a Symantec hotfix in the end, as I was working on this with Symantec yesterday for 4 hours and still have no idea on how to solve it.
β03-18-2013 03:18 AM
β03-18-2013 03:52 AM
I deleted subtask and tasks from the ev DB's, as after you create the task, you cannot stop it... you get an error saying : the following subtask could not be stopped [archive name] reason : 0x8013150b, followed with event 40966 Exception: Failed to get subtask for subtask ID 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver
Major fun
β04-04-2013 10:54 AM
I was wondering if you resolved this problem? I have performed the same version upgrade as you but this is a single server and it was from a 2003 to 2008 R2 server. I have a similar issue.
My rebuild and upgrade index tasks never start and just stay inactive, however I can perform synchronisation tasks. It appears as if new indexing is working ok.
β04-04-2013 11:05 AM
β04-04-2013 11:13 AM
I take it FileReRegister.bat doesnt help any?
β04-04-2013 12:18 PM
FileReRegister.bat was only suggested today, so it will be tested on Sunday when we are back to work.
Hoping that this will solve the issue...
Will update.
β04-05-2013 04:04 AM
Since I have the opportunity to work on this until the end of today. I tried the FileReRegister.bat, deleted the subtasks from the database, the task from the VAC, rebooted the server.
After trying a rebuild/upgrade again, the same problem persists with the same errors in the event log.
Although I originally thought new content was being indexed, this was checked against the journal archive. 706 items show in the 64bit index, but this has now ceased to increase despite the constant archiving of journal items.
Following the user archive last night, new 64bit indexes were opened but they contain no indexed content and a check from and end user perspecitve shows I cannot search for any of these newly archived items.
I don't know if this is related, but when I try to stop the index subtasks in the GUI I get a message stating thay there has been a logon failure and to check the account the task runs under has 'logon as a service' rights. I am logged on as the EV service account and it has all these and full admin rights already.
For our environment we don't have a massive amount of archived emails (approx 15 million), so I am considering just wiping all the index data and starting again, however the fact that new content now doesn't appear to be indexing doesn't fill me with confidence. At least I have some index content now, even if it is 'closed legacy 32bit'.
β04-05-2013 05:24 AM
I would not delete all the index data unless it were a last resort.
You could give this a try first.
Article:HOWTO59060 | | | Created: 2011-08-26 | | | Updated: 2012-09-13 | | | Article URL http://www.symantec.com/docs/HOWTO59060 |
β04-05-2013 07:04 AM
Thanks but that didn't work. It seems it is now worse, or better depending upon which way you look at it. EV has decided that at small random amount of 64bit indexes are now officially in a failed state. However, trying a rebuild task just results in the same issue of it never starting.
We have several mailboxes that belong to users no longer with the company. I would like to test a complete index deletion and build from scratch for one of these users. What would be the best way of achieving this?
β04-05-2013 07:15 AM
Did you see these entries?
Event ID: 41372
Indexing Service has started the synchronization of index volume metadata between the indexing engine repository and the Directory database.
Event ID: 41377
Indexing Service has completed the synchronization of index volume metadata between the indexing engine repository and the Directory database.
Total index volume(s) whose metadata was created: x
Total index volume(s) whose location was updated: y
β04-05-2013 07:23 AM
Yes those are present
β04-05-2013 10:25 AM
James,
Going back to that logon failure, I believe that's referring to the account that the indexing service itself is running under.
I know that in the past, the password for EV Services' accounts have gone corrupt for some odd reason, and the fix has been to just go into the properties of the service itself (via Services.msc) and re-enter the account's password from there.
May as well verify that it's using the EV service account as well, although I doubt that would've been changed.
May as well check that the EV service account does have 'Log on as a Service' rights, although the EV setup does grant this right to the service account right from the start. Always worth covering that base, however.
β04-07-2013 01:50 AM
FileReRegister didn't solve the issue for me too I'm afraid, on both nodes.
however - although Synch is not working, I can stop and delete synch task, upgrade task - cannot be either stopped or deleted.
Pass my files to Symantec.
β04-08-2013 03:19 AM
The account is used to start all EV services, so it wouldn't make any sense if it didn't have those rights. However, just to be sure I have re-entered the details against the index service and checked that the account has 'log on as a service' rights - which it does.
After leaving it over the weekend, I have now got:
300 32 bit indexes
127 64bit
87 failed
Clearly it has opened 127 64bit indexes total but there is nothing at all indexed in them. The event log continues with its continous index errors.
The latest is event 41329 (frenzyerror). The errors in general leading up to this seem to indicate that the indexer simply cannot read the items from the vault to index them. All EV services use the same service account and all other tasks bar indexing work fine. It doesn't look like this is a basic account rights issue.
β04-08-2013 03:22 AM
OK
Just got to another customer where I upgraded to 10.0.3
64bit index not working at all.
upgrade of indexes not working.
all items archives since upgrade not indexed.
I have the feeling that a microsoft update is causing it.
Want to compare windows updates and installed software from my 2 customers and your server?