Failed Index Volume

[EVRocks]

We have EV 10.0.3 for Exchange 2007 and recently we ended up with Indexing issue on only Journal mailbox. We cannot search on Journal mailbox from December 2012. Going to Indexing on the EV console we can see 1 failed index. Indexing and searching is fine for all other users.

We recently migrated from EV 8 to 10.0.1 and this is where after a while one of he Journal indexing volume (64-bit) failed and this is the only 64 bit indexing for Journal archive.

We have tried various fixes including:

verifying and Synchronise.

Some of the event logs suggested to upgrade to EV 10.0.3 and we done the upgrade but not luck with the fix.

Symantec support suggested to use EVSVR to locate the missing items. then we applied EVSVR in report mode and then in repair mode it identified the missing items and deleted them from the indexing.

then it was suggested to rebuild the indexes (only for the failed volume) and we done that and completed the rebuilding. the failed index on the Indexing went a way for a short time then restarting the index server it brought back the failed status and showing error id 28998 and 41329 (Error Type: FrenzyErrorDetected).

I am not sure what else we need to do as I believe Symantec is running out of options here.

Any suggestions or ideas please.

thanks

[TonySterling]

From the sounds of things it seems like there might be additional missing items from the journal archive.

Is that what the events indicate?

[Arjun_Shelke]

Yes to add on to this, Error type indicates more missing/corrupt items. If you want quick way to get index stable, then you can use poison pill setting to ignore the failed items.

[EVRocks]

Thank you Guys for your responses.

I did try widening the range but still the same errors and still one index of the Journal is still failing.

As for the poison pill setting... what will actually happened to these missing items. will this change will ignore the failed items means these items will not be indexed and they will not come up when search applied in Journal archive.

Am i right to think that the index rebuilding on that specific parition will remove those items from the sql databases.

Regards,

[JesusWept3]

Well it really depends what the frenzy failed on and what the event is.
And it also depends on what you did with the EVSVR repair, did it remove the database entries? because indexing won't remove anything from SQL, only the EVSVR will

So what would really help is

1. Paste the Event ID that caused the frenzy
2. Tell us what options you used for the EVSVR repair
3. If you can, attach the Index Rebuild report file for that user

[Rob_Wilcox1]

Fwiw if you use the poison pill setting then the items won't be inthe iindex and won't be used for searching. If you rebuild then it might be that the archive has the items and indexing can handle them this time, or they could be items which indexing engine still has problems with. Too many of them in succession and the rebuild will fail... Unless you use the poison pill settings.

[EVRocks]

Guys, I have now the top Symantec experts looking into this indexing issue and they are struggling to resolve this issue. apparently this is now a known issue with migrating from EV 8 to 10 when indexing house 32 and 64 bit platforms. We have tried so many fixes and now I am kind of losing the track of what we are doing. but it seems that we are close to fixing this problem. we identified the missing items. running EVSVR a few times and no change. We are now started working straight from the sql tables. While having this issue moving the location of indexing to another drive has complicated the troubleshooting. - the event that I get now is 28998 - see below - please see the attached report for the evsvr and the rebuilding index task Log Name: Symantec Enterprise Vault Source: Enterprise Vault Date: 25/03/2013 14:58:04 Event ID: 28998 Task Category: Storage Crawler Level: Error Keywords: Classic User: N/A Computer: APPSARC001FP.iprod.local Description: Retrieval of a saveset failed. Reason: The system cannot find the path specified. (0x80070003) Saveset Id: 201212014357667~201212010944210000~Z~615F24B6D2F275A8ED10BD038F25B951 Vault Id: 13694F6308BCD4D4886C68EFFED698AD81110000evault Index Sequence Number: 24466150 Internal Reference: Fetching Saveset for indexing For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=10.0.3.0&build=10.0.3.1090&error=V-437-28998 Event Xml: 28998 2 113 0x80000000000000 766555 Symantec Enterprise Vault APPSARC001FP.iprod.local The system cannot find the path specified. (0x80070003) 201212014357667~201212010944210000~Z~615F24B6D2F275A8ED10BD038F25B951 13694F6308BCD4D4886C68EFFED698AD81110000evault 24466150 Fetching Saveset for indexing thank you all

[Rob_Wilcox1]

This might be the issue that you are referring to in relation to 'known issue':

http://thingsilearnedtoday.net/2013/03/21/enterprise-vault-10-0-3-cumulative-hotfix-1-release/

[EVRocks · Accepted solution]

There were two subfolders that appear to be missing in the Storage area for Enterprise Vault.

EV pointing to missing DVS files in these locations that were attempting to be indexed. The Index Service was hanging on searching for these items causing the JournalArchive table to build up with items ready to be committed to index.

RESOLUTION:

Since there were not any backups to restore the DVS Files back to this folder, the pointers in the Vault Store database were removed using the DeleteSingleSaveset procedure.  Once these pointers were removed, the Journal Index Volume started to update correctly and cleared the items waiting to be indexed from the JournalArchive table of the Vault Store.

it's not known how and why these items gone missing completley despite of having backing up EV every night.

It took over three months to completly fix this issue. I hope this will not happen to anyone elae during the EV migration to 10.0.