250 Failed Indexes - Best way to fix?

Hi All,

I have an enterprise vault 6 installation (yes it needs upgrading) that has roughly 250 indexes reported as failed. When accessing the vault explorer these mailboxes report: 'failed to perform the search'.

What is the best way to fix these indexes, rebuild?

If so is there a way to initiate a full rebuild of multiple indexes?

To the best of my knowledge they have been broken for a while so backups would also contain the broken index files as well i guess.

Any advice gratefully received

Thanks

G.

1 Solution

Accepted Solutions
Accepted Solution!

GJP, IndexVolumeReplay If

GJP,

IndexVolumeReplay
If your users report receiving a message that the indexes are not up to date or that they are marked as Failed you should take action to update the index using the IndexVolumeReplay.exe. Remember, just because an index volume is marked as Failed it does not necessarily mean corruption.

IndexVolumeReplay.exe for EV 6 and EV 7 can be found in the EV installation directory.
(i.e. C:\Program Files\Enterprise Vault\) To launch it just double click the IndexVolumeReplay.exe. You can update a failed index volume by right clicking the volume that is marked as failed and choose Update Index Volume.

IndexVolumeReplay.jpeg

In 2007 and above you can access IndexVolumeReplay from the Vault Admin Console by right clicking the Archives container.
Remember to always do an Update first and not Rebuild, the Update just processes the items that are missing, whereas a Rebuild will delete the entire index volume and start from the beginning. An Update is quicker and takes up fewer resources.

If it is necessary, you can Rebuild all the associated index volumes of an archive.

Right click the target archive and choose Properties. The Advanced tab has the Rebuild Index button that will rebuild all the index volumes at once.

You can additionally manage index volumes by viewing the properties of the individual archive. There is an Index Volumes tab and a right click will show the option to Update,
Rebuild, or Repair. The Repair option is not available if no items have failed indexing.

15 Replies
Accepted Solution!

GJP, IndexVolumeReplay If

GJP,

IndexVolumeReplay
If your users report receiving a message that the indexes are not up to date or that they are marked as Failed you should take action to update the index using the IndexVolumeReplay.exe. Remember, just because an index volume is marked as Failed it does not necessarily mean corruption.

IndexVolumeReplay.exe for EV 6 and EV 7 can be found in the EV installation directory.
(i.e. C:\Program Files\Enterprise Vault\) To launch it just double click the IndexVolumeReplay.exe. You can update a failed index volume by right clicking the volume that is marked as failed and choose Update Index Volume.

IndexVolumeReplay.jpeg

In 2007 and above you can access IndexVolumeReplay from the Vault Admin Console by right clicking the Archives container.
Remember to always do an Update first and not Rebuild, the Update just processes the items that are missing, whereas a Rebuild will delete the entire index volume and start from the beginning. An Update is quicker and takes up fewer resources.

If it is necessary, you can Rebuild all the associated index volumes of an archive.

Right click the target archive and choose Properties. The Advanced tab has the Rebuild Index button that will rebuild all the index volumes at once.

You can additionally manage index volumes by viewing the properties of the individual archive. There is an Index Volumes tab and a right click will show the option to Update,
Rebuild, or Repair. The Repair option is not available if no items have failed indexing.

How can you optimise the

How can you optimise the Indexing of the Vault system? We've corrupted some of our indexes and so need to rebuild these at the same time as indexing all new items. Currently the JournalArchive table is showing over 6 million items awaiting to be indexed

Optimizing Indexing

PaulB-UK,

If you're getting enough corruptions that you have that many items backing up for processing there are a couple things you need to consider.

- First, check and re-check your antivirus exclusions. Nothing will corrupt indexes faster than an AV scan.
- Next, check your index locations. Do you have ample free space? Do you need to add a couple more index locations? The best practice for index folders is at least 8. In fact, v8 of EV will automatically create 8 index folders for you upon definition of a location.
- Are your index drives a RAID or other high speed medium? Indexes and SQL storage are the 2 areas you cannot afford to skimp on drive speed. You need fast disk for these. Also check for failed drives or RAIDs that are rebuilding or running in depreciated mode due to a drive issue.
- Last, if you have that many items backed up for indexing, have you considered adding additional EV servers to spread out the load a bit? You could very well be overloading the one indexing service if it has to reindex constantly as well as keep up with the new data.

Hope this helps,

Mark

Hi Wayne, Thanks for the

Hi Wayne,

Thanks for the reply, it seems that some indexes were set to 'update' by a colleague a few weeks ago and are still reporting as <failed/rebuilding>. I'm guessing that this means the update has failed and i need to rebuild them. Is it safe to tell them to rebuild and not 'update' them first?

So for each user with a failed index i will need to go through the list and right click rebuild, there is no simple way to tell it to rebuild all indexes?

Thanks for your help with this.

G.

Hi, Just because an index is

Hi,

Just because an index is set to failed it doesn't mean it is corrupt at all.  more often than not it isn't.  If you do an 'update' on one of these failed indexes which happens.  Does it goto failed right away and if it does what events does it log?

Later versions of EV will automatically do an 'update' operation on any indexes marked as failed. 
In version 8 you can use the indexcheck utility to force a rebuild on a list of indexes which have been marked as failed.

Hi EV, I have told one of the

Hi EV,

I have told one of the failed indexes to update and yes it seems to fall over straight away.

The event log lists two errors:

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7264
Date:  22/12/2009
Time:  17:06:40
User:  N/A
Computer: EDEN
Description:
Abnormal error occurred
 
Error:     The handle is invalid.  [0x80070006]
Reference: CSynchronizerThread/TR/dh
Index:     
Info:       
            
--------
and
--------

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7292
Date:  22/12/2009
Time:  17:06:40
User:  N/A
Computer: EDEN
Description:
The index volume has been marked as failed.
Index Volume: 117DF645FD72C9742AA6CFDBDA0D55A071110000vault.dsllan.net/Volume:7 ('username')
Index Volume Path: E:\Vault\Index\117DF645FD72C9742AA6CFDBDA0D55A071110000vault.dsllan.net
Reference: ValidateFileChecksum
           
 
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors.  The index volume will remain inaccessible until it has been repaired.

----------

Any ideas from the above?

I will be updating the vault version in the new year, i just figured best to rebuild the indexes now if it needs doing so it can run over our christmas shutdown period.

Thanks for your help

G.



 


 

Also....

Hi All,

I am also consistently seeing errors logged against two users/volumes in particular, could this be related to / the cause of the other index failures?

Here are the events that are logged for one of the users:

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7264
Date:  22/12/2009
Time:  10:12:08
User:  N/A
Computer: EDEN
Description:
Abnormal error occurred
 
Error:     The Index Server cannot complete the requested operation as it is stopping.    [0x80041c1a]
Reference: ST/TR/aah
Index:     1A8388924F802414F876D4EBAD7CF7CEB1110000vault.dsllan.net/Volume:644 (USERNAME2)
Info:       
           
---------------------------

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7264
Date:  22/12/2009
Time:  10:12:08
User:  N/A
Computer: EDEN
Description:
Abnormal error occurred
 
Error:     A low level indexing operation has failed   Error:    Index Id:    [0xc0041c43]
Reference: CAVUpdator::CountWords
Index:     1A8388924F802414F876D4EBAD7CF7CEB1110000vault.dsllan.net/Volume:644 (USERNAME2)
Info:       - DELETED - 
           %5
           %6

-----------------------------

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7235
Date:  22/12/2009
Time:  10:12:08
User:  N/A
Computer: EDEN
Description:
A low level indexing operation has failed
Error: a severe error occurred; cannot continue [AV:63]
Index Id: 1A8388924F802414F876D4EBAD7CF7CEB1110000vault.dsllan.net/Volume:644 (USERNAME2)

-----------------------------

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7235
Date:  22/12/2009
Time:  10:12:08
User:  N/A
Computer: EDEN
Description:
A low level indexing operation has failed
Error: an internal error [AV:57]
Index Id: 1A8388924F802414F876D4EBAD7CF7CEB1110000vault.dsllan.net/Volume:644 (USERNAME2)


Thanks for taking the time to look.

G.

 

Hi When you goto V7.5 sp6

Hi

When you goto V7.5 sp6 (presuming you goto the latest) then that will help in two ways
1. the check which is failing validatechecksum is now turned off by default
2. indexes marked as failed will automatically be re-tried.

So really then ironically though you are trying to fix them before the upgrade for some of the indexes, the upgrade will fix them for you.

the ones you could troubleshoot now are the ones that are giving an error that is not validatechecksum like those you report above with the wordcount error.

In the meantime where is a registry key for V6 which you can use to disable the validatechecksum feature.  So you could now at least create the registry key and do updte on those indexes if you so wished.


The registry key is
Create a DWORD registry key called
ValidateCheckSum and give it a value of 0 in
HKLM\software\kvs\enterprise vault\indexing\validatechecksum



Mike

Hi Mark, Thanks for

Hi Mark,

Thanks for replying.

The original cause of the Index failures was down to running out of space on the index server of sorts. The indexes are held on a Netapp filer which ran out of i-nodes. We have now fixed this and monitor the situation. However, the situation we are experiencing now seems to be that we just can make up for lost time.

Disk situation for both SQL and storage (both index and Vault storage) are on mutli-disk aggregates hosted on Netapp storage.

Your last point is something we have implemented but we are not sure this is configured optimnally. We introduced a new Vault specifically to do Indexing, however this still seems slow and we are unsure as to how much adding an Index server helps the overall situation. Does the index server act directly on the Vault sets or does it ask the the existing Vault servers to do a scrape of the information in their storage before it then processes it?

BTW we are running EVault 6.0 with SP6

Paul

Hi Mike, Thanks for the info,

Hi Mike,

Thanks for the info, adding the registry key worked and i was able to rebuild my index., however, one of the other users i attempted to fix (same person as yesterday) comes up with the following errors after a while:

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7292
Computer: EDEN
Description:
The index volume has been marked as failed.
Index Volume: 117DF645FD72C9742AA6CFDBDA0D55A071110000vault.dsllan.net/Volume:7 (username)
Index Volume Path: E:\Vault\Index\117DF645FD72C9742AA6CFDBDA0D55A071110000vault.dsllan.net
Reference: Too many consecutive failed items
           
 
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors.  The index volume will remain inaccessible until it has been repaired.

It is surrounded by these errors:

Event Type: Error
Event Source: Enterprise Vault
Event Category: Storage Crawler
Event ID: 6882

Unable to complete retrieval request
 
Reason:          The system cannot find the file specified.  [0x80070002]
Vault:             Sent Items
Vault Id:          1A88DE08E83A7294DA168C2FE5353D2EF1110000vault.dsllan.net
Item Id:           1A88DE08E83A7294DA168C2FE5353D2EF1110000vault.dsllan.net+607000000000000~200805161335560000~0~BFE0F40BD18C4DB6957EAC8A7E54696+0++0+2+39606.07756180555600000+33+27930++108+7+1620640
Reference:         LSFF/LF
Suplementary Info: \\homer-1\VaultStore\mailboxstore\2008\05\16\13\607000000000000~200805161335560000~0.DVS 
   

The first event log error mentions that the index will be unavailbe until it is repaired, how can i repair it?

Looking at the location specified above i can confirm that there is no file there matching that name.

Thanks again for all the help

G.


 

Hi, So the index marks itself

Hi,

So the index marks itself as 'failed' if there are number of items that cannot be recalled.  The theory being that if so many items have not been recalled perhaps the storage device is offline etc.

So why are those files not there, any idea?

You can set another registry key which controls this behaviour as well.  So setting this will allow indexing to carry on trying to index items and not set it to failed no matter how many items cannot be recalled.  However rememebr then the index will be quite incomplete which may be an issue for your users (which is related to my question of how come the files are not there)
Also by setting this registry key depending upon how many files are not there you will get quite a lot of events in the event log.

So create the DWORD registry key  called

MaxConsecutivePoisonPillItems

and give it a value of 0 to disable this check

under
HKLM\Software\KVS\Enterprise Vault\Indexing

and restart the Storage and Indexing Services

Hi Mike, Unfortunately i have

Hi Mike,

Unfortunately i have no idea where these items have gone. I'm frantically trawling through the backups to see if i can get anything back from these folders.

If i can't find them on a backup is there any way that they can be rebuild without losing data?

Thanks for the reg key, looking at it i think it would be wiser to try and find where the files have gone rather than accept the loss.

I submitted a few more index updates to see if it was a widespread issue and I am now seeing these events logged:
 
Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7264
Computer: EDEN
Description:
Abnormal error occurred
 
Error:     The AltaVista index is not open   [0xc0041c64]
Reference: ST/TR/gs
Index:     1A8923F4C5C69BC40996690458A85EAE51110000vault.dsllan.net/Volume:352 (USERNAME)
Info:       

Any ideas about this one?

Thanks so much for your time.

G.


 

are there other events logged

are there other events logged for the index at the same time?

Hi Mike, Yes, there are other

Hi Mike,

Yes, there are other errors logged:

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7235
Date:  24/12/2009
Time:  00:05:15
User:  N/A
Computer: EDEN
Description:
A low level indexing operation has failed
Error: an internal error [AV:57]
Index Id: 111D2D3EFFA15FE4D92AC8C379BD80C831110000vault.dsllan.net/Volume:94 (username)

and

Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7264
Date:  24/12/2009
Time:  00:05:15
User:  N/A
Computer: EDEN
Description:
Abnormal error occurred
 
Error:     A low level indexing operation has failed   Error: #1   Index Id: #2   [0xc0041c43]
Reference: CAVUpdator/OAVI/o(h)
Index:     111D2D3EFFA15FE4D92AC8C379BD80C831110000vault.dsllan.net/Volume:94 (username)
Info:      Params - format:-1, buckets:0, tiers:0, cache:268435456, options:0x27, charset:1 


Does this provide anymore of an insight?

Thanks

Gavin
 

Hi, yes that does look

Hi,

yes that does look corrupt.  Generally speaking indexes are mostly likely to become corrupt when you run out of disk space.  So given you said that's what happened i imagine quite a few are indeed corrupt.

Mike