Errors following Index Rebuild

[CadenL]

Hi All

Due to some issues I've needed to run an Index rebuild on a large number of user archives - this has been running in the background for many days and has now completed.

My understanding is that an index rebuild will delete and rebuild the index for every item within a users archive from scratch - hence why it takes so long.

Having completed the index rebuild I have several archives that completed with a status of 'Unsuccessful' and have errors. An example is below:

Archive Name:  A User

Index Volume Range:1 - 31215

Items Added:31053

Items With Errors:24

Status:Warnings

I'm curious on how to interprtet these warnings and what action to take?

As the Index has been rebuilt from scratch I can't see why it should have any errors as I thought the idea of the rebuild is to remediate any errors? Or are these errors identifying corrupt and\or items that no longer exist within the archive and therefore fail to index (so have been removed) and therefore EV is reporting on errors that were encountered but remediated during the build and wouldn't reoccur if I were to run a subsequent rebuild on the same users archive.

many thanks

[TonySterling]

That status is defined as this:

• For upgrade, rebuild and synchronization subtasks, this shows the number of items that could not be retrieved, usually due to storage issues.
• In the case of upgrade and rebuild subtasks, items with errors cause the subtask to wait for user input if you did not select the Ignore items that cannot be indexed option when you submitted the task.

So basically those items have savesets on disk but could not be opened or re-indexed.  

[CadenL]

Thanks Tony

What action can or should be taken?

Perhaps run a rebuild against those particular archives with a hope they may succeed on a subsequent pass?

Is it that the item can still be opened via an outlook shortcut or archive explorer, but wouldn't show if a user attempted a search for the item? 

Is there a log file for the items that errored (some archives only had 1 error) that may help in identifying the root issue?

[Prone2Typos]

THe items will be retrievable in Outlook but not AE or Search (or DA/CA). "EVSVR interactive ds" should be able to retrieve them and should show you any specific issues encountered along the way with the added benifit of being able to see what the item is. You can always reprocess and dtrace the operation to get more log info on why they are failing.

[Prone2Typos]

Oh also there were probably events logged in the EV event log at the point of failing to index these items.

[Nishant_p]

indexing engine is build to provide archive items accessible via vault search hence there is no relationship between shortcut therefore you can easily open archive items from outlook shortcut. 

for the items where a rebuild is failed, check the report, you can also take support from SE to check things from the database side also. 

in some cases, I have also noticed that second attempt to rebuild can help to rebuild remaining items 

[GertjanA]

You first need to verify the report of the rebuild task. That will list why items failed to index. Warnings can be cuased by (amongst others):

An attachment which cannot be indexed, an encrypted item, an item being too large, item still in storage queue, item indexed, but not archived, item being corrupted, item not retrievable. 

I do check the warnings (you will also see these when syncing an index), but most of them are 'ignorable'. Only if I see warnings related to issues related to items that should be on disk, I investigate further. If all the indexes rebuild fine, I would first do a Sync of the indexes with the failed items. you can also run a Verify (full verify, include non-indexed items), to see what the warnings are. A re-rebuild is last resort..

[CadenL]

Hi

I've attempted a further rebuild of all the failed indexes (it doesn't take too long), however, they are still showing as failed. Below is an example of what I see in the event viewer all each failed index.

Archive Name: A User
Index Volume ID: 12D490BE177412A45AD52FBB8BA9FFD3011100003663VEVFSA01_24815
Reason: TooManyConsecutiveFailedItems
Error Type: FrenzyErrorDetected

I believe this suggests that there are too many failed items for the index to be repaired and brought back online. What is the best way to remediate this if even a rebuild is failing? I recall reading about the ability to change the number of consecutive items that EV can sustain but not sure if this is what I need to do here and what the impact is of making this change.

kind regards

[plaudone1]

Hi CadenL,

You can update the setting in the following location.  Site Properties > Advanced tab > Indexing > Maximum consecutive failed items value. I would set it to 1000 and restart EV services. Then run a Sync on the volume to see if it comes online.  From there you can determine if you need to raise the number.  Also, should determine why the items cannot be retrieved from storage to be indexed.   

Regards,

Patrick 

[CadenL]

OK thanks - I'll give that a go.

To help me better understand the issue..... does this error indicate that the index has a reference to a saveset that doesn't exist? So when EV attempts to fetch the saveset (referenced by the index entry) in order to re-index it (as part of the index rebuild) it simply cannot find the file?

So the actual cause of the error is with missing files on the EV vault store partitions rather than an issue with the files in the index locations?

If I was able to restore the EV partition data from a backup to a scratch location - could I potentially find the missing files and copy them back to their original locations and the index rebuild would succeed?

[Marcde]

Hi, 

the rebuild report usually shows you what it failed to index and depending on the error there can also be a url (download.asp call for that specific item) mentioned that you can try to access with the vsa from the ev server directly to see if this is failing as well. You might also be able to see the storage crawler event id 28998 with an error similar to the following:

Retrieval of a saveset failed.

Reason: The system cannot find the path specified. (0x80070003)

.....

In addition to that I like to perform a dumpsaveset operation of a specific item to see which data is missing exactly. 

How to extract the Saveset and SIS parts of a specific archived item using EVSVR

https://www.veritas.com/support/en_US/article.100022284

But to answer your question, yes if it fails to index specific items missing data on the partition can be the reason for that. As stated the dumpsaveset operation should help you to understand which exact part is missing from the partition so that you can try to extract it from a backup. If the amount of lost data seems to be bigger than a few I usually like to perform a evsvr verify operation to get a better idea of how much items are affected.

Regards

Marc