03-18-2020 10:28 AM - edited 03-18-2020 10:29 AM
Hi All
Due to some issues I've needed to run an Index rebuild on a large number of user archives - this has been running in the background for many days and has now completed.
My understanding is that an index rebuild will delete and rebuild the index for every item within a users archive from scratch - hence why it takes so long.
Having completed the index rebuild I have several archives that completed with a status of 'Unsuccessful' and have errors. An example is below:
Archive Name: A User
Index Volume Range:1 - 31215
Items Added:31053
Items With Errors:24
Status:Warnings
I'm curious on how to interprtet these warnings and what action to take?
As the Index has been rebuilt from scratch I can't see why it should have any errors as I thought the idea of the rebuild is to remediate any errors? Or are these errors identifying corrupt and\or items that no longer exist within the archive and therefore fail to index (so have been removed) and therefore EV is reporting on errors that were encountered but remediated during the build and wouldn't reoccur if I were to run a subsequent rebuild on the same users archive.
many thanks
03-18-2020 10:59 AM
That status is defined as this:
So basically those items have savesets on disk but could not be opened or re-indexed.
03-18-2020 11:21 AM - edited 03-18-2020 11:22 AM
Thanks Tony
What action can or should be taken?
Perhaps run a rebuild against those particular archives with a hope they may succeed on a subsequent pass?
Is it that the item can still be opened via an outlook shortcut or archive explorer, but wouldn't show if a user attempted a search for the item?
Is there a log file for the items that errored (some archives only had 1 error) that may help in identifying the root issue?
03-18-2020 11:53 AM
03-18-2020 11:58 AM
03-18-2020 02:59 PM
indexing engine is build to provide archive items accessible via vault search hence there is no relationship between shortcut therefore you can easily open archive items from outlook shortcut.
for the items where a rebuild is failed, check the report, you can also take support from SE to check things from the database side also.
in some cases, I have also noticed that second attempt to rebuild can help to rebuild remaining items
03-19-2020 01:01 AM
You first need to verify the report of the rebuild task. That will list why items failed to index. Warnings can be cuased by (amongst others):
An attachment which cannot be indexed, an encrypted item, an item being too large, item still in storage queue, item indexed, but not archived, item being corrupted, item not retrievable.
I do check the warnings (you will also see these when syncing an index), but most of them are 'ignorable'. Only if I see warnings related to issues related to items that should be on disk, I investigate further. If all the indexes rebuild fine, I would first do a Sync of the indexes with the failed items. you can also run a Verify (full verify, include non-indexed items), to see what the warnings are. A re-rebuild is last resort..
03-25-2020 11:48 AM - edited 03-25-2020 11:49 AM
Hi
I've attempted a further rebuild of all the failed indexes (it doesn't take too long), however, they are still showing as failed. Below is an example of what I see in the event viewer all each failed index.
Archive Name: A User
Index Volume ID: 12D490BE177412A45AD52FBB8BA9FFD3011100003663VEVFSA01_24815
Reason: TooManyConsecutiveFailedItems
Error Type: FrenzyErrorDetected
I believe this suggests that there are too many failed items for the index to be repaired and brought back online. What is the best way to remediate this if even a rebuild is failing? I recall reading about the ability to change the number of consecutive items that EV can sustain but not sure if this is what I need to do here and what the impact is of making this change.
kind regards
03-25-2020 01:46 PM - edited 03-25-2020 01:47 PM
Hi CadenL,
You can update the setting in the following location. Site Properties > Advanced tab > Indexing > Maximum consecutive failed items value. I would set it to 1000 and restart EV services. Then run a Sync on the volume to see if it comes online. From there you can determine if you need to raise the number. Also, should determine why the items cannot be retrieved from storage to be indexed.
Regards,
Patrick
03-25-2020 02:10 PM
OK thanks - I'll give that a go.
To help me better understand the issue..... does this error indicate that the index has a reference to a saveset that doesn't exist? So when EV attempts to fetch the saveset (referenced by the index entry) in order to re-index it (as part of the index rebuild) it simply cannot find the file?
So the actual cause of the error is with missing files on the EV vault store partitions rather than an issue with the files in the index locations?
If I was able to restore the EV partition data from a backup to a scratch location - could I potentially find the missing files and copy them back to their original locations and the index rebuild would succeed?
03-26-2020 12:13 AM
Hi,
the rebuild report usually shows you what it failed to index and depending on the error there can also be a url (download.asp call for that specific item) mentioned that you can try to access with the vsa from the ev server directly to see if this is failing as well. You might also be able to see the storage crawler event id 28998 with an error similar to the following:
Retrieval of a saveset failed.
Reason: The system cannot find the path specified. (0x80070003)
.....
In addition to that I like to perform a dumpsaveset operation of a specific item to see which data is missing exactly.
How to extract the Saveset and SIS parts of a specific archived item using EVSVR
https://www.veritas.com/support/en_US/article.100022284
But to answer your question, yes if it fails to index specific items missing data on the partition can be the reason for that. As stated the dumpsaveset operation should help you to understand which exact part is missing from the partition so that you can try to extract it from a backup. If the amount of lost data seems to be bigger than a few I usually like to perform a evsvr verify operation to get a better idea of how much items are affected.
Regards
Marc