Unable to Rebuild or Repair a failing index

[H_PHX]

Hi All,

Environment:

EV 9.0.2 running on Server 2008 R2 Standard.

Problem Descrition:

I have come across a small number of indexes which are refusing to rebuild and repair in my current EV environment.

To briefly explain, we had a number of failed indexes identified by using the "Manage Index Volumes" tool. The majority of them rebuilt successfully and the more difficult one were corrected by following this thread: http://www.symantec.com/business/support/index?page=content&id=TECH174186

I am now left with six indexes that produce an event id of 7292:

The index volume has been marked as failed.

index volume: <index volume id followed by username>

index volume path <self explanitory, and is correct>

Reference: Too many consecutive failed items.

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible unitl it has been repaired.

Subsequently, the total items that are waiting to be indexed in the vault store has now reached a limit that has raised a critical status alert in the VAC which is slowly increasing as each day passes.

I did read the following thread: https://www-secure.symantec.com/connect/forums/unable-rebuildrepair-failing-index which mentioned making a change to the registry in order to increase the threshold of  known errors so that the task of a rebuild\repair does not fail, but as the thread goes, I don't want a quick fix and would rather know what is causing this to happen.

That being said I have run a DTrace on one of the affected achive but I require some assistance with identifying the problem.

This is where I was hoping someone would be able to point me in the right direction.

I have attached the DTrace to this discussion and would appreciate any input should anyone take a look.

Thanks for taking the time to read my post and hopefully someone out there can provide me with some much needed assistance.

Regards,

H_PHX

[GabeV]

Hi,

Do you see any Storage Crawler errors or warnings before the event ID 7288 occurs?

[H_PHX]

Hi Gabe,

Thanks for your reply.

I see a number of Storage Crawler Errors before 7288 is flagged:

Event ID are as follows: 7116, 7083.

It looks like they cycle a couple of times and then eventually 7288 appears.

[GabeV]

Could you please post one of those Storage Crawler errors?

[H_PHX]

No Problem.

Event ID: 7083.

Retrieval of saveset failed.

Archive Id: 1ED2C0728D5114B4796A61BD161FF7B5A1110000evsite.mydomain.com

Transaction Id: 9A95319F45D14BF9A24527B0FD1D8A4

Extended information: The system cannot find the path specified. (0x80070003)

Internal Ref: Vault/RI

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp

[JesusWept3 · Accepted solution]

You're going to need to run EVSVR to check for missing Savesets and then try and restore them from backup If you can't then you're going to need to run evsvr is repair mode to remove those SQL entries to the Savesets that no longer exist

[GabeV]

Did you have a storage issue recently? It appears that there is data missing from the storage. You can run EVSVR with Verify > ArchiveObjects > SavesetValid against the archive to determine how many items are missing. The EVSVR report will tell you what folders you need to restore. Copy the archive ID, hold ctrl+shift in your keyboard and do right click over the Archives in the VAC, then click in Find Archive or Folder, paste the archive ID so you will find what archive is being reported in that error. Then, you can select the archive in EVSVR.

Once you have the report, you can restore the data. If you can't restore it, I'd suggest you to open a case with support to assist you further.

I hope this helps.

[H_PHX]

Hi Gabe\JW3,

thank you both for your suggestions, which I will carry out soon.

In regards to the question posed by Gabe, "did you have a storage issue recently?", other than the parition growing in size and then being rolled over I can't recall any other problems.

With growing paritions we found that performance of EV in general was affected so it was decided that we needed to roll over the paritions when they reached 200GB, which we have done on a few occassions now.

I can't say that I recall any other type of storage issues to be honest.

I'll give your suggestions a shot and see where this takes me.

Many thanks for your time and hopefully I can get an update on here as soon as I have one.

Thanks again.

[EV_Ajay]

Hi,

From Event ID : 7292

Determine whether the Savesets that are attempting to be scanned by the Indexing Service are present at their specified location and are available to the Enterprise Vault Server. If the Savesets are not found in the Vault Store they should be restored from backup and the Index Update should be tried again.
 

[Rob_Wilcox1]

@H_PHX Do you ned any more assistance with this issue?

[GabeV]

Hello H_PHX,

Do you have any updates on this thread? Do you need more assistance regarding this topic? If not, please mark the post that best solves your problem as the answer to this thread.

[EV_Ajay]

Hi,

You can check the Indexmissing.log file to understand which saveset is missing.

You can refer following technote with saveset id present in the Indexmissing.log to find out whether DVS files is missing from location or not.

How to find the .DVS file name corresponding to an Enterprise Vault (EV) archived item.

http://www.symantec.com/docs/TECH137949

[H_PHX]

Hi Guys,

Apolgies for the late reply, unfortuntely I have not been able to look at this as yet due to unforseen circumstances. I hope to have a look at some point this week. I will provide an update as soon as I have one, starting with .EVSVR.

Many thanks to those that have tried to help or offer assistance. I apolgies for taking my time to reply to the suggestions put forward. IT infrastructure is so unpredictable your plans can change in a moment :)

[H_PHX]

Hi Gabe,

just wanted to ask how long can this process normally take to complete?

When it completes, should there be some sort of confirmation to notify me that this has now completed?

I only ask as I started the process at 9:15am this morning.

Cheers.

[GabeV]

Hi H_PHX,

When you start EVSVR you get a message in the console and when it ends you get another message as well. Typing 'status' you can see if the tool is still running. However, there is no progress bar or something that could tell you how long would take or how much time is left.

[H_PHX]

Thanks Gabe.

In that case it is still active and running.

I'll just need to be patient I guess.