Index volume not able to be repaired

[WallySimmonds]

Hi guys,

Wall of text to follow!

I'm an engineer for a company that is about to take over server support from another company in the coming months.  Now, part of the support we provide currently is for EV - and so we've been called in to fix an issue that quite probably has been caused by the outgoing outsourcing vendor.

Basically what has happened is that they've created an index store on the C drive.  Don't ask me how that happened, but that's how it seems..  By the looks of things, they've managed to fix things up and set it up correctly on the correct drive, but not before EV ran out of space and it looks like there's all sorts of errors in the event logs since.  This may be unrelated, but I thought it would be worth mentioning.

I come in weekly to do checks on the system, and notice that one of the index volumes has failed.  Attempting to repair does not work, and neither does a rebuild. Further investigation reveals that there are too many consecutive items failing as per this post here:  

https://www-secure.symantec.com/connect/forums/index-volume-has-been-marked-failed

Further investigation reveals it can't find certain DVS files.  I've done a search on the entire server and they're definitely not anywhere.  So, I'm not really sure what to do about that.

My question is - with the information given, do you think it'd be alright to proceed with the 'poison pill' registry modification as mentioned in the post linked?  I realise they've probably lost data here, but my main concern is getting the index up and going again. 

I will also log a support case with Symantec, but thought I'd also post here in case anyone else has any suggestions.  We are running Exchange Server 2003 SP2 and EV 7.5 btw.  And yes, next time I shall do things myself rather than leave it in the hands of the outgoing vendor.  Apologies if I have mixed up the terminology as I am not very experienced with EV myself.

Brendan

[Itzik_Gur]

HI Brendan,

Using the 'Poison Pill' is a good option if you are willing o accept that there will be files missing (henceyour search results might not be accurate). Then again, seeing that you say that there are missing DVS files, then you might not have those file in the archive anyway.

Do you have any backups you can restore from? if the missing DVS files are from old mail items, you might be abel to minimise the impact of lost data by recovering some of those files from an old backup!

Another suggestion is that you run a Vault Store check to ensure that your vault stores are OK or repair (ignore) missing files so you won't carry them to future upgrades. You will probably need to get Symantec support to help you with the Vault Store check.

Regards,
Itzik

[MichelZ]

Brendan

You should first get two tools, called DVSChecker and the oposite of it. (Can't remember the name.. :) )
Those tools will check the Database and the physical DVS files for consistency, and you then can restore missing DVS files prior to do anything with the index.

As a second step, you should then try to rebuild those failed indexes again, and only if they fail again you should consider posion pill.

(BTW: You can get those tools only trough support....)

Cheers

[John_Chisari]

Yep, agree with MichelZ and Igur - it's cleaner if you get VSVerify  (that's the other one Michel) - so you can clean up the database - and then EV won't try and access them anymore.

It will also tell you how many DVS files are missing and which ones.  Gives you a chance to restore them from backup, if you have them.

[MichelZ]

Thx! :)

[WallySimmonds]

Thanks guys, appreciate the feedback.  I'll take this to Symantec support and see what they say.  I must admit, using this forum is a great way to get help!

Suspect I'll be using this site a helluva lot more in the coming months when we roll out SEP and SEE :p

[Liam_Finn1]

 I agree with MichelZ

The poison pill setting is a way to rebuild no matter how many errors you get and should only be used as a last resort, but in saying that the one thing it will provide is after the index is rebuilt with the failed items missed you will get an indexmissing.log in each index which will contain the list of items that failed to be added to the index. This can be usefull information when you are trying to identify what dvs files you need to find on tape and recover to the server before doing a repair to add them back to the indexes