Enterprise Vault - Corrupt Index

[C_P2]

Hello,

Recently our security team decided to deploy AV to our EV 10 server without implementing any kind of exclusions.

About a week later, we noticed EV using 100% cpu constantly and spawning new index processes every second and logging the following errors in E:\Program Files (x86)\Enterprise Vault\EVIndexing\data\indexmetadata\reporting\system-reporting:

[live] [indexer] for collection [15F3CD023737CBB48B951077FCB4736C4_594] terminated unexpectedly: [A file [.\viv_idx_Q71BY7] that we created has now disappeared. Cannot continue ([index-merger.c]:1160 [Fri Apr 04 08:31:29 2014
]]

Could not start the [live] [indexer] ([E:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [E:\Enterprise Vault\Indexes\Indexes 01\Indexes\index4\103326B3E1C0BBD47B2614F0BC4251A79_434\live] for collection [103326B3E1C0BBD47B2614F0BC4251A79_434]:  <log ><log ><error time="177390" date="1396564288" id="SERVICE_EXEC_FAILED" >Failed to run <string name="command" >E:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service</string> in mode <string name="mode" >--go</string> in path <string name="path" >E:\Enterprise Vault\Indexes\Indexes 01\Indexes\index4\103326B3E1C0BBD47B2614F0BC4251A79_434\live</string> using port <int name="port" >52864</int>: <string name="error" >The media is write protected.
</string>.  </error></log></log>

Those errors are logged every few seconds as the indexing service fails and spawns another process.

I'm not sure why it is saying the media is write protected either, I can see that it's not and that the service account has access to write.

As an interim workaround, I closed all the indexes in question and set them to backup mode so nothing can be written to them and uninstalled the AV, I then added a set of new indexes and that has been working for now.

As this is a patchy fix, I was wondering if it is possible at all to recover the indexes that are now corrupt? 

[AndrewB]

i'm guessing by the logs that you're on EV 10?

you could restore a backup from before the AV was deployed, run an index maintenance task (verify + sync) and then rebuild any missing indexes.

[C_P2]

Hi Andrew,

Thanks for the quick reply. The version is 10 (I've just updated the post to reflect that).

I'm concerned that if I restore from backup and rebuild the index then archiving and access to archived emails will be unavailable for several weeks.  We have several million emails in the archive and it will cause the business significant problems.

Is there any other option?

[TonySterling · Accepted solution]

Access to archived items via shortcuts will not be affected.  Searching, however, will be greatly affected if the indexes are corrupted. 

You could first run the verify process to see how bad the indexes are out of align. Then if necessary you can restore the Indexes from backup prior to AV scanning will at least allow them to be searchable up to that point and you will be able to run the synchronize process to get them up to date.

[C_P2]

Thanks for the input guys.  We are going to delete the indexes and rebuild them from scratch this weekend.  It's a long weekend which will be good and we'll also allocate some extra resources to the EV VM for that period so hopefully it can complete in 4 days.

I'm going to go and read through some documentation now, but is there anything I should know before doing this?

[TonySterling]

Unless you do not have a backup I personally wouldn't go that route.  I would do the backup and and synchronize.

You will need to make it clear that you will not have complete searches until the Index rebuild completes, so if you have a journal archive you would start with that one.

[C_P2]

Hey guys,

Just following this up for historical reference.  I deleted 2 indexes and rebuilt from scratch, that took too long for my liking so I restored the rest from a backup and then synchronized, which is working well.

Thank you all for your input, it was much appreciated.