Index Volume has been marked as failed.

We are using Enterprise Vault v8.0.1.1560, that was upgraded from v7.0 a few months ago.

The system is Archiving WIndows Exchange Mailboxes and Journal Mailbox.

The User Archives are fine, but the Journal Archive has Failed.

A new Journal Archive was created a Month ago, so we could continue Archiving a new Journal Mailbox and repair or rebuild the old one.

I had to resort to the Rebuild Option and after several days, the process completed 85% of the final update stage and FAILED. I tried to repair
several times, with the same results.

The Event log states Index Failed. Reference: Too many consecutive failed items.

Should I try to restore the Index from a few months ago and repair again ? Or is there another option I should try first.

If I do restore, will it effect the New Index in the same location ?

Thanks,

Enterprise Vault

Information Governance

Recovering

Restore

Liam_Finn1
16 years ago
What events are created in the Event log when you try to repair the index?

Also do a DTrace of the indexbroker service and the storagebroker service in verbose mode when you attempt the repair and attach the output file so we can see the result

There is a way to set a poison pill setting in the registry to bypass the default number of failures to allow your index to rebuild but you will be missing items in the index after it completes

Here is how to make the setting

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\Indexing\PoisonPillCount
DWORD
Default: 1
This will set the number of attempts to process an item before it is deemed to be a poison pill. The action taken depends upon what's being processed...
e.g.
After PoisonPillCount consecutive failed attempts to open an index volume it is deemed to be corrupt and is marked as 'failed'
After PoisonPillCount failed attempts to add an item to the index the addition is abandoned and the index update steps onto the next item.

If you do not have the following then please create it, if you do then change the value to '100000' :-

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\Indexing\MaxConsecutivePoisonPillItems
DWORD
Default: 100
The maximum number of consecutive poison pill items before an index update is deemed to be a poison pill and abandoned. The index update will be restarted on indexing service restart or the hourly check for pending updates.

Remember please backup your registry before making any changes and this setting is to be removed after you rebuild the index

15 Replies

TonySterling
Moderator
16 years ago
You will want to look at what is causing the failed items. This is typically a failure connecting to the storage device. Make sure that is ok and try to do an update on the index volume.
RBostwick
Level 3
16 years ago
Thanks Tony for the suggestion.
I am able to Browse the attached Storage device from the Archive Server without a problem and have the other Archives
on this storage device as well.

The Original Index has over a years worth of Indexed Data on it for the Journal Mailbox. The new Journal mailbox has the Index located on the same storage device and can be searched with no problem.

Being that the original has Failed to Rebuild or Update, it is sitting in an unusable state.

I was hoping there would be another solution, without having to restore the Index Folders.
Liam_Finn1
Level 6
16 years ago
What events are created in the Event log when you try to repair the index?

Also do a DTrace of the indexbroker service and the storagebroker service in verbose mode when you attempt the repair and attach the output file so we can see the result

There is a way to set a poison pill setting in the registry to bypass the default number of failures to allow your index to rebuild but you will be missing items in the index after it completes

Here is how to make the setting

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\Indexing\PoisonPillCount
DWORD
Default: 1
This will set the number of attempts to process an item before it is deemed to be a poison pill. The action taken depends upon what's being processed...
e.g.
After PoisonPillCount consecutive failed attempts to open an index volume it is deemed to be corrupt and is marked as 'failed'
After PoisonPillCount failed attempts to add an item to the index the addition is abandoned and the index update steps onto the next item.

If you do not have the following then please create it, if you do then change the value to '100000' :-

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\Indexing\MaxConsecutivePoisonPillItems
DWORD
Default: 100
The maximum number of consecutive poison pill items before an index update is deemed to be a poison pill and abandoned. The index update will be restarted on indexing service restart or the hourly check for pending updates.

Remember please backup your registry before making any changes and this setting is to be removed after you rebuild the index
Maverik
Level 6
16 years ago
Hi Basically the issue will be occurring because if an index cannot add 25 ( I believe is the default) items consecutively then it will be marked as failed. Each failure will be written to the event log. You can increase this poison pill limit by use a regards key that I cannot recall at the mo. I will add it tomorrow unless someone beats me to it. What this should allow is say you have 26 consecutive failed items out of 5 million you can rebuild the index at least. Would this be accepatable for you?
RBostwick
Level 3
16 years ago
Thanks for the suggestions.

I added the Registry settings, restarted the Index and Storage Service.

The Index Update is running currently. Hopefully it will finish in a day or so and not Fail again.

I will keep you posted.
Liam_Finn1
Level 6
16 years ago
This should allow it to rebuild for you but it will be missing items.

You will need to investigate the reason for the failure, correct it and then repair the index to have the missing items repopulated back into the index

Keep us posted.

I would advise you open a support case for this.

The rebuild could be failing because of missing files in the archive
Orphan save sets or missing entries in the archive folder tables in the Enterprise vault Directory database...to name a few

Chances are you will need support to assist in getting the issue corrected
Liam_Finn1
Level 6
16 years ago
Make sure you get a DTrace of the indexbroker service and the storagebroker service so we can see the details of the errors you are receiving when trying to rebuild
RBostwick
Level 3
16 years ago
Thanks for all the HELP and suggestions.

The Registry settings allowed the Index Update to continue and finish.

I am able search the Indexed Archives correctly now.

I will back up and remove Registry settings as the next step.
Deepak_W
Level 6
16 years ago
even i faced the same issue, solution for the same is as mentioned below...

follow below listed steps

1. open the EV console
2. right click on Archives
3. check index volume
4. check mark on the failed index volume
5. click on search
6. it will show the failed indexes in some time
7. right click those indexes and select Rebuild
8. after some time again perform steps 2-5 steps
9. failed index will be repaired and no error will be there

mark this as solution if this resolves your problem...
Liam_Finn1
Level 6
16 years ago
OK good

at least we got you back running

you now have items missing from your index because of the failure. We need to find the reason why they filed

When you did the rebuild did you run a dtrace on the indexbroker and storage services?

If not please setup a dtrace of these and try to repair the index. A repair will try to go out and insert the failed items into the index and the dtrace will show whats happening and give an idea as to why it is failing

Also any event logs you have would be helpful to dientify why the items failed

To see the number of items go to the the folder where the index is and fine a file called indexmissing.log this file will list the savesets that failed to be added to the index. A count of these will tell you how many items you are missing from your index and give you an idea as to how big the issue is