Highlighted

Have I just broken an index?

Hi all,

I was getting a few errors about a few users having index errors in the event log so I though I would have a look.

This was the error I was getting for one of the users in question.

-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

so, what I did was at this point was check the folder mentioned and looked at the avtrace.log file, which was 0kb.

Then decided that as it was failed it must need a rebuild.

So in the EV Admin Console, I found the mailbox, went into the advanced tab and hit rebuild.

Checking the event log I got these two messages

-----------------------------------------------------------------------------------------------------------
Event 7294
-----------------------------------------------------------------------------------------------------------
Starting index volume update

Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:0 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Job Id: Vol_239
Job Author: MAG\vault.admin
Job Description: Rebuild Index
-----------------------------------------------------------------------------------------------------------

then

-----------------------------------------------------------------------------------------------------------
Event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: Open


Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.

For more information, see Help and Support Center at
-----------------------------------------------------------------------------------------------------------


So now, I thought I would check the avtrace file again to see if it had any info - but the folder was gone. Totally gone. That users Index folder has just vaporised.

Have I just made things worse for myslef?

Any ideas anyone, please?

I have about 7 users in this state and would ideally like to sort them all out.

Cheers

1 Solution

Accepted Solutions
Accepted Solution!

Not yet. What will now happen

Not yet.

What will now happen is next time the services restart (during you backup) you can check the index folder to find a file called indexmissing.log. You can check now if you wish

This file will hold a list of items missing from the indexe

In the VAC you will see the list of indexes with the number of missing items. When you see this then do the repair on the index with the missing items.

EV will then go out, identify the items missing from the index and then it will attempt to add them to the index. The index will show as rebuilding during this time but it is only repairing not rebuilding so no need to panic

Once they are added you will see in the VAC that there are no items missing from the index. When you see this go back to the index directory and delete the indexmissing.log file.

That should bring your indexes up to date and back to a healthy state.

Here is a SQL query that can check the status in SQL if there are items missing from your indexes or not

I presently have SQL reporting services run this query twice a day and email me the results so I know if any items are missing from my indexes daily so I can perform a quick fix

Run this query against the Enterprisevault Directory database

SELECT FolderName AS 'Folder Name', Offline, Failed, Rebuilding, FailedItems, IndexRootPathEntryId
FROM IndexVolume
WHERE (FailedItems <> '') OR
(FailedItems <> '0') OR
(Offline = 'True') OR
(Failed = 'True') OR
(Rebuilding = 'True')





View solution in original post

36 Replies

Let me check about the error

Let me check about the error you presently have

Do you have a backup of the index that you can restore to?

The first issue you saw looks like the checksum file in the index folder is out of sync with wjhat the database expects so all you needed to do is to stop the indexing service and i also stop the storage service just for my own level of comfort then i rename the checksum.dat file to something else and then restart the services. This would have recreated the checksum.dat file and the chances are your index would have come back online


If yuo have a backup let me know as there is a way to stop the index rebuilding and then we can restore the backup and restart the services. The index may then show missing items which can be updated or if the same error (first error) shows up you can perform the process i mentioned above

The index folder is gone

The index folder is gone because you choose to do a rebuild.  The first step in a rebuild is to delete the index.

EV is telling you it is having problems accessing the path: Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\

Can you browse to this path from the EV server?

For the best O365 Reporting Solution click here Radar-Reporting

Also, for future reference

Also, for future reference always try the Repair or Update option first.  Rebuild should be your last resort.  Smiley Happy

As scanner said, if you have a back up of the Index restore that to the location.

Cheers,

For the best O365 Reporting Solution click here Radar-Reporting

 The initial error you had

 The initial error you had was because of Invalid Checksum - as you can see from the event - ValidateFileChecksum

Technote on how to fix it for the other 6 (if it's the same) - http://support.veritas.com/docs/294109




James, Any update, did you

James,

Any update, did you manage to do the restore?

Did renaming the dat file fix the others that were giving the error

Liam

James, any news? I'm

James, any news?
I'm interested about solution

Update

Sorry, the update is no update.

I just got back to work from pig-flu... so lots of catching up to do.

I will have a read through this lots and see what happens, what I will say is thanks to everyone for posting!

Cheers - James 

Proper Update

OK, For this one user I mentioned, I had a look in the event viewer and couldnt see any mention of him.

I opened up his mailbox properties and checked the location folder of his index and went there, loads of Archive IDs, had a look at this users Archive ID and found that folder, there are a bunch of files and all certainly looks well now.

So, for this user, I think I am done - does anyone know any differently?

For the other users that are having this issue

-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

I will just follow Scanners / Johns instruction to sort the checksum.

I'll try one user and let you know how I get on.

Many thanks

James

Scanner001...

Should this work straight away?

I just did exactly as you said for a user and while it has changed the message, it doesn't inspire confidence!

-----------------------------------------------------------------------------------------------------------
OLD event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: ValidateFileChecksum

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------
NEW event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: Open

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------

I will have a look through the link John sent to see what I can work out, but is this expected behaviour or does this indicate something more serious?

Many thanks

James

OK, rebuilding

Having read through the link John gave, it makes sence that I need to "update" it.

Solution:
To correct the failed index,
1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.

Also, I think this is all because of no shutdown process in place on the server. I am going to write a little BAT file to stop the services and restart the box.

If you rename the

If you rename the Checksum.dat file and restart the services it will create a new checksum.dat

BUT

You will not be able to tun an update or repair because the index will show healthy

AND

Next day when you check the system you will see that there are items missing from the index. At this stage you do a repair not an update

Back to this again

Other dramas over so I can spend the week looking at all my broken indexes.

I have16 users with dodgy indexes at the moment, and i have created a list to work on.

Here is what most are getting.

-----------------------------------------------------------------------------------------------------------
Event 7292

The index volume has been marked as failed.
Index Volume: 10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite/Volume:3631 (Victorino Ramirez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location06\10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

As I said, this is what most of the 16 users are getting. What shoudl I do next to fix these issues?


A guy who I tried an Update / repair on a while back however is getting these now:

-----------------------------------------------------------------------------------------------------------
Event 7264

Abnormal error occurred

Error: The AltaVista dataset file is missing. [0xc0041c7d]
Reference: CAltaVistaIndex/OAVI/ie
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info: H:\EVStorage\PRVAULTDB01\Indexes\Location03
14F77730E08A89146A3D6DCD135DA148C1110000vaultsite
-----------------------------------------------------------------------------------------------------------

followed by

-----------------------------------------------------------------------------------------------------------~
Event 7264

Abnormal error occurred

Error: The AltaVista index is not open [0xc0041c64]
Reference: ST/TR/gs
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info:
-----------------------------------------------------------------------------------------------------------

For this user, I did the following:

Found his mailbox in Vault, went into properties, in the Index Volumes tab, just right clicked and chose update / or repair (I dont remember which but this I have tried both settings on different users with the same results).

What I have done now is told this user to rebuild and will see what that yeilds.

If anyone has any advise for me on either of the 2 scenarios that would be great.

Thanks James


To clarify...

Basically, I have been advised to do this if I get event 7292:

1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.

Is this correct? From what you guys have said above you seem to put me off.

If I don't have a backup, do I have any other options?

Cheers

 

 James the line item

 James

the line item "Reference: ValidateFileChecksum" basically tells you that the checksum file shows a difference between that is on disk and what it is expecting

To correct this the steps are as follows
1. Stop the index service
2. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
3. Navigate to the folder listed in the Event ID 7292 description field.
4. Locate and rename the file Checksum.dat.
5. Start the index service
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Repair option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail. 

Also check the directory where the indexes reside  for a new checksum.dat file which will be created by the index service

As always if possible take a backup of the indexes before making any changes

Thanks Scanner

I have taken indexing offline, deleted the checksum file and restarted indexing.

When I went to have a look at the first users "Index Volume" tab in EV 2007, it was saying it was NORMAL.

Would I still need to do a REPAIR?

Cheers

James
Accepted Solution!

Not yet. What will now happen

Not yet.

What will now happen is next time the services restart (during you backup) you can check the index folder to find a file called indexmissing.log. You can check now if you wish

This file will hold a list of items missing from the indexe

In the VAC you will see the list of indexes with the number of missing items. When you see this then do the repair on the index with the missing items.

EV will then go out, identify the items missing from the index and then it will attempt to add them to the index. The index will show as rebuilding during this time but it is only repairing not rebuilding so no need to panic

Once they are added you will see in the VAC that there are no items missing from the index. When you see this go back to the index directory and delete the indexmissing.log file.

That should bring your indexes up to date and back to a healthy state.

Here is a SQL query that can check the status in SQL if there are items missing from your indexes or not

I presently have SQL reporting services run this query twice a day and email me the results so I know if any items are missing from my indexes daily so I can perform a quick fix

Run this query against the Enterprisevault Directory database

SELECT FolderName AS 'Folder Name', Offline, Failed, Rebuilding, FailedItems, IndexRootPathEntryId
FROM IndexVolume
WHERE (FailedItems <> '') OR
(FailedItems <> '0') OR
(Offline = 'True') OR
(Failed = 'True') OR
(Rebuilding = 'True')





View solution in original post

 You may also find this one

 You may also find this one more helpful as it provides the name of the archive so it is easier to identify it in the VAC

I prefer this one to the one in the previous post

SELECT ArchiveName AS 'Archive', FolderName AS 'Folder', [Offline] AS 'Index Offline', Failed AS 'Index Failed', Rebuilding AS 'Index Rebuilding',
failedItems AS 'Failed Items'
FROM Archive a, IndexVolume iv
WHERE a.RootIdentity = iv.RootIdentity and
([Offline] = 'True' or Failed = 'True' or Rebuilding = 'True' or failedItems <> '')

Thanks again Scanner

Thanks for this info.

I am having a read and learn about this now. I am currently struggling to find the indexmissing.log.

While looking where this log could be I stumbled across this item:
http://seer.entsupport.symantec.com/docs/280895.htm  which is "How to verify the integrity of indexes with the IndexCheck utility when using Enterprise Vault (EV)."

Cheers

This is also pretty useful info