cancel
Showing results for 
Search instead for 
Did you mean: 

Have I just broken an index?

James_Slack
Level 6

Hi all,

I was getting a few errors about a few users having index errors in the event log so I though I would have a look.

This was the error I was getting for one of the users in question.

-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

so, what I did was at this point was check the folder mentioned and looked at the avtrace.log file, which was 0kb.

Then decided that as it was failed it must need a rebuild.

So in the EV Admin Console, I found the mailbox, went into the advanced tab and hit rebuild.

Checking the event log I got these two messages

-----------------------------------------------------------------------------------------------------------
Event 7294
-----------------------------------------------------------------------------------------------------------
Starting index volume update

Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:0 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Job Id: Vol_239
Job Author: MAG\vault.admin
Job Description: Rebuild Index
-----------------------------------------------------------------------------------------------------------

then

-----------------------------------------------------------------------------------------------------------
Event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: Open


Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.

For more information, see Help and Support Center at
-----------------------------------------------------------------------------------------------------------


So now, I thought I would check the avtrace file again to see if it had any info - but the folder was gone. Totally gone. That users Index folder has just vaporised.

Have I just made things worse for myslef?

Any ideas anyone, please?

I have about 7 users in this state and would ideally like to sort them all out.

Cheers

1 ACCEPTED SOLUTION

Accepted Solutions

Liam_Finn1
Level 6
Employee Accredited Certified
Not yet.

What will now happen is next time the services restart (during you backup) you can check the index folder to find a file called indexmissing.log. You can check now if you wish

This file will hold a list of items missing from the indexe

In the VAC you will see the list of indexes with the number of missing items. When you see this then do the repair on the index with the missing items.

EV will then go out, identify the items missing from the index and then it will attempt to add them to the index. The index will show as rebuilding during this time but it is only repairing not rebuilding so no need to panic

Once they are added you will see in the VAC that there are no items missing from the index. When you see this go back to the index directory and delete the indexmissing.log file.

That should bring your indexes up to date and back to a healthy state.

Here is a SQL query that can check the status in SQL if there are items missing from your indexes or not

I presently have SQL reporting services run this query twice a day and email me the results so I know if any items are missing from my indexes daily so I can perform a quick fix

Run this query against the Enterprisevault Directory database

SELECT FolderName AS 'Folder Name', Offline, Failed, Rebuilding, FailedItems, IndexRootPathEntryId
FROM IndexVolume
WHERE (FailedItems <> '') OR
(FailedItems <> '0') OR
(Offline = 'True') OR
(Failed = 'True') OR
(Rebuilding = 'True')





View solution in original post

36 REPLIES 36

Liam_Finn1
Level 6
Employee Accredited Certified
Let me check about the error you presently have

Do you have a backup of the index that you can restore to?

The first issue you saw looks like the checksum file in the index folder is out of sync with wjhat the database expects so all you needed to do is to stop the indexing service and i also stop the storage service just for my own level of comfort then i rename the checksum.dat file to something else and then restart the services. This would have recreated the checksum.dat file and the chances are your index would have come back online


If yuo have a backup let me know as there is a way to stop the index rebuilding and then we can restore the backup and restart the services. The index may then show missing items which can be updated or if the same error (first error) shows up you can perform the process i mentioned above

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified
The index folder is gone because you choose to do a rebuild.  The first step in a rebuild is to delete the index.

EV is telling you it is having problems accessing the path: Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\

Can you browse to this path from the EV server?

TonySterling
Moderator
Moderator
Partner    VIP    Accredited Certified
Also, for future reference always try the Repair or Update option first.  Rebuild should be your last resort.  :)

As scanner said, if you have a back up of the Index restore that to the location.

Cheers,

John_Chisari
Level 6
Partner Accredited
 The initial error you had was because of Invalid Checksum - as you can see from the event - ValidateFileChecksum

Technote on how to fix it for the other 6 (if it's the same) - http://support.veritas.com/docs/294109




Liam_Finn1
Level 6
Employee Accredited Certified
James,

Any update, did you manage to do the restore?

Did renaming the dat file fix the others that were giving the error

Liam

Nicola_Rositani
Level 5
James, any news?
I'm interested about solution

James_Slack
Level 6
Sorry, the update is no update.

I just got back to work from pig-flu... so lots of catching up to do.

I will have a read through this lots and see what happens, what I will say is thanks to everyone for posting!

Cheers - James 

James_Slack
Level 6
OK, For this one user I mentioned, I had a look in the event viewer and couldnt see any mention of him.

I opened up his mailbox properties and checked the location folder of his index and went there, loads of Archive IDs, had a look at this users Archive ID and found that folder, there are a bunch of files and all certainly looks well now.

So, for this user, I think I am done - does anyone know any differently?

For the other users that are having this issue

-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

I will just follow Scanners / Johns instruction to sort the checksum.

I'll try one user and let you know how I get on.

Many thanks

James

James_Slack
Level 6

Should this work straight away?

I just did exactly as you said for a user and while it has changed the message, it doesn't inspire confidence!

-----------------------------------------------------------------------------------------------------------
OLD event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: ValidateFileChecksum

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------
NEW event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: Open

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------

I will have a look through the link John sent to see what I can work out, but is this expected behaviour or does this indicate something more serious?

Many thanks

James

James_Slack
Level 6
Having read through the link John gave, it makes sence that I need to "update" it.

Solution:
To correct the failed index,
1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.

Also, I think this is all because of no shutdown process in place on the server. I am going to write a little BAT file to stop the services and restart the box.

Liam_Finn1
Level 6
Employee Accredited Certified
If you rename the Checksum.dat file and restart the services it will create a new checksum.dat

BUT

You will not be able to tun an update or repair because the index will show healthy

AND

Next day when you check the system you will see that there are items missing from the index. At this stage you do a repair not an update

James_Slack
Level 6
Other dramas over so I can spend the week looking at all my broken indexes.

I have16 users with dodgy indexes at the moment, and i have created a list to work on.

Here is what most are getting.

-----------------------------------------------------------------------------------------------------------
Event 7292

The index volume has been marked as failed.
Index Volume: 10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite/Volume:3631 (Victorino Ramirez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location06\10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

As I said, this is what most of the 16 users are getting. What shoudl I do next to fix these issues?


A guy who I tried an Update / repair on a while back however is getting these now:

-----------------------------------------------------------------------------------------------------------
Event 7264

Abnormal error occurred

Error: The AltaVista dataset file is missing. [0xc0041c7d]
Reference: CAltaVistaIndex/OAVI/ie
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info: H:\EVStorage\PRVAULTDB01\Indexes\Location03
14F77730E08A89146A3D6DCD135DA148C1110000vaultsite
-----------------------------------------------------------------------------------------------------------

followed by

-----------------------------------------------------------------------------------------------------------~
Event 7264

Abnormal error occurred

Error: The AltaVista index is not open [0xc0041c64]
Reference: ST/TR/gs
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info:
-----------------------------------------------------------------------------------------------------------

For this user, I did the following:

Found his mailbox in Vault, went into properties, in the Index Volumes tab, just right clicked and chose update / or repair (I dont remember which but this I have tried both settings on different users with the same results).

What I have done now is told this user to rebuild and will see what that yeilds.

If anyone has any advise for me on either of the 2 scenarios that would be great.

Thanks James


James_Slack
Level 6

Basically, I have been advised to do this if I get event 7292:

1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.

Is this correct? From what you guys have said above you seem to put me off.

If I don't have a backup, do I have any other options?

Cheers

 

Liam_Finn1
Level 6
Employee Accredited Certified
 James

the line item "Reference: ValidateFileChecksum" basically tells you that the checksum file shows a difference between that is on disk and what it is expecting

To correct this the steps are as follows
1. Stop the index service
2. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
3. Navigate to the folder listed in the Event ID 7292 description field.
4. Locate and rename the file Checksum.dat.
5. Start the index service
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Repair option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail. 

Also check the directory where the indexes reside  for a new checksum.dat file which will be created by the index service

As always if possible take a backup of the indexes before making any changes

James_Slack
Level 6
I have taken indexing offline, deleted the checksum file and restarted indexing.

When I went to have a look at the first users "Index Volume" tab in EV 2007, it was saying it was NORMAL.

Would I still need to do a REPAIR?

Cheers

James

Liam_Finn1
Level 6
Employee Accredited Certified
Not yet.

What will now happen is next time the services restart (during you backup) you can check the index folder to find a file called indexmissing.log. You can check now if you wish

This file will hold a list of items missing from the indexe

In the VAC you will see the list of indexes with the number of missing items. When you see this then do the repair on the index with the missing items.

EV will then go out, identify the items missing from the index and then it will attempt to add them to the index. The index will show as rebuilding during this time but it is only repairing not rebuilding so no need to panic

Once they are added you will see in the VAC that there are no items missing from the index. When you see this go back to the index directory and delete the indexmissing.log file.

That should bring your indexes up to date and back to a healthy state.

Here is a SQL query that can check the status in SQL if there are items missing from your indexes or not

I presently have SQL reporting services run this query twice a day and email me the results so I know if any items are missing from my indexes daily so I can perform a quick fix

Run this query against the Enterprisevault Directory database

SELECT FolderName AS 'Folder Name', Offline, Failed, Rebuilding, FailedItems, IndexRootPathEntryId
FROM IndexVolume
WHERE (FailedItems <> '') OR
(FailedItems <> '0') OR
(Offline = 'True') OR
(Failed = 'True') OR
(Rebuilding = 'True')





Liam_Finn1
Level 6
Employee Accredited Certified
 You may also find this one more helpful as it provides the name of the archive so it is easier to identify it in the VAC

I prefer this one to the one in the previous post

SELECT ArchiveName AS 'Archive', FolderName AS 'Folder', [Offline] AS 'Index Offline', Failed AS 'Index Failed', Rebuilding AS 'Index Rebuilding',
failedItems AS 'Failed Items'
FROM Archive a, IndexVolume iv
WHERE a.RootIdentity = iv.RootIdentity and
([Offline] = 'True' or Failed = 'True' or Rebuilding = 'True' or failedItems <> '')

James_Slack
Level 6

Thanks for this info.

I am having a read and learn about this now. I am currently struggling to find the indexmissing.log.

While looking where this log could be I stumbled across this item:
http://seer.entsupport.symantec.com/docs/280895.htm  which is "How to verify the integrity of indexes with the IndexCheck utility when using Enterprise Vault (EV)."

Cheers