cancel
Showing results for 
Search instead for 
Did you mean: 

Index rebuilding insight needed

dgoodyear
Level 5

I have two index volumes for our journal mailbox that are in rebuilding states.  I can verify this by checking for "rebuilding" index items in the admin console.  These have been in a rebuilding state after the wrong option was selected in December, approximately 6-7 weeks ago. 

I've read through this article here for some insight on how long will my index rebuilds take: https://www-secure.symantec.com/connect/articles/how-long-will-my-index-take-rebuild

 

I used the sql commands at the end of the article to locate two paths that are currently rebuilding and took a look at the updates.log files for each.  It looks like it does not seem to be doing anything.  I see lines for today for example but nothing that seems to indicate that any rebuilding is going on.  Also, after a certain time, there are no new lines in the updates.log file for today.  However, they will show up the following day for a little while after the site processing schedule kicks in.  This applies to both locations that are rebuilding.

I copied the few lines from today for the updates.log.  Perhaps I am interpreting it wrong?

2012-02-15 07:06:26.618 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57373217 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 07:13:06.500 M Index commit (Makestable) Reason=Commit Request Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=30000 Elapsed(secs)=0.0
2012-02-15 07:13:09.891 S Update session ended Index commits(Makestables)=2 Index update commits(Makestables_DynamicAttributes)=0 Items added=0 Words added=0 Items Deleted=30000 Items Updated=0 Minor compactions=0 Minor compaction steps=0 Major Compactions=0 Major compaction steps=0 Elapsed(secs)=406.0
2012-02-15 07:13:44.298 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 07:14:34.580 C Compaction incomplete. Type=Major Steps=1 Time compacting(secs)=0.0 Elapsed(secs)=50.0
2012-02-15 07:14:34.580 S Update session ended Index commits(Makestables)=1 Index update commits(Makestables_DynamicAttributes)=0 Items added=0 Words added=0 Items Deleted=0 Items Updated=0 Minor compactions=0 Minor compaction steps=0 Major Compactions=0 Major compaction steps=1 Elapsed(secs)=50.0
2012-02-15 07:43:45.504 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 07:43:45.614 E Index Failed. Reference= Compact
2012-02-15 12:39:13.657 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 12:49:33.747 C Compaction incomplete. Type=Major Steps=1 Time compacting(secs)=0.0 Elapsed(secs)=620.0
2012-02-15 12:49:33.747 S Update session ended Index commits(Makestables)=1 Index update commits(Makestables_DynamicAttributes)=0 Items added=0 Words added=0 Items Deleted=0 Items Updated=0 Minor compactions=0 Minor compaction steps=0 Major Compactions=0 Major compaction steps=1 Elapsed(secs)=623.0

 

One thing I notice is the line : 2012-02-15 07:43:45.614 E Index Failed. Reference= Compact
 When I check the event viewer for EV, I have the corresponding Event ID 7264 error:

Abnormal error occurred
 
Error:     Exception occurred.  (0x80020009)
Reference: CAVUpdator::CompactAVIndex
Index:     1BDD3912805B7D54EAB57B4F5420551431110000sev-amer.pwav.com/Volume:755 (SevJrnl01)
Info:      EM/e

 

I'm not sure if I'm experiencing an indexing problem with both of these currently rebuilding or if they even truly are rebuilding or not.  I've not seen any status updates in the event log for progress.  How does EV process indexing?  Does it round-robin all of the indexing requirements for rebuilds and newly archived mail? 

Thanks for any insight into this!

1 ACCEPTED SOLUTION

Accepted Solutions

dgoodyear
Level 5

I opened a case with Symantec in the end.  It took a couple of weeks to resolve since it would take several days for some of the tools to run against our EV environment.  In the end, we had more than one issue with our environment.  The first issue was that the sql maintenance jobs for reindexing were not running properly and had not been in awhile.  Consequently, many of our sql tables had obscene levels of fragmentation.  That was resolved by stopping all of the EV services and allowing the sql maintenance jobs to run with EV services down.  Trying to run these "live" only resulted in the reindexing failing every time.  This did not fix our reindexing issue directly, but it could only help.

What was curious about these two problem index volumes is that they had a high number of items indexed (300,000+ and 400,000+) each as I wrote in the previous post.  However, the date range for the indexed items was outside of our journal retention.  We only retain journal messages for 2 years, yet the date range for these two indexes was older than two years.  Definitely fishy.  (This will lead to a new forum post question after this relating to that heh). 

In the end, we located the specific folder used by the problematic index volumes, stopped the EV services, and then renamed those two index folders to <foldername>_old.  Effectively we were removing it completely and forcing a hard rebuild.  There must be some low level difference in rebuilding through the admin console versus a rebuild by removing the index from the file system (apart from the obvious).  I guess a rebuild through the admin console does not completely wipe out the index, that or we had some sort of corruption in the file system for that index.    So, I started the EV services back up.  The journal reindexing went back into rebuild mode for those two locations since technically they no longer existed and within a few hours the rebuilds completed.  When I rechecked each volume, they each had less than 100 items in their index.  This seems to fit more appropriately with all of the other index volumes from around that time (all having just a few items in them). 

We haven't had any problems with this since. 

View solution in original post

11 REPLIES 11

JesusWept3
Level 6
Partner Accredited Certified

Hmm usually if the index fails due to compaction, or fails during the compact it typically means that there is something holding the files open , most likely the Anti-Virus may not be excluded from it

So on the queries i wrote at the bottom, do you know how many items it *has* succesfully indexed?
and this is a journal index too right? so you've most likely got millions upon millions of items to index.

Also what version of Enterprise Vault are we talking about here?

https://www.linkedin.com/in/alex-allen-turl-07370146

JesusWept3
Level 6
Partner Accredited Certified

Also ten minutes to perform a compaction or fail at it? how big is the index volume physically on the drive?

https://www.linkedin.com/in/alex-allen-turl-07370146

dgoodyear
Level 5

Oops.. forgot to mention that we are currently using Enterprise Vault 9 SP2. 

 

Our entire Index is about 162gb.  The index volume folders in question are:

Index3 = 23gb

Index6 = 20gb

 

The first one is a subfolder of Index3, the second line is a subfolder of Index6.  I couldn't work out the formatting here well enough not to make it annoying to read. 

firstisn lastisn indexeditems rebuilding Failed Faileditems
19742307 20212374 365906 1 0 0
20212375 20721688 490424 1 0

 

Our entire journal archive on this server has about 17.5 million items in it.

dgoodyear
Level 5

Our antivirus has the correct exclusions for the Index that I verified.  But just to be safe I unloaded it from memory temporarily.

CFreeX
Level 4

This doesn't seems like a common index issue. Do you have a dtrace? We may even need to enable the AVTrace logging (http://www.symantec.com/business/support/index?page=content&id=TECH65258). Perhaps, it maybe best that you to open a support case.

dgoodyear
Level 5

I do not see a lot of dtrace activity that seems to suggest much.  I've opened a support case on the issue.

Prone2Typos
Moderator
Moderator
Partner    VIP    Accredited Certified

THis does seem strange.

 

I was just wondering if you have opertunisitic locking enabled on the EV server and/or the index server location. Also the AvTrace log shoudl ssay if something happened during the compaction. Typically you do not see failures during compaction that do not reference 'open'

Prone2Typos
Moderator
Moderator
Partner    VIP    Accredited Certified

Just wondering where the status of this was and if a solution was found if you could flag it as such and share it with us so we know. . .Many thanks.

Prone2Typos
Moderator
Moderator
Partner    VIP    Accredited Certified

Just wanted to check if this was resolved... if so please flag it as such. It seems your posting and your index are expirencing the same issues.....wink

dgoodyear
Level 5

I opened a case with Symantec in the end.  It took a couple of weeks to resolve since it would take several days for some of the tools to run against our EV environment.  In the end, we had more than one issue with our environment.  The first issue was that the sql maintenance jobs for reindexing were not running properly and had not been in awhile.  Consequently, many of our sql tables had obscene levels of fragmentation.  That was resolved by stopping all of the EV services and allowing the sql maintenance jobs to run with EV services down.  Trying to run these "live" only resulted in the reindexing failing every time.  This did not fix our reindexing issue directly, but it could only help.

What was curious about these two problem index volumes is that they had a high number of items indexed (300,000+ and 400,000+) each as I wrote in the previous post.  However, the date range for the indexed items was outside of our journal retention.  We only retain journal messages for 2 years, yet the date range for these two indexes was older than two years.  Definitely fishy.  (This will lead to a new forum post question after this relating to that heh). 

In the end, we located the specific folder used by the problematic index volumes, stopped the EV services, and then renamed those two index folders to <foldername>_old.  Effectively we were removing it completely and forcing a hard rebuild.  There must be some low level difference in rebuilding through the admin console versus a rebuild by removing the index from the file system (apart from the obvious).  I guess a rebuild through the admin console does not completely wipe out the index, that or we had some sort of corruption in the file system for that index.    So, I started the EV services back up.  The journal reindexing went back into rebuild mode for those two locations since technically they no longer existed and within a few hours the rebuilds completed.  When I rechecked each volume, they each had less than 100 items in their index.  This seems to fit more appropriately with all of the other index volumes from around that time (all having just a few items in them). 

We haven't had any problems with this since. 

Prone2Typos
Moderator
Moderator
Partner    VIP    Accredited Certified

Thanks for the very complete update ... great information