Forum Discussion

dgoodyear's avatar
dgoodyear
Level 5
13 years ago
Solved

Index rebuilding insight needed

I have two index volumes for our journal mailbox that are in rebuilding states.  I can verify this by checking for "rebuilding" index items in the admin console.  These have been in a rebuilding state after the wrong option was selected in December, approximately 6-7 weeks ago. 

I've read through this article here for some insight on how long will my index rebuilds take: https://www-secure.symantec.com/connect/articles/how-long-will-my-index-take-rebuild

 

I used the sql commands at the end of the article to locate two paths that are currently rebuilding and took a look at the updates.log files for each.  It looks like it does not seem to be doing anything.  I see lines for today for example but nothing that seems to indicate that any rebuilding is going on.  Also, after a certain time, there are no new lines in the updates.log file for today.  However, they will show up the following day for a little while after the site processing schedule kicks in.  This applies to both locations that are rebuilding.

I copied the few lines from today for the updates.log.  Perhaps I am interpreting it wrong?

2012-02-15 07:06:26.618 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57373217 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 07:13:06.500 M Index commit (Makestable) Reason=Commit Request Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=30000 Elapsed(secs)=0.0
2012-02-15 07:13:09.891 S Update session ended Index commits(Makestables)=2 Index update commits(Makestables_DynamicAttributes)=0 Items added=0 Words added=0 Items Deleted=30000 Items Updated=0 Minor compactions=0 Minor compaction steps=0 Major Compactions=0 Major compaction steps=0 Elapsed(secs)=406.0
2012-02-15 07:13:44.298 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 07:14:34.580 C Compaction incomplete. Type=Major Steps=1 Time compacting(secs)=0.0 Elapsed(secs)=50.0
2012-02-15 07:14:34.580 S Update session ended Index commits(Makestables)=1 Index update commits(Makestables_DynamicAttributes)=0 Items added=0 Words added=0 Items Deleted=0 Items Updated=0 Minor compactions=0 Minor compaction steps=0 Major Compactions=0 Major compaction steps=1 Elapsed(secs)=50.0
2012-02-15 07:43:45.504 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 07:43:45.614 E Index Failed. Reference= Compact
2012-02-15 12:39:13.657 M Index commit (Makestable) Reason=Major Compaction Item Seq Num=20212374 Delete Seq Num=57438089 Items added=0 Words added=0 Items deleted=0 Elapsed(secs)=0.0
2012-02-15 12:49:33.747 C Compaction incomplete. Type=Major Steps=1 Time compacting(secs)=0.0 Elapsed(secs)=620.0
2012-02-15 12:49:33.747 S Update session ended Index commits(Makestables)=1 Index update commits(Makestables_DynamicAttributes)=0 Items added=0 Words added=0 Items Deleted=0 Items Updated=0 Minor compactions=0 Minor compaction steps=0 Major Compactions=0 Major compaction steps=1 Elapsed(secs)=623.0

 

One thing I notice is the line : 2012-02-15 07:43:45.614 E Index Failed. Reference= Compact
 When I check the event viewer for EV, I have the corresponding Event ID 7264 error:

Abnormal error occurred
 
Error:     Exception occurred.  (0x80020009)
Reference: CAVUpdator::CompactAVIndex
Index:     1BDD3912805B7D54EAB57B4F5420551431110000sev-amer.pwav.com/Volume:755 (SevJrnl01)
Info:      EM/e

 

I'm not sure if I'm experiencing an indexing problem with both of these currently rebuilding or if they even truly are rebuilding or not.  I've not seen any status updates in the event log for progress.  How does EV process indexing?  Does it round-robin all of the indexing requirements for rebuilds and newly archived mail? 

Thanks for any insight into this!

  • I opened a case with Symantec in the end.  It took a couple of weeks to resolve since it would take several days for some of the tools to run against our EV environment.  In the end, we had more than one issue with our environment.  The first issue was that the sql maintenance jobs for reindexing were not running properly and had not been in awhile.  Consequently, many of our sql tables had obscene levels of fragmentation.  That was resolved by stopping all of the EV services and allowing the sql maintenance jobs to run with EV services down.  Trying to run these "live" only resulted in the reindexing failing every time.  This did not fix our reindexing issue directly, but it could only help.

    What was curious about these two problem index volumes is that they had a high number of items indexed (300,000+ and 400,000+) each as I wrote in the previous post.  However, the date range for the indexed items was outside of our journal retention.  We only retain journal messages for 2 years, yet the date range for these two indexes was older than two years.  Definitely fishy.  (This will lead to a new forum post question after this relating to that heh). 

    In the end, we located the specific folder used by the problematic index volumes, stopped the EV services, and then renamed those two index folders to <foldername>_old.  Effectively we were removing it completely and forcing a hard rebuild.  There must be some low level difference in rebuilding through the admin console versus a rebuild by removing the index from the file system (apart from the obvious).  I guess a rebuild through the admin console does not completely wipe out the index, that or we had some sort of corruption in the file system for that index.    So, I started the EV services back up.  The journal reindexing went back into rebuild mode for those two locations since technically they no longer existed and within a few hours the rebuilds completed.  When I rechecked each volume, they each had less than 100 items in their index.  This seems to fit more appropriately with all of the other index volumes from around that time (all having just a few items in them). 

    We haven't had any problems with this since.