Forum Discussion

Cdee's avatar
Cdee
Level 6
12 years ago

Compliance Acclerator Searches goes in hung state.

Hi Guys,

I have come across a small issue.

My compliance team is trying to perofrm a search on each dept having 5 searches in each dept and 1000 hotwords in it.

CA server version 9.0.2
CA Client Version 9.0.2
No of Dept           8
No of searches in each dept at a same time 5

So when the compliance guys try to perfrom a search in the system the searches go in never ending loop and does not complete at all.

Is there a limit on number of hotwords used in a search criteria, secondly is there a way i can fine tune the server in order that the searches compelte.

Compliance Accelerator
Information Governance
Tip-How to
Troubleshooting
Windows Server (2003-2008)
  • Kenneth_Adams's avatar
    Kenneth_Adams
    12 years ago

    While Scott's SQL query will report on all searches in a 'Searching', 'Accepting...', or 'Pending Acceptance (Errors)' status, just restarting the Enterprise Vault Accelerator Manager Service (EVAMS) nightly may not cause stuck searches to complete.  It may, but it also may not depending on where the issue causing the stuck condition resides.

    In your CA environment (just like in any other CA or DA 9 environment), there are resources on the EV Indexing Server that may cause a search to become hung.  Just restarting EVAMS won't clear those resources.  Restarting the EV Indexing Service may clear them, or a reboot of the EV Indexing Service server may be needed.  It really depends on the actual root cause of the 'stuck' condition.

    There are also other issues in the CA environment that could cause searches to become 'stuck' that restarting EVAMS won't clear.  For example. if an archive has had an adjustment made to its index volumes - say the AVSMaxLoc registry value was set to a higher value than previously set and the archive's index rebuilt to result in fewer index volumes - EV could have a different number of Index Volumes for an archive than CA has.  In such cases, a CA search would be looking for more index volumes to search than actually exists.  Such a condition would result in CA showing all index volumes as being finished, but the search never completes because it is awaiting the return from non-existent index volumes.  (Note that this condition requires stopping EVAMS, truncating the tblIndexVolumeSet table in the CA Customer database, starting EVAMS and letting the archive synchronization process repopulate the tblIndexVolumeSet table with the current index volumes in EV.)

    If you've optimized your environment per TECH56172 as I recommended in my first post AND you're still having CA searches become stuck, please open a support case so we can look into why those searches are becoming stuck.

     

  • Kenneth_Adams's avatar
    Kenneth_Adams
    Level 6
    12 years ago

    Hello, Cdee;

    If you've not already done so, please go through TECH56172, "Recommended steps to optimize performance on Enterprise Vault (EV), Compliance Accelerator (CA), Discovery Accelerator (DA), and SQL Servers in an EV environment", available at http://www.symantec.com/docs/TECH56172.  This is for all versions of EV, CA, and DA, but there are notes that some settings are not appropriate for 64-bit OS, so note those as appropriate.

    Technically, there is no programmed limit to the number of hot words or hot phrases a search criteria can hold.  There is, however, a pratical limit based on the available resources of the DA, EV, and SQL servers.  Wild cards in the hotwords add a lot of complexity to the criteria, too, to the use of wild cards should be kept to the mimimun necessary.

    Even after implementing the optimizations recommended in TECH56172, you may still have to break the searches into smaller sets of criteria.  Each index volume is searched by an instance of the IndexServer process.  Each IndexServer process is allocated a maximum of 2 GB  of RAM for its use.  Into this 2 GB of RAM must fit the search criteria, the index volume contents, and the search hits.  That's a lot to fit into 2 GB of RAM, which also tends to become unstable after reaching 1.5 GB used.

    If the physical size on disk of your index volumes is over 3 GB, you may need to implement the AVSMaxLoc registry value (set to 500 million decimal) and rebuild the indexes of the archives being searched.  Hopefully, you don't have to do this as the rebuild would prevent any searches against the archives as they are rebuilding.

    I hope you find this information helpful.

     

  • Scott__Hastings's avatar
    Scott__Hastings
    Level 5
    12 years ago

    At the end of the day, I think you'll always have searches in Searching, Accepting, and, Pending Acceptance (Errors).

     

    This is a really problem because searches that are "stuck" don't go into review sets on a timely basis.

     

    I would do 2 things.

     

    1) Run the following query against the CA Customer database to check for these items"

    SELECT tc.CaseID, tc.Name 'Case/Dept Name'

    , tis.SearchID, tis.Name 'Search Name', tis.StatusID, tis.RunDate

    , ts1.Name 'Search Status'

    FROM tblIntSearches tis

    JOIN tblCase tc ON tis.CaseID = tc.CaseID

    JOIN tblStatus ts1 ON tis.StatusID = ts1.StatusID

    WHERE tis.StatusID IN (2, 6, 10)

     

    2) Create a Task that restarts the Accelerator Service each day. This will clear any searches that are Searching or Accepting. Pending Acceptance w Errors need to be resubmitted.

  • Kenneth_Adams's avatar
    Kenneth_Adams
    Level 6
    12 years ago

    While Scott's SQL query will report on all searches in a 'Searching', 'Accepting...', or 'Pending Acceptance (Errors)' status, just restarting the Enterprise Vault Accelerator Manager Service (EVAMS) nightly may not cause stuck searches to complete.  It may, but it also may not depending on where the issue causing the stuck condition resides.

    In your CA environment (just like in any other CA or DA 9 environment), there are resources on the EV Indexing Server that may cause a search to become hung.  Just restarting EVAMS won't clear those resources.  Restarting the EV Indexing Service may clear them, or a reboot of the EV Indexing Service server may be needed.  It really depends on the actual root cause of the 'stuck' condition.

    There are also other issues in the CA environment that could cause searches to become 'stuck' that restarting EVAMS won't clear.  For example. if an archive has had an adjustment made to its index volumes - say the AVSMaxLoc registry value was set to a higher value than previously set and the archive's index rebuilt to result in fewer index volumes - EV could have a different number of Index Volumes for an archive than CA has.  In such cases, a CA search would be looking for more index volumes to search than actually exists.  Such a condition would result in CA showing all index volumes as being finished, but the search never completes because it is awaiting the return from non-existent index volumes.  (Note that this condition requires stopping EVAMS, truncating the tblIndexVolumeSet table in the CA Customer database, starting EVAMS and letting the archive synchronization process repopulate the tblIndexVolumeSet table with the current index volumes in EV.)

    If you've optimized your environment per TECH56172 as I recommended in my first post AND you're still having CA searches become stuck, please open a support case so we can look into why those searches are becoming stuck.

     

  • Scott__Hastings's avatar
    Scott__Hastings
    Level 5
    12 years ago

    Kenneth, I optimized my environment as per TECH56172 and still had trouble. I was simply sharing my  experience and how I was able to resolve it. I certainly didn't intend to contradict you.

    Scott

     

  • Kenneth_Adams's avatar
    Kenneth_Adams
    Level 6
    12 years ago

    Hi, Scott;

    I understand you were not trying to contradict me.  I just wanted Cdee to understand that just restarting EVAMS may not allow the hung searches to complete.  In your environment it does, but you should not have to restart EVAMS nightly just to get searches to complete unless your SQL Server is loosing network connectivity with your CA server.  We must have continuous connectivity between the CA and SQL servers in order to update the search with hits and status changes.  If that connection breaks, it would throw Event ID 158 errors in the CA server's Symantec Enterprise Vault Event Log every 10 minutes for each CA Customer (including Custodian Manager) that you have running.

  • Scott__Hastings's avatar
    Scott__Hastings
    Level 5
    12 years ago

    Right Kenneth.. I have a hard time getting SQL DBA's to do proper maintenance, and SQL Servers are usually a bit overloaded. Sometime back  (at a very, very large company) you guys advised me to disable TCP Chimney and TCPIP Offload Engine.

    The customer was very resistant to this, but when I finally convinced them, the issue was immediately resolved.

     

    I was just tossing in a little "real world"

     

    Thanks!

     

  • Kenneth_Adams's avatar
    Kenneth_Adams
    Level 6
    12 years ago

    Ah.  Another example of following our advice to see the issue resolved.  ;-)

    Thank you for sharing that information, Scott.

  • EV_Ajay's avatar
    EV_Ajay
    Level 6
    12 years ago

    Hi,

    Do you have any updates on this thread? Do you need more assistance regarding this topic? Please mark the post that best solves your problem as the answer to this thread.