Forum Discussion

Next012's avatar
Next012
Level 3
11 years ago

CA Random Sampling (Last Sample) Not Progressing

Hello all,

 

This issue started happening a week or so back and we've tried a number of the resolutions available through Symantec's knowledge base that generally work after a restart of the database, Windows updates gets turned back on and restarts the system at 3 AM (during processing) or otherwise and none have kickstarted the Last Sample metric again.  The database tblConfig appears to be stuck with Sampling : Status - Started and the date for Sampling : Last Sample remains the same.  The odd part is that the Sampling : Last First Pass Sample metric is consistently updated?

We've set the thread count lower, changed the cache timeouts, adjusted the Last Sample date back one day, restarted all connected systems but still, the random sampling never seems to start back up again appropriately.

 

The Indexing service on the CA server appears to not be started and has an error when attempting to start - does this tie into the Random Sampling system?

 

Is there any other way to kickstart the Sampling : Last Sample or Random Sampling in general via script or otherwise?

 

Enterprise Vault 9.0.1

Compliance Accelerator 9

 

Thank you!

Compliance Accelerator
Information Governance
Windows Server (2003-2008)
  • Kenneth_Adams's avatar
    Kenneth_Adams
    11 years ago

    I'm looking at the dtrace and see it was run on machine remora.  I presume this is the EV server as there are no lines with AcceleratorService threads.

    Please run the dtrace from the CA server's Command Prompt as follows:

    1. Log onto the CA server as the Vault Service Account (VSA) if you've logged on as some other account.
    2. Open a Command Prompt.
    3. Change the location within the Command Prompt to the EV installation folder (default location on 64-bit OS is 'C:\Program Files (x86)\Enterprise Vault' or on 32-bit OS is 'C:\Program Files\Enterprise Vault').
    4. Type dtrace at the Command Prompt and press the Enter key.
    5. At the 'DT>' prompt, type the letter v and press the Enter key.
    6. Scroll up throgh the listing of processes as needed to locate the AcceleratorService process and note the number associated to it (typically, this would be the number 2).  Note that will have to manually type the process name at the next prompt if you don't use the 'v' (for View) option to assign a shortcut process number to each process.
    7. At the 'DT>' prompt, type 'set 2 v', without the single quotes and replacing the number 2 with the number associated with the AcceleratorService process if that number is other than 2.
    8. Press the Enter key to complete the process logging selection.
    9. At the 'DT>' prompt, type 'log C:\Logs\AcceleratorServiceSamplingIssue.log', without the single quotes and using the drive letter, folder path and file name of your choice (we just ask that you have the extension as '.log').
    10. Press the Enter key to complete the log file specification and start the actual logging.
    11. At the 'DT>' prompt, type 'Mon' and press the Enter key to monitor the process log entries.
    12. Let the dtrace run for about 10 minutes, monitoring the log through use of the 'Mon' command.
    13. Look carefully as the information may scroll by quickly at times and slowly at others, but you'll be looking for Step 6, then Step 7.  If the processing gets to Step 7, it should complete soon (within a couple of hours or so, depending on the amount of data it has yet to process).
    14. If you see Step 7, you can stop the dtrace and delete it as we know the processing will complete.
    15. If you see any out of memory errors in Step 6, you'll need a Support ticket for us to help you clear the tables and reset Sampling.
    16. To stop the monitoring, just hold the Ctrl key and press the C key.  This combination may have to be tried a few times if the system is busy scrolling lines across the screen.
    17. To end the logging, just type 'log' at the 'DT>' prompt and press the Enter key twice.

    I'm hoping you'll report back that the Step 7 has appeared, but I'm ready to review the dtrace log if needed.

     

27 Replies

Sort By
Replies have been turned off for this discussion
  • EV_Ajay's avatar
    EV_Ajay
    Level 6
    11 years ago

    Hi,

    1. Could you please let me know whether CA Server & EV Server is on same box ?

    2. What's the version of EV Server including Service Pack ?

    3. What's the Version of CA Server including Service Pack ?

    4. The EV Binaries installed on CA Server should match the EV Binaries installed on EV Server.

    5. The CA Version should be equal or greater than EV Version ( It is recommended)

    6. All EV Servervices should be in running stage because when CA perform search / sampling it communicate with EV Server Services (Indexing & Storage) to collect the data.

     

  • Next012's avatar
    Next012
    Level 3
    11 years ago

    Hi Ajay,

    Thank you for your prompt response:

     

    1. Could you please let me know whether CA Server & EV Server is on same box ?

    CA is on a different box than the EV host (where Administration Console resides).  I do see an installation of EV on the CA server but it is not the primary host?

     

    2. What's the version of EV Server including Service Pack ?

    These values are straight from the console:

    EV 9.0 SP1

    File version: 9.0.1.1073

     

    3. What's the Version of CA Server including Service Pack ?

    These are from Add/Remove programs, Support Info, any other location where these details might be present?:

    Build 9.0.1.1039

    Version: 9.0.5135

     

    4. The EV Binaries installed on CA Server should match the EV Binaries installed on EV Server.

    The installation of EV on the CA server appears to be 8.0.3.1845.  As stated, this may need to be updated?

     

    5. The CA Version should be equal or greater than EV Version ( It is recommended)

    Per the CA Version above, this is accurate.

     

    6. All EV Servervices should be in running stage because when CA perform search / sampling it communicate with EV Server Services (Indexing & Storage) to collect the data.

    The Indexing service on the CA server may tie into the older EV instance that is present.  If this is the case, that might explain the lack of a starting service.

     

     

    Some back history on the installation as well.  This environment has been up and running for many years now.  The issue only recently started after restarts of the database server cluster due to updates in the worst time possible (during collection).  The systems were functioning without issue prior to the restart date though it was not noticed if all of the services were running on the CA server as uncovered now.

     

    If EV needs to be updated on the CA server, that should be easily accomplished.  If EV needs to be removed from the CA server, then that would be a next step.

     

     

    Thank you!

  • Kenneth_Adams's avatar
    Kenneth_Adams
    Level 6
    11 years ago

    Hello, Next012;

    In addition to the information requested by EV_Ajay, please also record a 10 minute long dtrace of the AcceleratorService on the CA server.  We need to see if you are getting any Out of Memory Errors during Step 6 of the Random Sampling processing.  If you are, then your solution will require a Support case to be opened as we will need to manipulate some information in the CA Customer database (set the sampling status and date, clear the sampling tables) and also run one or more catch-up searches to close the gap in the review set from when Random Sampling last completed successfully and now.

    One issue we've found with Random Sampling not finishing is having too many items to be parsed into the different Department review sets.  When this happens, we get an out of memory error in the AcceleratorService dtrace during Step 6 of the Random Sampling processing.

    Now, if you get this cleared and it happens again, you may need to break your Departments into smaller Departments (i.e., move some Monitored Employees from their current Department in to one or more new Departments).  We've found having a large number of Monitored Employees in a Department can cause these out of memory errors as there are just too many messages to hold in the 2 GB of RAM that Windows allocates to the AcceleratorService process.

  • Next012's avatar
    Next012
    Level 3
    11 years ago

    Hi Kenneth,

     

    I'll run a dTrace here once the servers are brought back up - service issues this morning :)

     

    As near as I can tell, the count for items waiting to be sampled is over 300k.  The issue was not caught immediately due to common fluctuations in CA review queue counts.  Thus it was caught after the listed 30k mark.  We journal roughly 50k emails per day so it has grown fast and will continue to grow.

     

    Thanks!

     

  • Next012's avatar
    Next012
    Level 3
    11 years ago

    I've successfully updated the EV instance that is on the CA server to 9.0.1.  Confirmed it is the same instance version as what is installed on the storage host.

  • Kenneth_Adams's avatar
    Kenneth_Adams
    Level 6
    11 years ago

    Hello again, Next012;

    Your update seems to have occurred while I was writing my post above.  Thank you for answering EV_Ajay's questions.  We now see what is likely wrong.

    You noted that the EV binaries on the CA server are version 8.0.3.1845. (EV 8.0 SP3), while the EV binaries on the EV server are 9.0.1.1073 (EV 9.0 SP1).  This binary version mismatch is likely causing the issue as the Step 1 in Random Sampling is to translate the information collected by the Journal Collector into the KVSSavesetID for each message.  With the EV binaries mismatched,that translation is not occurring properly.

    The EV binaries or EV Runtime API is required on the CA server in order to interface with the appropriate EV services on the EV server.  With the mismatch, searches will not always run, export will not always complete, and reviews may fail as CA can't communicate properly with the EV server.

    Please stop the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA server, upgrade the EV binaries to match your EV server (direct upgrade of EV from 8.0 SP3 to 9.0 SP1 is supported), set the Startup for any EV services other than EVAMS to Manual or Disabled, then reboot the CA server.  Note that you should only have the Enterprise Vault Admin Service and EVAMS installed on the CA server unless there are other EV functions (such as mailbox archiving or Vault Stores) configured on that server.  If you DO have other EV services installed on the CA server, you'll need to leave all EV services startup set to Automatic until such time as you can confirm none are needed on the CA server and uninstall them.

    After you've upgraded EV and rebooted the server, monitor the sampling processing by running a dtrace of the AcceleratrorService process and looking for Step 6 instances.  There should be multiple instances for each Department.  If you don't see any progress with the sampling processing, you'll need to open a Support ticket so we can reset the Sampling status and date to allow it to progress.

     

  • Kenneth_Adams's avatar
    Kenneth_Adams
    Level 6
    11 years ago

    Wow!  We're just crossing each other's messages.  I just saw where you've already upgraded the EV binaries on the CA server to match those of the EV server.  That's good.  Sampling should pick up and continue, but the dtrace will tell for sure.

    My concern is the KVSSavesetID translation may not have finished and we'll have to reset the Sampling status and time to get it to work.

    We look forward to your next update.

  • Next012's avatar
    Next012
    Level 3
    11 years ago

    I must've missed this response - now I understand your "crossing messages" comment :)

     

    Judging by the content of the dTrace, it is not collecting the data appropriately so I'm assuming it's running against the wrong category?  I'm also running the dTrace from our primary EV system via Administration Console.  I can run it from the CA server via command line if preferred.

  • Next012's avatar
    Next012
    Level 3
    11 years ago

    I've restarted the dTrace - please disregard the previous!

     

    Via command line dTrace, I've set against AcceleratorService and will upload the log once a good collection is taken.  It is cycling through and scrolling fast so I can see some parts are working appropriately.

     

    A quick question - if I set the Sampling start time and it has already started, if the time is adjusted again (assuming to restart the sampling), could that cause issues?

     

     

    Thanks!

  • Next012's avatar
    Next012
    Level 3
    11 years ago

    Thankfully it's a slow day for our Compliance staff so the system being offline for a little bit has been a blessing in diguise for them.

     

    I've started the dTrace but only against the Compliance Accelerator category.  I've also adjusted the start time for the sampling but could not start the dTrace prior to that start time.  I will upload the log once it's finished the 10 minutes lap, in fact it's finished.

     

    Edit: I've unlisted the attachment.