Enterprise Vault Accelerator Manager Service causes SQL to utilized 100% disk time.

[jason_lucy]

Please help.

Last week we deleted more than two million emails from DA.  On Monday, our EV SQL server was sluggish and when investigating we found that DA delete request was causing the drive to reach 100% utilization.  Here are the corresponding events when attempting to start the EV Accelerator Manager Service.

Event Type: Error
Event Source: Accelerator Service Processor
Event Category: None
Event ID: 93
Description:
APP AT - Customer ID: 1 - An error occurred while updating Vaults for Vault Store: ID=6 KVS ID: 1AF3F9AB203E4F0479537826A59689E0F1210000dhsv. System.Threading.ThreadAbortException: Thread was being aborted.
   at KVS.EnterpriseVault.Interop.IndexSearch2Class.ListIndexVolumeSets(String processingInstruction)
   at KVS.Accelerator.Application.Vaults.UpdateVaults(Int32 VaultStoreID, String KVSVaultStoreEntryID, Boolean bUpdateIndexServer)

Event Type: Error
Event Source: Accelerator Service Processor
Event Category: None
Event ID: 117
Description:
APP AT - Customer ID: 1 - Error adding Background SQL to the queue. System.Threading.ThreadAbortException: Thread was being aborted.
   at KVS.Accelerator.Common.BackgroundSQL.InitializeComponent()
   at KVS.Accelerator.Common.BackgroundSQL..ctor()
   at KVS.Accelerator.Common.BackgroundSQLTaskQueue.QueueWork(ThreadSafeQueue& theQueue, Object myData)

Event Type: Error
Event Source: Accelerator AD Synchronizer
Event Category: None
Event ID: 34
Description:
APP AT - Customer ID: 1 - An error occured in ProfileSynchroniser::SynchroniseEmployeeProfile while retrieving properties. System.Runtime.InteropServices.COMException (0x8007054B): The specified domain either does not exist or could not be contacted. (Exception from HRESULT: 0x8007054B)
   at ActiveDs.NameTranslateClass.Init(Int32 lnSetType, String bstrADsPath)
   at KVS.Accelerator.ActiveDirectory.ADProfileSynchroniser.SynchroniseEmployeeProfile(ProfileRow profileRow)

[JolaineY · Accepted solution]

Deleting archives, or numerous archived items can cause high utilization on the SQL servers that host the EV Vault Store databases, which may result in the behavior you're seeing, if the DA databases are on the same SQL servers.  The following article has instructions for creating an index on the EV Vault Store databases, which may help to alleviate the problem you're seeing.  After you have created the SQL index you should restart the EV Storage Service.

Excessive I/O may occur on SQL server when archive(s) are deleted from Enterprise Vault
http://support.veritas.com/docs/318493

 

[jason_lucy]

How does this apply when it is only related to Discovery Accelerator? 

The EV Vault Store databases appear to be fine.

[Liam_Finn1]

 Check in your Task manager and see how many copies of the delete task is running.

DON NOT STOP THEM just tell us how many are running

Also

Check your event logs to look for event 6604 and see if they are still being created


Also unless you have a very high power SQL server deleting 2million items at once is going to make your SQL server very busy.

I have deleted up to 3.5 million at once but i had 2 of the latest  quad core servers with 4GB per server and it took me 48 hours so unless your SQL is very high power this may take time

[jason_lucy]

Thanks.  I think that we are going to try to schedule this for off hours.  We already stopped the Discovery services and killed the job due because we keep getting a request with the same string in the WHERE clause:

SELECT ArchivePointId,ArchivePointIdentity,ArchivedItems,ArchivedItemsSize,

CreatedDate,HighestIndexSeqNo,IndexAddsPending,IndexDeletesPending,ModifiedDate

FROM ArchivePoint

WHERE ArchivePointId = '1AB24D6D22E0E5246AC391ED2C65CEC561110000dhsv'

Also the same string is in requests like this:

exec GetArchive N'1AB24D6D22E0E5246AC391ED2C65CEC561110000dhsv'

[Liam_Finn1]

 This query is a search for the archive point ID 1AB24D6D22E0E5246AC391ED2C65CEC561110000dhsv and trying to gather the following data "ArchivePointId,ArchivePointIdentity,ArchivedItems,ArchivedItemsSize,

CreatedDate,HighestIndexSeqNo,IndexAddsPending,IndexDeletesPending,ModifiedDate" from the Archive Point ID


I'm not sure if this is expiry.

did you check to see how many expiry tasks are running on the EV servers?

 

[Liam_Finn1]

Did you check to ensure you have plenty of disk space and memory in the SQL box

I ask because if your memory or hard drive are low on free capacity then  your server will page alot to disk and this will cause a backload in the processor and cause it to run high

Check your processes CPU usage to verify if it is sql

If you have 4GB of memory on the SQL server have you restricted within SQL how much memory SQL can use if you dont SQL will eat up all the memory it can to do its tasks and leave nothing for the OS

[jason_lucy]

How do I find out how many expiry tasks are running? 

[Liam_Finn1]

 In task manager, under processes sort it by name 

[jason_lucy]

Unfortunately I don't see any expiry tasks running.

[jason_lucy]

Unfortunately I don't see any expiry tasks running.

[Liam_Finn1]

Then expiry is not your issue

[Liam_Finn1]

Check performance monitor on the SQL server to see whats happening on the server

[JolaineY]

When you said that you  deleted 2 million items I assume you are talking about archived items, and those records are stored in the EV Vault Store databases.. not the DA databases. 

In my experience deleting a large # of archived items (and in turn a large # of records from the EV Vault store database) can have a significant impact on performance if the SQL index referenced in the technote does not exist.

[JolaineY]

ArchivePoint is a table in the EV Vault Store database, and does not exist in any DA databases that I am aware of.

[jason_lucy]

Not that there was a clear resolution, but here are a few things that we had to do.

In DA we lowered the chunks.  The SQL I/O was still at 100% disk util, however, it may have helped as the DELETE did complete and our SQL, DA, and EV servers are back to normal operation.