EventID: 6651 Storage Deletion Server has failed

[Sean_Waite]

We recently upgraded to EV11 from EV10.0.4 - we have three EV servers and on one I am seeing the following 2 errors in the Event Log every 15 minutes:

1.
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          29/09/2014 11:30:09
Event ID:      13410
Task Category: Directory Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <snip>
Description:
An error was detected while accessing the Vault Database 'EnterpriseVaultDirectory' (Internal reference: {CADODataAccess::ExecuteSQLCommand} [.\ADODataAccess.cpp, lines {1320,1322,1341,1359}, built Apr 23 16:33:46 2014]), error code <0x80040e2f>.

Description: Value violated the integrity constraints for a column or table.

2.
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          29/09/2014 11:30:09
Event ID:      6651
Task Category: Storage Delete
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <snip>
Description:
Storage Deletion Server has failed
Reason:  <0x80040e2f>

No errors on the other two EV Servers. I ran a DTrace against the StorageDelete task and as far as I can see the only reference to the error is in the following part:

1473 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CVaultStoreDB::DeleteArchiveFolders(rDeletedFoldersXML: <?xml version="1.0" encoding="UTF-8" ?><EnterpriseVault><DeletedFolder VaultEntryId="1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01"/></EnterpriseVault>
1474 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CArchiveFolderMonitorRequest::Execute - tidying up deleted folders from directory for Vaultstore 19B255AD591E89C40843A11F6719C69BA1210000AMSEVAULT01. [DeleteFoldersResultsXML = <?xml version="1.0" encoding="UTF-8" ?><EnterpriseVault><DeletedFolder VaultEntryId="1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01"/></EnterpriseVault>]
1476 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
1477 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
1479 11:30:09.877  [9440] (StorageDelete) <13020> EV:H {CADODataAccess::GetParameterCount} (Exit) Status: [Exception]
1480 11:30:09.877  [9440] (StorageDelete) <13020> EV:H {CADODataAccess::DeleteParameter} (Exit) Status: [Exception]
1481 11:30:09.877  [9440] (StorageDelete) <13020> EV:H {CADODataAccess::ResetParameters} (Exit) Status: [Exception]
1482 11:30:09.877  [9440] (StorageDelete) <13020> EV~E Event ID: 6651 Storage Deletion Server has failed |Reason:  <0x80040e2f> |

Nothing else has changed, no archives have been deleted - any ideas?

Thanks
Sean

[GertjanA]

Hello,

You might need to call support to assist, but I would check this:

does this archive exist : 1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01

(in 11, i believe you can righclick archives, then find archive, paste value, if not press CTRL+SHIFT, then find archive)

Can you verify on this one server if you have registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\Storage\DeleteEmptyFolders

which value does that have?

[Sean_Waite]

Hi Gertjan,

I can confirm that the following archive is not found:

1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01

Why would that be?

Also, in the Registry, that entry does not exist either. Looking at the Registry Values PDF the description for this is:

When enabled and the storage service restarted, a new thread examines all vaults in all stores owned by that Storage Service. Empty folders in structured archives are deleted.

Special folders such as the following are never deleted: "Inbox", "Drafts", "Sent Items", "Outbox", "Deleted Items".

Note:
Resets to zero (disabled) when all stores and archives have been processed.

Would you recommend setting this registry value and restarting the Storage Service?

Thanks
Sean

[JesusWept3]

It sounds like it has an entry in the Vault table of the vaultstore database that does not exist in the Root table of the directory database

[Sean_Waite]

Thanks JesusWept3,

I checked those tables and there does appear to be corresponding entries:

From the Vault Table of the Vault Store Database:

VaultID VaultIdentity ArchivePointIdentity ArchivedItems ArchivedItemsSize Restores Recalls MeanRecallTime CreatedDate ModifiedDate DeletionRunDate
1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01 294 17 -1 -53 0 0 0 05:30.0 45:33.9 1753-01-01 00:00:00.000

From the Root Table of the DirectoryDatabase:

RootIdentity VaultEntryId Type ContainerRootIdentity OwningTrusteeIdentity Hidden ArchiveFolderIdentity AutoSecurityDescID ManualSecurityDescID
857 1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01 10 852 NULL NULL 5 108 NULL

Any other ideas?

Sean

[ChayDouglas]

Hi Sean,

I had a similar error on my closed lab environment. As it was not live, it allowed me to trial and error a few queries on SQL. 

Don't know if you found a solution, yourself, but after a series of sql queries I have worked out that the vaultID shown in the log was due to folders I had tried to delete for test users. Not sure why EV was having trouble deleting them, but I have managed to remove the SQL entries for them safely and the error has disappeared.

First query you could run is the following, that displays the folder name and the archive it is having trouble with:
 

select r.VaultEntryId, r.RootIdentity, af.FolderName, a.ArchiveName
from Root r, ArchiveFolder af, archive a
where r.VaultEntryId = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01'
AND af.RootIdentity = r.RootIdentity

Then you can investigate if this folder exists in the archive and/or if it's empty.

If you can confirm that this folder can be deleted, as was the case with me, you can remove the entries from SQL.

begin transaction
delete from ExchangeMbxArchiveFolder where RootID = '857'
delete from ArchiveFolder where RootIdentity = '857'
delete from ACE where RootIdentity = '857'
delete from Root where VaultEntryId = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01'
commit

Obviously, this is a SQL hack and I had the benefit of working in a closed environment with backups of the db for roll-back...

If the Select query provides multiple user archives, it may not be advised to proceed with the hack either.

I would definitely recommend using the select query to determine which folder EV is complaining about though.

Good Luck!

Cheers
Chay

[Sean_Waite · Accepted solution]

Hi Chay,

The error was due to a user deleting a folder folder their Vault but SQL still had an entry listed for the folder in the directory database. After contacting Symantec Support I was advised to run the following SQL commands on the database:

USE EnterpriseVaultDirectory
select * from dbo.ArchiveFolderViewWithDeletedFolders
WHERE VaultEntryID = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01' < the vault archive ID with the missing folder

USE EnterpriseVaultDirectory
UPDATE dbo.ArchiveFolderViewWithDeletedFolders
SET Deleted = '0'
WHERE VaultEntryID = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01'

This fixed the error.

Thanks,
Sean