Forum Discussion

Sean_Waite's avatar
Sean_Waite
Level 3
11 years ago
Solved

EventID: 6651 Storage Deletion Server has failed

We recently upgraded to EV11 from EV10.0.4 - we have three EV servers and on one I am seeing the following 2 errors in the Event Log every 15 minutes:

1.
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          29/09/2014 11:30:09
Event ID:      13410
Task Category: Directory Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <snip>
Description:
An error was detected while accessing the Vault Database 'EnterpriseVaultDirectory' (Internal reference: {CADODataAccess::ExecuteSQLCommand} [.\ADODataAccess.cpp, lines {1320,1322,1341,1359}, built Apr 23 16:33:46 2014]), error code <0x80040e2f>.

Description: Value violated the integrity constraints for a column or table.

2.
Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          29/09/2014 11:30:09
Event ID:      6651
Task Category: Storage Delete
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <snip>
Description:
Storage Deletion Server has failed
Reason:  <0x80040e2f>

No errors on the other two EV Servers. I ran a DTrace against the StorageDelete task and as far as I can see the only reference to the error is in the following part:

1473 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CVaultStoreDB::DeleteArchiveFolders(rDeletedFoldersXML: <?xml version="1.0" encoding="UTF-8" ?><EnterpriseVault><DeletedFolder VaultEntryId="1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01"/></EnterpriseVault>
1474 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CArchiveFolderMonitorRequest::Execute - tidying up deleted folders from directory for Vaultstore 19B255AD591E89C40843A11F6719C69BA1210000AMSEVAULT01. [DeleteFoldersResultsXML = <?xml version="1.0" encoding="UTF-8" ?><EnterpriseVault><DeletedFolder VaultEntryId="1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01"/></EnterpriseVault>]
1476 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
1477 11:30:09.846  [9440] (StorageDelete) <13020> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
1479 11:30:09.877  [9440] (StorageDelete) <13020> EV:H {CADODataAccess::GetParameterCount} (Exit) Status: [Exception]
1480 11:30:09.877  [9440] (StorageDelete) <13020> EV:H {CADODataAccess::DeleteParameter} (Exit) Status: [Exception]
1481 11:30:09.877  [9440] (StorageDelete) <13020> EV:H {CADODataAccess::ResetParameters} (Exit) Status: [Exception]
1482 11:30:09.877  [9440] (StorageDelete) <13020> EV~E Event ID: 6651 Storage Deletion Server has failed |Reason:  <0x80040e2f> |

Nothing else has changed, no archives have been deleted - any ideas?

Thanks
Sean

 

Enterprise Vault
Error messages
Information Governance
Troubleshooting
Windows Server (2003-2008)
  • Sean_Waite's avatar
    Sean_Waite
    11 years ago

    Hi Chay,

    The error was due to a user deleting a folder folder their Vault but SQL still had an entry listed for the folder in the directory database. After contacting Symantec Support I was advised to run the following SQL commands on the database:

    USE EnterpriseVaultDirectory
    select * from dbo.ArchiveFolderViewWithDeletedFolders
    WHERE VaultEntryID = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01' < the vault archive ID with the missing folder

    USE EnterpriseVaultDirectory
    UPDATE dbo.ArchiveFolderViewWithDeletedFolders
    SET Deleted = '0'
    WHERE VaultEntryID = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01'

    This fixed the error.

    Thanks,
    Sean

  • GertjanA's avatar
    GertjanA
    Moderator
    11 years ago

    Hello,

    You might need to call support to assist, but I would check this:

    does this archive exist : 1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01

    (in 11, i believe you can righclick archives, then find archive, paste value, if not press CTRL+SHIFT, then find archive)

    Can you verify on this one server if you have registry key:

    HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\Storage\DeleteEmptyFolders

    which value does that have?

     

  • Sean_Waite's avatar
    Sean_Waite
    Level 3
    11 years ago

    Hi Gertjan,

    I can confirm that the following archive is not found:

    1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01

    Why would that be?

    Also, in the Registry, that entry does not exist either. Looking at the Registry Values PDF the description for this is:

    When enabled and the storage service restarted, a new thread examines all vaults in all stores owned by that Storage Service. Empty folders in structured archives are deleted.

    Special folders such as the following are never deleted: "Inbox", "Drafts", "Sent Items", "Outbox", "Deleted Items".

    Note:
    Resets to zero (disabled) when all stores and archives have been processed.

    Would you recommend setting this registry value and restarting the Storage Service?

    Thanks
    Sean

  • JesusWept3's avatar
    JesusWept3
    Level 6
    11 years ago
    It sounds like it has an entry in the Vault table of the vaultstore database that does not exist in the Root table of the directory database
  • Sean_Waite's avatar
    Sean_Waite
    Level 3
    11 years ago

    Thanks JesusWept3,

     

    I checked those tables and there does appear to be corresponding entries:

    From the Vault Table of the Vault Store Database:

    VaultID VaultIdentity ArchivePointIdentity ArchivedItems ArchivedItemsSize Restores Recalls MeanRecallTime CreatedDate ModifiedDate DeletionRunDate
    1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01 294 17 -1 -53 0 0 0 05:30.0 45:33.9 1753-01-01 00:00:00.000


    From the Root Table of the DirectoryDatabase:

    RootIdentity VaultEntryId Type ContainerRootIdentity OwningTrusteeIdentity Hidden ArchiveFolderIdentity AutoSecurityDescID ManualSecurityDescID
    857 1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01 10 852 NULL NULL 5 108 NULL

     

    Any other ideas?

    Sean

  • ChayDouglas's avatar
    ChayDouglas
    Level 4
    11 years ago

    Hi Sean,

    I had a similar error on my closed lab environment. As it was not live, it allowed me to trial and error a few queries on SQL. 

    Don't know if you found a solution, yourself, but after a series of sql queries I have worked out that the vaultID shown in the log was due to folders I had tried to delete for test users. Not sure why EV was having trouble deleting them, but I have managed to remove the SQL entries for them safely and the error has disappeared.

    First query you could run is the following, that displays the folder name and the archive it is having trouble with:
     

    select r.VaultEntryId, r.RootIdentity, af.FolderName, a.ArchiveName
    from Root r, ArchiveFolder af, archive a
    where r.VaultEntryId = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01'
    AND af.RootIdentity = r.RootIdentity

    Then you can investigate if this folder exists in the archive and/or if it's empty.

    If you can confirm that this folder can be deleted, as was the case with me, you can remove the entries from SQL.

    begin transaction
    delete from ExchangeMbxArchiveFolder where RootID = '857'
    delete from ArchiveFolder where RootIdentity = '857'
    delete from ACE where RootIdentity = '857'
    delete from Root where VaultEntryId = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01'
    commit


    Obviously, this is a SQL hack and I had the benefit of working in a closed environment with backups of the db for roll-back...

    If the Select query provides multiple user archives, it may not be advised to proceed with the hack either.

    I would definitely recommend using the select query to determine which folder EV is complaining about though.

    Good Luck!

    Cheers
    Chay

  • Sean_Waite's avatar
    Sean_Waite
    Level 3
    11 years ago

    Hi Chay,

    The error was due to a user deleting a folder folder their Vault but SQL still had an entry listed for the folder in the directory database. After contacting Symantec Support I was advised to run the following SQL commands on the database:

    USE EnterpriseVaultDirectory
    select * from dbo.ArchiveFolderViewWithDeletedFolders
    WHERE VaultEntryID = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01' < the vault archive ID with the missing folder

    USE EnterpriseVaultDirectory
    UPDATE dbo.ArchiveFolderViewWithDeletedFolders
    SET Deleted = '0'
    WHERE VaultEntryID = '1D6CB4DAD741CD74D919D6839E9A09FEB1110000AMSEVAULT01'

    This fixed the error.

    Thanks,
    Sean