Access denied when trying to delete archives

[yarg]

Hi,

I've got an old EV8 environment from which I am unable to delete archives.  I get 6643 events logged, saying:
Delete Vault failed
Reason: Access is denied.  [0x80070005] 

I don't see why this should happen though, since I'm logged in as the vault service account, that account is a member of local admins and local admins has full control over the disk where the vault store group is.  I've run dtrace to try and determine exactly which file it was trying to access in generating the error, but unfortunately it doesn't seem to give that level of detail.  Here's an extract of the log:

473 13:16:45.178 [7472] (StorageDelete) <6792> EV:L CStorageSession::~CStorageSession (Entry) |

474 13:16:45.178 [7472] (StorageDelete) <6792> EV:M CStorageSession::~CStorageSession (Exit) |

475 13:16:46.756 [7472] (StorageDelete) <4904> EV:H VaultCoCreateInstanceEx: An error occurred - RequestedServerName = [server.domain.org], UsedServerName = [servername], hrCCI = [0x80070005], hrResultsQI = [0x80004005], NumTried = [6], bLocalMachine = [True]

476 13:16:46.756 [7472] (StorageDelete) <4904> EV~E Event ID: 6643 Delete Vault failed |Reason: Access is denied.  [0x80070005] |Vault Name: Username ABC [2007-05-05 - 2011-07-05] |Vault Store: Exchange Vault Store |Vault Id: 1FEE6BF6FA38DBE41B3DD4A1643DCEF231110000EVSite |

477 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CDeleteVault::DeleteArchive (Exit) |Delete Vault failed   Reason: %1   Vault Name: %2   Vault Store: %4   Vault Id: %3      [0xc00419f3] |

478 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CDeleteVault::ProcessQueueItems Error: Failed to delete a queue item havingItemType : [0] Item Name : [Username ABC [2007-05-05 - 2011-07-05]] Item EntryId: [1FEE6BF6FA38DBE41B3DD4A1643DCEF231110000EVSite] Error Code: [0xC00419F3]

479 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CDeleteVault::ProcessQueueItems Information: Processing deletion queue item having ItemType : [0] Item Name : [Username XYZ [2007-05-12 - 2011-07-08]] Item EntryId: [1FAE88B7CBAFB8A4CAAD2CB2B5870A5431110000EVSite]

480 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CDeleteVault::DeleteArchive (Entry) |

481 13:16:46.756 [7472] (StorageDelete) <4904> EV:H CDeleteVault::DeleteArchive|Deleting Archive | Archive Name: Username XYZ [2007-05-12 - 2011-07-08]| Vault Store Name: Exchange Vault Store

485 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CDeleteVault::InitVSBDConnection (Entry) VaultStoreEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite]

486 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CDeleteVault::InitVSBDConnection Information:No new VSDB connection needed as current VSEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite] and input VSEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite]

487 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CDeleteVault::InitVSBDConnection (Exit). hr=Success  [0]

488 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CDeleteVault::EmptyArchive (Entry) ArchiveEntryId = [1FAE88B7CBAFB8A4CAAD2CB2B5870A5431110000EVSite] VaultStoreEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite]

489 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CVaultStoreDB::GetSavesetCountByArchive Information: ArchiveEntryId = [1FAE88B7CBAFB8A4CAAD2CB2B5870A5431110000EVSite]

490 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CVaultStoreDB::GetSavesetCountByArchive Information: SavesetCount = [0]  LowRange = [0] HiRange = [0] ArchivePointIdentity = [147]

491 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CDeleteVault::EmptyArchive Information: Archive having name = [1FAE88B7CBAFB8A4CAAD2CB2B5870A5431110000EVSite] and entryId = [Username XYZ [2007-05-12 - 2011-07-08]] is already empty.

492 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CDeleteVault::EmptyArchive (Exit). hr=Success  [0]

493 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CDeleteVault::DeleteArchive|Archive emptied

495 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]

496 13:16:46.756 [7472] (StorageDelete) <4904> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine

497 13:16:46.756 [7472] (StorageDelete) <4904> EV:M CCIndexBrokerPtr::FindIndexingMachine ArchiveEID: 1FAE88B7CBAFB8A4CAAD2CB2B5870A5431110000EVSite ==> Computer Name: server.domain.org (hr=Success  [0])

498 13:16:46.756 [7472] (StorageDelete) <4904> EV:M VaultCoCreateInstanceEx: IsLocalMachineExtendedCheck returned true for server [server.domain.org]  as the IP Address matched that of the local machine

499 13:16:49.788 [7472] (StorageDelete) <4904> EV:H VaultCoCreateInstanceEx: An error occurred - RequestedServerName = [server.domain.org], UsedServerName = [servername], hrCCI = [0x80070005], hrResultsQI = [0x80004005], NumTried = [6], bLocalMachine = [True]

500 13:16:49.788 [7472] (StorageDelete) <4904> EV~E Event ID: 6643 Delete Vault failed |Reason: Access is denied.  [0x80070005] |Vault Name: Username XYZ [2007-05-12 - 2011-07-08] |Vault Store: Exchange Vault Store |Vault Id: 1FAE88B7CBAFB8A4CAAD2CB2B5870A5431110000EVSite |

501 13:16:49.788 [7472] (StorageDelete) <4904> EV:M CDeleteVault::DeleteArchive (Exit) |Delete Vault failed   Reason: %1   Vault Name: %2   Vault Store: %4   Vault Id: %3      [0xc00419f3] |

502 13:16:49.788 [7472] (StorageDelete) <4904> EV:L CDeleteVault::ProcessQueueItems Error: Failed to delete a queue item havingItemType : [0] Item Name : [Username XYZ [2007-05-12 - 2011-07-08]] Item EntryId: [1FAE88B7CBAFB8A4CAAD2CB2B5870A5431110000EVSite] Error Code: [0xC00419F3]

503 13:16:49.788 [7472] (StorageDelete) <4904> EV:L CDeleteVault::ProcessQueueItems Information: Processing deletion queue item having ItemType : [0] Item Name : [Username LMN [2007-05-12 - 2011-07-13]] Item EntryId: [1F42C93BDE4778443A8D9A571FA60AECA1110000EVSite]

504 13:16:49.788 [7472] (StorageDelete) <4904> EV:L CDeleteVault::DeleteArchive (Entry) |

505 13:16:49.788 [7472] (StorageDelete) <4904> EV:H CDeleteVault::DeleteArchive|Deleting Archive | Archive Name: Username LMN [2007-05-12 - 2011-07-13]| Vault Store Name: Exchange Vault Store

509 13:16:49.788 [7472] (StorageDelete) <4904> EV:M CDeleteVault::InitVSBDConnection (Entry) VaultStoreEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite]

510 13:16:49.788 [7472] (StorageDelete) <4904> EV:L CDeleteVault::InitVSBDConnection Information:No new VSDB connection needed as current VSEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite] and input VSEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite]

511 13:16:49.788 [7472] (StorageDelete) <4904> EV:M CDeleteVault::InitVSBDConnection (Exit). hr=Success  [0]

512 13:16:49.788 [7472] (StorageDelete) <4904> EV:M CDeleteVault::EmptyArchive (Entry) ArchiveEntryId = [1F42C93BDE4778443A8D9A571FA60AECA1110000EVSite] VaultStoreEntryId = [16EE40FE4EDBCA64B8A90AE27CB64367C1210000EVSite]

513 13:16:49.788 [7472] (StorageDelete) <4904> EV:L CVaultStoreDB::GetSavesetCountByArchive Information: ArchiveEntryId = [1F42C93BDE4778443A8D9A571FA60AECA1110000EVSite]

514 13:16:49.788 [7472] (StorageDelete) <4904> EV:L CVaultStoreDB::GetSavesetCountByArchive Information: SavesetCount = [0]  LowRange = [0] HiRange = [0] ArchivePointIdentity = [153]

515 13:16:49.788 [7472] (StorageDelete) <4904> EV:L CDeleteVault::EmptyArchive Information: Archive having name = [1F42C93BDE4778443A8D9A571FA60AECA1110000EVSite] and entryId = [Username LMN [2007-05-12 - 2011-07-13]] is already empty.

516 13:16:49.788 [7472] (StorageDelete) <4904> EV:M CDeleteVault::EmptyArchive (Exit). hr=Success  [0]

Any suggestions please on how to get these archives deleted?

Thanks!
G

[TonySterling]

Do you have DA?  

You are seeing the same error code as this post. https://www-secure.symantec.com/connect/forums/delete-vault-failed

[yarg]

Hi,

Yes we do have DA, but unfortunately I am not able to access it to confirm whether that is the root cause.  This EV environment has not been used for about 3yrs now, so machines which had the client installed (8.0 SP5) have since been decommissioned and we no longer have the source.  Symantec support have told me that since that version is no longer supported, then we may not download the client.  So as far as I can tell, although DA is running on the server, there is no way for me to access it to see what the situation is.

I saw that link you posted and since I could not launch DA itself, I looked at the VaultInterest table and all I see is this:

VaultEntryId ConsumerGUID ConsumerName RegistrationDateTime Type

1AC95F8FDC0275B4CA491076FC9CA931E1110000EVSite 63CFFCE5-1E2C-4107-B483-3B62047C3757 Discovery Accelerator: EVAdmins 2010-05-31 14:14:46.357 17

That's the only entry in the table, and since we I have 30 or so archives which will not delete, I was not sure if this single entry in the table is related to those archives.  Is it safe to remove that entry from the table and then see if the deletion works perhaps?

Thanks,
G

[TonySterling]

If you only have that one entry then I doubt that is causing the issue.  You will want to have a look at what is in DA to make sure all your legal holds have been released.  I have attached the DA 8.0.5 client for you.

[yarg]

 Thanks very much for the client.  I've tried installing it but get the following error now:

"Source file not found: C:\Temp\AcceleratorClient.exe.config.  Verify that the file exists and that you can access it."

I thought the client was a standalone installer and didn't need anything else?

Cheers!
G

[TonySterling]

It is a stand-alone client but it does need that config file.  Sorry, I must have negleted to include it.

Here is a new zip with the file included.

[yarg]

That works, thanks!

So, now when I go into the client, I see that under Cases -> All cases, I see only two cases, both of which have a "Legal Hold" state of "Off".  Both cases are configured to only search the journal and not the exchange vault store.  

Is there something in particular that I should be searching for?

Thanks!
G

[TonySterling]

Nope, that pretty much rules Legal Holds out as an issue.

[TonySterling]

Your storage isn't by chance a NetApp:

Event ID 6643 when attempting to delete an Archive which is stored on a NetApp filer device

Article:TECH68797

|

Created: 2009-01-16

|

Updated: 2012-02-27

|

Article URL http://www.symantec.com/docs/TECH68797

If not a NetApp you should check the VSA permissions on the storage location.

[yarg]

Having never worked with a NetApp, I'm afraid I don't even know what one is!  The EV server is linked via fiber HBAs to an EMC SAN where the archives are stored on their LUNs.  Does that help?

[TonySterling]

NetApp is a storage platform.  Can you verify the VSA has full control over the vault store partitions?

[JesusWept3]

if i was to guess at anything, it would be that it couldn't create a storage server request to StorageDelete on the servername that was listed there, maybe because of a DCOM option

If you go to that machine and look in the event viewer under "System", do you see any Distributed COM errors that describe a failure to launch etc?

Maybe worth doing a reset of the password in the VAC, this will update all the DCOM packages on each EV Server in the environment and alert you to any servers that couldnt be updated or contacted etc

[yarg]

The VSA is a member of local admins and local admins has full access rights to the drive where the vault partitions are, so there are full access rights inherited to each of the vault store partitions.

In the system log there are some DCOM errors, but they are relating to permissions for 2 users in particular...

Thanks,
G

[yarg · Accepted solution]

So while I was digging around in the System logs, I noticed "Plug and play" events which indicated disk issues.  I ran chkdsk /F, rebooted and let it do it's thing (it did find a number of errors and fixed them).  When I logged in and launched the Vault console, it had correctly deleted these 30 or so archives, so it seems to have been related to these disk issues.

Thanks for trying to help!
G