UNABLE TO DELETE PARTICULAR ARCHIVED ITEMS FROM A SINGLE FOLDER

[Usman_Saeed]

Unable to Delete Individual Items from a user Archive. Received popup of “Bad Request” error

• checked the event viewer for any error messages or warnings and filter the current log and searched different event ID’s which are 6643, 8453,13360 and 28997
• confirmed that migrated from the old server so some Events are showing old server stats
• went to administration console of EV and checked the retention categories and specifically business category
• different TECH articles in front of us, we choose TECH 49804 based on its links to the specific issue and read the desired queries
• opened user archives and selected one archive which was “user” and checked its permissions and settings
• confirmed from the customer when he hit the first delete process and he confirmed us that it was done on yesterday
• opened tasks and select “Mailbox archiving task” and specifically apply synchronize on that one archive which is “user”
• open SQL server to apply the queries which we have in the article “TECH 49804”  and opened “SQL server management studio 2014” and open databases.
• on EV and checked “vault store name” of “user” and its “Archive ID/Vault entry ID” because we need this details to embed in that specific query of article “TECH 49804”.
• Vault store name was = “vault_store” and also copied that “archive ID/Vault entry ID” in notepad to later copy in that query.
• ran that query from database “vault_store” after editing it by adding “vault store name” and “vault entry ID” and executed the query.
• The result of first query was

                Not on legal hold

               79803 items were detected

               Retention category value was 2

• confirmed from customer that he only wants to delete those specific 18 items.
• copied the second query and edit the retention category value to 2 and executed the query.
• The result was

                That its business retention

                On hold = 0

               Value is 2

• went to EV and opened policies ->Business policy->Check all settings and confirmed that users are able to delete the items settings was enabled.
• went to server manager ->Roles and features->Computer management->MSMQ’s there was no item in queue and all queues were empty.
• verified that the items are not on legal hold by applying all the required steps in TECH articles recommended by veritas and nothing suspicious activity was observed.
• rebooted the EV server.

Performed the DTRACE

1. Further investigated issue using Enterprise Vault Dtrace diagnostic utility
2. Results show Below are the 18 items that cannot be deleted 

 72446 16:03:24.765 [14976] (StorageOnlineOpns) <18752> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
72720 16:03:25.687 [1944] (StorageOnlineOpns) <18528> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
72998 16:03:26.718 [7244] (StorageOnlineOpns) <4616> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
73270 16:03:27.406 [14160] (StorageOnlineOpns) <20356> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
73542 16:03:28.328 [5048] (StorageOnlineOpns) <15000> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
73797 16:03:29.062 [14976] (StorageOnlineOpns) <12668> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
74052 16:03:29.765 [1944] (StorageOnlineOpns) <15872> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
74307 16:03:30.437 [7244] (StorageOnlineOpns) <15088> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
74562 16:03:31.437 [14160] (StorageOnlineOpns) <18788> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
74817 16:03:32.093 [5048] (StorageOnlineOpns) <7384> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
75072 16:03:32.703 [14976] (StorageOnlineOpns) <3304> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
75327 16:03:33.312 [1944] (StorageOnlineOpns) <15456> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
75582 16:03:33.953 [7244] (StorageOnlineOpns) <17424> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
75837 16:03:34.797 [14160] (StorageOnlineOpns) <12204> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
76092 16:03:35.422 [5048] (StorageOnlineOpns) <4420> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
76352 16:03:36.109 [14976] (StorageOnlineOpns) <16532> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
76612 16:03:36.750 [1944] (StorageOnlineOpns) <4284> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57
76889 16:03:38.062 [7244] (StorageOnlineOpns) <4616> EV:H {CStorageOnline::CanItemBeDeleted}|CanItemBeDeleted result: CanBeDeleted: False, Reason: 0xc0041a57

• error indicates that items are on hold according to technote https://www.veritas.com/support/en_US/article.TECH61293

• After dtrace log analysis asked customr whether they have some EMC Centera or compliance devices in the environment. Customer confirmed from storage engineer and yes they have EMC as the target storage for EV partitions.
• Customer was briefed on how these compliance devices work, these devices don’t allow the user to delete the item until the retention expiry time period finishes on the individual items.
• We analyzed the individual save sets. If any of them were on legal hold in DA. The query used is enlisted in the technote https://www.veritas.com/support/en_US/article.000039278
• Executed query on SQL server on target vault store database. The result was that those items were not on hold. 
• Sent a new email in that folder  to archive and test delete which was a success.
• Next was to delete one archived email item from the 18 items and system prompted with the option to delete both item plus shortcut or just the shorcut but for both options result was “Access Denied” which along with the logs indicated issue is the same as https://www.veritas.com/support/en_US/article.TECH61293.

out of ideas and options. please remember main objective is to delete the items and the folder completly from archive also tried to export and remove the item from folder but unsuccessful.

[WiTSend]

Based on what information you've provided the only way to "remove" those items would be to export the archive to a PST, open it, remove them items, create a new archive and import into the new archive.  Once this is verified you can delete the old archive.  Those offending items will still be on the storage platform marked for deletion, but wont' be visible to the user.