Unable to Delete Archives

[AKL]

Hello

We are trying to delete orphaned archives from our environment and have hit a hurdle.

Environment: Enterprise vault 9.0.1, SQL 2005 SP2, Centera - Basic Mode.

Settings: 2 Retention categories - 1 being used for all archives, another modified by me for this specific purpose, both have "Prevent archives from deletion" Unchecked, they are'nt locked. Both inherit expiry from site level. Difference being the one in production has retention of forever and one I modified has retention of 1 day.

Site properties have Process vault stores set to "Never" on storage archives and "Users can delete items" is not enabled/checked.

Issue: We are unable to delete archives, DTRace lines from Storage Delete Component being:

16340 23:13:32.019 [248] (StorageDelete) <8848> EV:M CVault::DeleteSaveset (Exit) |The deletion of the item is not currently permitted. Either the item is on hold, or its retention category does not allow deletions, or a compliance device retention period prevents it from being deleted.      [0xc0041a57] |

16341 23:13:32.019 [248] (StorageDelete) <8848> EV:M CStoreAccessor::DeleteItem (Exit) |The deletion of the item is not currently permitted. Either the item is on hold, or its retention category does not allow deletions, or a compliance device retention period prevents it from being deleted.      [0xc0041a57] |

16342 23:13:32.019 [248] (StorageDelete) <8848> EV:L CDeleteVault::DeleteSavesets Information: Saveset [entryId = 122CDFD541AB40A495A7AD08D77FD630] can't be deleted as it is on hold.

Troubleshooting done: Since these archives had retention which is in production (named Business for example) and they do not have mailbox/AD account associated with it - so I was unable to figure a way to assign these accounts to a separate provisioning group.

1. Changed the saveset retention categories using SQL Query:

use EVVaultStore01

update View_saveset_archive_vault

set retentioncategoryidentity = '5'

from View_saveset_archive_vault ss

Inner Join enterprisevaultdirectory.dbo.archiveview av on av.vaultentryid = ss.archivepointid 

where ss.RetentionCategoryIdentity = '2' AND av.ArchiveName = Donald Duck'

But the DTrace still points them to the old retention category (Business):

16418 23:13:32.534 [248] (StorageDelete) <8848> EV:L CVault::IsDeletePermitted|Retention Category: Business|ONHOLD: No

16419 23:13:32.534 [248] (StorageDelete) <8848> EV:L CVault::ReadPartitionEntry (Entry) |

16420 23:13:32.534 [248] (StorageDelete) <8848> EV:M CVault::ReadPartitionEntry|Id: 0

2. Changed Retention properties on vault store partition from All Centera mode to Never.

3. We do not use Discovery / Compliance accelarator - so they are out of picture.

4. No saveset is present in dbo.Hold table on SQL - so not sure how Enterprise vault is identifying it as a hold item.

Not sure if I am too eager to see it working and should wait for retention category change to prevail OR should I see immediate change in DTRace after running SQL query?

On side note - I attempted to delete a dozen of large archives before changing category so not sure if that compounds.

I am believing as soon as Saveset identifies the new retention category - it should allow to delete them right?

Attached is DTRace log for more review if needed.

References made so far:

http://www.symantec.com/business/support/index?page=content&id=TECH126131

http://www.symantec.com/business/support/index?page=content&id=TECH68712

http://www.symantec.com/business/support/index?page=content&id=TECH76735

Thanks a lot in advance for any assistance

[JesusWept3]

Good troubleshooting, are you 100% it's a basic mode centera??

[AKL]

Hi, I would say I am 60% sure as I mentioned that based on old configuration details I had.

I will re-check and confirm rest 40%.

[JesusWept3 · Accepted solution]

Search your DTrace for the words compliance and governance, if you have textpad just search and bookmark all results and paste to a new document, it should give a true or false statement mentioning the mode used for the centera Also if you change the retcats in the DB, that will allow for expiry and deletion from a logical standpoint, but it will not retroactively update the CDF's on the centera itself, and the same when you set to Never that will only affect new items being stored What you may want to do is use EVSVR in interactive mode to retrieve the CDF for a particular saveset and see if there is an attached TTL on which the item will be eligible for expiry And lastly it's definitely not a retention issue as it mentions that the item onhold = no; and the site level users can delete would not stop you as an admin from deleting the archive

[Paul_Grimshaw]

Your Centera is not in basic mode so that is your issue:-

16148 23:13:26.206  [248] (StorageDelete) <8848> EV:M CVaultStoreEMCCentera::SetPool -- Compliance Device mode: True

I do not know what sort of access you have to your centera but if you have Centera Viewer and you can connect you want to run the command show security all and that will tell you what mode your centera nodes are in.

From your DTRACE you can see that there is an expiry on the CLIP:-

ClipGetExpiryDate -- Clip-Id: ECLDOU0DLI1PEe0M9M7BLFO8QVEG4126IM62SL0DFAH6UV25LSOK6, Expiry date: 2958190.018449

[AKL]

Thanks a lot guys for your review and you are right that DTrace does shows compliance.

However the replica IP address it shows (snipped in the DTRace uploaded) - are pointing to old centera nodes - we have configured EV to point to new centera which is in Basic mode.

So - just wondering why storage service is pointing to old centera for deletions and if anything could be done to bypass/workaround this..

[AKL]

Please ignore my previous response. I see why Enterprise vault is pointing to old centera..

Basically we have three partition for the vault store - two of which are closed. One of them is being reported in these DTRace logs.

So the next question will be why Enterprise vault is trying to delete archives from a closed partition?

[AKL]

Also, if someone knows or have a sample of successful deletion DTRace log lines which I can look for in my side of DTRace to see any items is getting deleted from original Partition/Centera

[Paul_Grimshaw]

Ajay,

So the next question will be why Enterprise vault is trying to delete archives from a closed partition? 

So if you are deleting archives and the items were archived in what you call your old partition then why would it not try to go to that Centera as that is where those archived items are right?

That partition in the partitionentry table will have an ip address List and replica ip address list which will be pointing to your old Centera. Are you saying that all the data that was on your old centera is on your new centera and you want the deletes to be performed on there? Not quite following what you are saying here.

Also, if someone knows or have a sample of successful deletion DTRace log lines which I can look for in my side of DTRace to see any items is getting deleted from original Partition/Centera.

Below are some important snippets from a deletion of an archive on my test system whose 5 items were stored on a Centera:-

CDeleteVault::DeleteSavesets Information: Total [5] savesets were found.

{CVault::DeleteItemFromCenteraStorage} (Entry)

{CVault::DeleteItemFromCenteraStorage}|Store Identifier: 0VTUI6RF62170e5VC20ILJV1OA8G416EFV2PG90VF7RUH3TUB4EJ1

CVaultStoreEMCCentera::ClipDeleteCentera -- Calling FPClip_Open|Clip-Id:      0VTUI6RF62170e5VC20ILJV1OA8G416EFV2PG90VF7RUH3TUB4EJ1|VaultStoreId: 107225EC78D98914FA558B36CBC20137F1210000evalias|VaultId:      189741B0CD7D15B45A92E295246C30D5D1110000evalias|SavesetId:    201111141237896~201110260609020000~Z~10F5DDD0E16BB480A7608AB4D17ACD21

{CVault::DeleteItemFromCenteraStorage} (Exit) Status: [Success]

{CDeleteVault::DeleteArchive}|Archive emptied

{CDeleteVault::DeleteArchive}|Index deleted

{CDeleteVault::DeleteArchive}|Storage database records removed

{CDeleteVault::DeleteArchive}|Archive deleted from Directory

[AKL]

Thanks Paul - You got my point correct.

Basically all centera storage data was copied from old device to new device. The old device still exist though but in compliance mode. And new device being in Basic mode.

I am thinking if I point all partitions in vault store group to the new centera device - it should look for savesets/clip IDs in the new centera and delete from there accordingly?

Thanks for the sample success lines.

[Paul_Grimshaw]

Yeh it is not something I have ever tried myself and cannot repro to test but it sounds logical enough that if you but the IP details of your new centera into the sql columns for your closed partition then it should go to your new centera.