cancel
Showing results for 
Search instead for 
Did you mean: 

FSA Placeholder Deletion

Mohawk_Marvin
Level 6
Partner

I have a problem and I cannot seem to figure it out:-

I have EV9 SP1 and FSA is causing me a headache! My file server is a Windows 2008 server with NTFS drives.

My Policy states:

Delete Archived file when placeholder is deleted

My Retention Category states:

Does not have any holds checked, so it should allow deletions

My File Server target states:

On placeholder deletion, delete archived file.

My Centera retention is set to "Never" as my EV storage, so EV should be allowed to delete instantly.

Pass-thru recall is not enabled on the file server.

 

So now the issue. When I delete a placeholder on the file server the placeholder (obviously) disappears. If I wait for a few hours (24 so far) and open Archive Explorer, my file still shows in the location, and is openabel, restorable etc. If I delete via Archive Explorer, the file is immediately gone leaving an orphaned placeholder.

Any ideas why when I delete the placeholder, it doesnt remove the backend item?

In my dtraces all my DOD flags are showing as set to 1 as well. There are no event errors or warnings around this either.

28 REPLIES 28

Mohawk_Marvin
Level 6
Partner

Additional information:

After running the EV FSA Archiving task the items I had deleted are now showing again in AE. However in the IIS logs I see a request go to deleteo2k for that item no

Item is now gone and I can see this particular item now in JournalDelete table. However no of the other placeholders I deleted are showing in JournalDelete.

Rob_Brenner
Level 5
Employee

Are you doing a hard or soft delete? The PH are not still in the recycle bin are they?

Rob_Brenner
Level 5
Employee

Have you run through the docs? Setting_up_File_System_Archiving.pdf page 40.

The doc referes to the reg key which you didn't mention, so i am guessing you may have missed this.

Set the DeleteOnDelete registry value on the Enterprise Vault server whose

File System Archiving task processes the root volume.

Set the value as follows:

Start the Windows registry editor regedit on the Enterprise Vault server.

Find the following registry key:

HKEY_LOCAL_MACHINE

\Software

\KVS

\Enterprise Vault

\FSA

\ArchivedFilesFlags

You must create the ArchivedFilesFlags key if it does not exist.

 

Mohawk_Marvin
Level 6
Partner

Hi Rob

 

I have tried both hard and soft deletes, and have identical results. And I have DeleteOnDelete set to a value of 1 in the correct path.

Rob_Brenner
Level 5
Employee

I guess, if everything is in place as per the docs, the option here is to try to follow what's hapenning with that request via Dtrace.

Let me try to identify which processes should be enabled  

Rob_Brenner
Level 5
Employee

Looks like the only process you would need to enable with dtrace on the EV server is  EVFileSvrArcMngr.

It might be helpful to also enable the Dtrace of PHS on the file server so you have the initial request being detected with the ID for the item which will help locate on the other dtrace log.

There are some considerations for situations where the file is not deleted from the Archive. Is this a general behavior or you are seeing this for specific folders?

Here is something from documentation which explains one condition when the deletion can fail:

....if any of the parent folders in hierarchy does not have EVFolderPoint.xml then this error would be event logged and file in archive will not be deleted.

Mohawk_Marvin
Level 6
Partner

Hi Rob

 

All dolders are effected on the file server. And as I stated before there are no errors, warnings or even events logged on EV or the EV logs on the file server.

 

Will capture the Dtrace tomorrow and post it.

 

I appreciate the help smiley

Jeff_Shotton
Level 6
Partner Accredited Certified

This might not be a placeholder deletion "problem"

The information shown in archive explorer is xml downloaded by the webserver. By default this information is cached for 24hr before it is junked by the client and downloaded again. However, it CAN be updated by the client in such situations as when you delete an item from archive explorer. The item then appears to go instantly.

However, it hasn't. The archive explorer item view is actually showing data from the index, and the index is updated asynchronously to the storage device. Therefore while EV might have deleted the saveset from disk, it might take a while longer to remove it from the index, and therefore longer than your 24hr that you have already left this.

To confirm, all you need to do is a CTRL+F5 refresh of archive explorer, as this forces the client to download the xml again from the server.

By the way - that journaldelete table you checked has a column called indexcommitted. If the value is set to 1, then the index has been updated. If it is 0, then you are still waiting for that index to update.

Regards,

Jeff

Mohawk_Marvin
Level 6
Partner

Jeff,

I have refreshed AE a few times. If I delete via AE the item disappears from AE view and is added to JournalDelete table with indexcommited is set to 1 for my AE deletes.

My placeholder deletes(from file server) never hit the JournalDelete table.

Mohawk_Marvin
Level 6
Partner

For the purpose of making the dtrace more fun the files "deleted and methods used are as follows:

 

\\ccfilep02\bos_live\documents\protog\BALM\BALM0006227 ==> PH hard delete (shift = del)
File name ==> BALM0006227-SS_office_detail.doc
\\ccfilep02\bos_live\documents\protog\BALM\BALM0006763 ==> soft delete (just del key)
File Name ==> BALM0006763-SS_office_detail
\\ccfilep02\bos_live\documents\protog\BALM\BALM0007721 ==> AE delete
File Name ==> BALM0007721-SS_office_detail
 
I have replaced customer name with my connect ID, so any parts that look to be private, please PM me so I can edit them out blush
 

Rob_Brenner
Level 5
Employee

Looks like the EV server process I suggested you to dtrace is not the correct one as there is no relevant content logged for the DoD request. Sorry about this.

I will need to check again.

The dtrace of the files server does show quite a lot of entries which seem to imply that the processing of DoD may be working at least from the file server side.

I need to review the docs to see if I can find anything that helps understand why the items are not being removed from the Archive.  

I am attaching an edited version of the file server dtrace (connectfileserver_Filtered.txt) with a summary of the calls.

Rob_Brenner
Level 5
Employee

Actually, the files you listed as used in your tests don't seem to have been caught by the PHS dtrace on the file server.

i.e - your hard deleted PH 'BALM0006227-SS_office_detail.doc' is not shown in the dtrace.

Mohawk_Marvin
Level 6
Partner

I saw that, however, I did delete them, and I can recreate the PH with the FSAUtil. Pretty cool right?

Also recalls work fine for these files, perhaps my delete method is the fault?

Rob_Brenner
Level 5
Employee

I think it would be better to try to capture the deletion again. On the PHS dtrace there is a lot of noise from other operations, like PH recall requests.

The Delete request that I see in the dtrace is for file: 

"[EvFileDeleted] Queueing a delete vault item request for deleted file: D:\resources\chpk\chpk0211099\original_chpk0211099-1.jpg"

 It then tries to locate the parent root folder id:

Checking whether folder '\\?\D:\resources\chpk\chpk0211099' is a folder target
Root folder id does not exists on folder 'D:\resources\chpk\chpk0211099'
Checking whether folder '\\?\D:\resources\chpk' is a folder target
Root folder id does not exists on folder 'D:\resources\chpk'
Checking whether folder '\\?\D:\resources' is a folder target
Root folder id does not exists on folder 'D:\resources'
 

But it doesn't seem to actually locate the root folder id. Which of these folders have you defined as a target folder in the VAC?

the documentation actually states:

"If Enterprise Vault is unable to identify the parent target folder for a deleted placeholder, it logs an error in the event log. It does not delete the archived file."

 

 

Mohawk_Marvin
Level 6
Partner

Rob for what its worth I have a case open as well now as I am out of idea's. SYMC case number is:

Case 414-708-324

Mohawk_Marvin
Level 6
Partner

The resources tree is different entirely, but it appears some prior to me missed a "\" as the folder target for that tree, which I have corrected now, thanks Rob! wink

However initial issue remains broken heart

Rob_Brenner
Level 5
Employee

Ok, after getting some internal help, it sems you should also try to include the StorageOnlineOpns process on the EV server dtrace. I would still keep the EVFileSvrArcMngr as it is responsible for DOD policy enforcement and other stuff.

The deletion of the placeholders must be captured by the dtrace of the PHS on the file server, though.

Rob_Brenner
Level 5
Employee

If you have the possibility to add an additional test target Folder with its own test ArchivePoint, it might be a valid exercise to try and test agasint a new data set.

In this case, just for the sake of proving the point, instead of adding a target using '\'  (root), add a subfolder explicitly under the share and ceate the AP at this folder target level.

Rob_Brenner
Level 5
Employee

I tested this and got following notes for you, hopefully it helps

From your Dtrace:

[EvFileDeleted] Queueing a delete vault item request for deleted file: D:\resources\chpk\chpk0211099\original_chpk0211099-7.jpg

=> This line is good as it confirms that DOD is working for the folder otherwise the returned response would be something like:

 [EvFileDeleted] Retain archived flag is set. So it is NOT queued for deletion.
 

The problem starts when we see that the parent root Folder Id cannot be determined:

Processing a delete vault item request for deleted file: \\?\D:\resources\chpk\chpk0211099\original_chpk0211099-1.jpg, placeholder version: 1
Checking whether folder '\\?\D:\resources\chpk\chpk0211099' is a folder target
Root folder id does not exists on folder 'D:\resources\chpk\chpk0211099'
Checking whether folder '\\?\D:\resources\chpk' is a folder target
Root folder id does not exists on folder 'D:\resources\chpk'
Checking whether folder '\\?\D:\resources' is a folder target
Root folder id does not exists on folder 'D:\resources'

=> Here we should see something like:

 Processing a delete vault item request for deleted file: \\?\C:\STRG_FSA\DODTestAP\wild-card-archive-run-2.log, placeholder version: 1
 Checking whether folder '\\?\C:\STRG_FSA\DODTestAP' is a folder target
 rootFolderEID (folderId)= '1C18E0CFFEA6AAF43B879FEAB3142A9931011100EVHA.rb31314uk.gpk'


I think this is likely to be the cause of the issues on your setup.

In order to confirm if the requests are being passed to the EV server StorageOnlineOpns would show the following tracing outout:

CStorageOnline::DeleteItem2 (Entry) |
CStorageOnline::DeleteItem2|Delete Item|SavesetID: 201105135327346~201012151646070000~Z~F0672A371B8A6A29170E0C97F1D0E2E1|Untrusted VaultID: 1798D1E86F6DE1043B3BDCEA5EAAB0A8C1110000EVHA.RB31314UK.GPK

CVaultStoreDB::DeleteSaveset Information: Total 1 SISParts found for saveset having TID:'F0672A37-1B8A-6A29-170E-0C97F1D0E2E1'.
CVaultStoreDB::DeleteSaveset Information: Total 1 SISParts added to output vector for saveset having TID:'F0672A37-1B8A-6A29-170E-0C97F1D0E2E1'.
CVault::DeleteSaveset|SSID: 201105135327346~201012151646070000~Z~F0672A371B8A6A29170E0C97F1D0E2E1|ItemSeqNo: 5|IndexDeletesPending: 0|IndexingDisabled: 0|NotifyIndexer: 0|Ref Count: 0
CVault::DeleteSaveset|Notifying indexing

CNTFSStorageDeviceDeleter::DeleteSisPart (Exit). hr=Success  [0]
SavesetSISManager::DeleteNonCollectedUnreferencedSisParts (Exit) |Success  [0] |
SavesetSISManager::DeleteSisParts|Removing deleted items from the fingerprint catalogue

CSavesetSISManager::DeleteSaveset (Exit). hr=Success  [0]
CVault::DeleteSaveset (Exit) |Success  [0] |
CStoreAccessor::DeleteItem (Exit) |Success  [0] |
CStoreAccessor::DeleteItem (Exit) |Success  [0] |
CStorageOnline::DeleteItem2|DeleteItem Completed - Status: 0x0