Archives stuck in "marked by deletion"

[Tom_L_]

We're using  9.0.3.

3 months ago we started getting repeated errors every 15 minutes.  The errors are 13360,  13360, 8453, 8390, & 6651.  I'll post the details below.  The errors didn't seem to effect anything but now we can't delete archives & there are no new errors showing just the ones listed.   Any ideas?

Event Type: Error
Event Source: Enterprise Vault
Event Category: Directory Service
Event ID: 13360
Date:  9/13/2012
Time:  2:57:59 PM
User:  N/A
Computer: MAILEV01
Description:
An error was detected while accessing the Vault Database 'EnterpriseVaultDirectory' (Internal reference: {CADODataAccess::ExecuteSQLCommand} [.\ADODataAccess.cpp, lines {1407,1409,1424,1442}, built Nov 11 21:02:49 2011]):

Description: 

Cannot insert the value NULL into column 'Result', table 'tempdb.dbo.#DeletedFoldersXML__________________________________________________________________________________________________0000000002B4'; column does not allow nulls. INSERT fails.
 
 
SQL Command:
 DeleteArchiveFolders
 

Event Type: Error
Event Source: Enterprise Vault
Event Category: Directory Service
Event ID: 13360
Date:  9/13/2012
Time:  2:57:59 PM
User:  N/A
Computer: MAILEV01
Description:
An error was detected while accessing the Vault Database 'EnterpriseVaultDirectory' (Internal reference: {CADODataAccess::ExecuteSQLCommand} [.\ADODataAccess.cpp, lines {1407,1409,1424,1442}, built Nov 11 21:02:49 2011]):

Description: 

The statement has been terminated.
 
 
SQL Command:
 DeleteArchiveFolders
 

Event Type: Error
Event Source: Enterprise Vault
Event Category: Directory Service
Event ID: 8453
Date:  9/13/2012
Time:  2:57:59 PM
User:  N/A
Computer: MAILEV01
Description:
The EnterpriseVault.DirectoryService object reported an error.
 
The error code was 0x80040e2f 

Event Type: Error
Event Source: Enterprise Vault
Event Category: Storage Delete
Event ID: 8390
Date:  9/13/2012
Time:  2:57:59 PM
User:  N/A
Computer: MAILEV01
Description:
The EnterpriseVault.DirectoryConnection object reported an error.
 
0x80040e2f
Internal references:
Error [0x80040e2f]
{CDirectoryConnectionObject::DeleteArchiveFolders} [.\DirectoryConnectionObject.cpp, lines {11308,11318,11320}, built Nov 11 21:02:52 2011]

Event Type: Error
Event Source: Enterprise Vault
Event Category: Storage Delete
Event ID: 6651
Date:  9/13/2012
Time:  2:57:59 PM
User:  N/A
Computer: MAILEV01
Description:
Storage Deletion Server has failed
Reason:  <0x80040e2f>

 

[JesusWept3]

What version of EV? To be honest I think your best be would be to call support, but if you want to DTrace StorageDelete and restart the storage service, would be happy to have a look

[Jeff_Shotton · Accepted solution]

When archives are deleted they are queued up by storagedelete and then processed one at a time. It sounds like you have one particular archive which contained a less-than-perfect structure (maybe this is why you were trying to delete it) and then bad data in the archivefolder table has caused the process to fail out.

As long as that archive is the first one to be processed by the storage service (the order is based on archivepointidentity) all your queued deletes will never happen - which is why you can't delete any other archives.

What might help here, if you can recall the name of the archive which caused all this to go wrong, is that you export the archivefolderview for that archive and post it here. Of course, if you post the dtrace we will be able to give you the GUID of the archive to then run the query against.

However, it is very very likely you are going to be running SQL scripts to modify the inconsistencies that pop out, and that means calling support. There were a bunch of problems with archive structures being mangled in EV8sp2>EV8sp4, and while those issues were fixed, the legacy data remaining was not. A tool was made available called archivefolderfix - support have it, and it may help.

A flavour of these issues can be found here:

http://www.symantec.com/docs/TECH52763

[Tom_L_]

We cleared the list of pending deletions in sql a few days ago & i marked just one for deletion. It hadn't deleted when i posted yesterday but sometime during the night it did.  I don't think it should take 2 days to delete but then it could be worse.  I've marked a few more to delete and they deleted relatively fast.  We have 700 or so archives that need to be deleted so it will be hard to tell which ones have problems etc As far as the 5 repeating errors go, they come every 15 minutes whether or not we have anything qued up for deletion.  As long as we can delete archives, I guess they won't be too big of a problem. 

[Tom_L_]

Dtrace attached:

[Jeff_Shotton]

Thanks Tom

relevant bits:

14:13:42.709  [17420] (StorageDelete) <18160> EV~E Event ID: 6761 Error opening EMC Centera C-Clip |Function call: NetworkPacket.checkControl(2)<HPPReadBlobTransaction.readBytes(HPPImportData&)<HPPReadBlobTransaction.run()<ClusterCloud::readBlob(FPBasicGenericStream*,3E6E20HAR1V9Ee1KMJGU5UH9G0LG414HLK1MC5013VL4USEABFE3P,,0,-1,-1,0)<FPClip::Open(pool, 3E6E20HAR1V9Ee1KMJGU5UH9G0LG414HLK1MC5013VL4USEABFE3P, 1)<FPClip_Open(-,3E6E20HAR1V9Ee1KMJGU5UH9G0LG414HLK1MC5013VL4USEABFE3P,1) |Status: FP_CLIP_NOT_FOUND_ERR (-10021) |Reason: Clip CDF is not found in the pool (transid='mailev01/16/READ_CLIP') |Pool Address: 10.4.10.223,10.4.10.222?\\Maildc01\fileshare\EMC\Centera\M_LADOA_EVAULT.PEA |?4837 
 

14:13:42.600  [17420] (StorageDelete) <18160> EV:M {CVault::GetItemProperties}|Sel:A34A868F4DD8487B9DA082C6BAA93A60 Seq No:41792 Partition ID:2 Store Id:3E6E20HAR1V9Ee1KMJGU5UH9G0LG414HLK1MC5013VL4USEABFE3P CollectionId:False?4787 

check the vault store database for the saveset record associated with this storeid - it may be causing an issue

Also:

14:13:50.537  [17420] (StorageDelete) <7160> EV~E Event ID: 6651 Storage Deletion Server has failed |Reason:  <0x80040e2f> |?7297 

This dtrace error matches the following technote:

http://www.symantec.com/docs/TECH186647

though this might be the side affect of the above missing CLIP, so i'd have a look for that first.

Regards,

Jeff

[Prone2Typos]

Hey Tom.

I jsut wanted to follow-up and see if you still had outstanding concerns related to this. From review it appears that you may have a solution or need to call support for further support in clearing up SQL integrety issues.

One thing I think I will throw out there is that not only were there issues in early versions of EV 8 that contributed to DB inconsistancy, but also ensure your backups are taken from the same point in time of all DBs. Restoring DB backups from differnt points in time has been known to lead to these sorts of inconsistancies if there were additions or deletions to the folders or archives in the environement.

If you have no further questions or concerns related to this posting, can you please pick the appropriate posting and flag it as teh solution (or write one showing how you fixed it and flag that)? If not please let us know how we can help more.

Regards.

J

[Tom_L_]

Our dba found that saveset in a _RESTORE db.   What should we do to it?

[Jeff_Shotton]

Tom,

Im assuming you are talking about a restored vault store database? Well it might be there, but i was more bothered about it being in the live one and somehow stopping the deletion process :)

So what should you do with it? In a restored database - nothing.

Have you looked at the technote http://www.symantec.com/docs/TECH186647 yet and run the SQL queries to see if there are folders 'stuck'?

Regards,

Jeff

[Tom_L_]

Honestly, I didn't even know we had a restored vault database.  That's where they said it was & there were 0 results from the queries in the link.  We tried that before i came to this forum.    Archives are deleting though, just very slowly (around 9 a day).   We are still getting a steady spam of the errors above, but if they're not really hurting anything, we can live with it.   Thanks for the help though :)