cancel
Showing results for 
Search instead for 
Did you mean: 

Image Cleanup jobs failing with RC88

Tomas_Pospichal
Level 4

Hello,

I would like to ask you regarding following issue.
Almost all of the Image Cleanup jobs ends in error "(88) Open Storage WORM lock error."

In the Details of each job, there are always same lines with following message (for various backup images):
Info bpdm (pid=15004) Failed to delete WORM locked image XXXXXX_1675098989_C2_F1_R1: error 2060069.

I was trying to find any record for backup image "XXXXXX_1675098989" using "bpimagelist" however NetBackup does not know about it and returns "no entity was found".

Do you have any idea how to get rid of it?

 

Thanks.
Tom

1 ACCEPTED SOLUTION

Accepted Solutions

Nicolai
Moderator
Moderator
Partner    VIP   

Try nbdelete with the -force option. Likely there is entries in the EMM table where NBU track images it can't delete 

DOC link : https://www.veritas.com/support/en_US/doc/15263389-127350397-0/v14669226-127350397

If there is fragment left on the storage device and you use the force option, the fragment will remain on the storage device forever, as NBU no longer have any idea that they exist. So use option carefully.

View solution in original post

6 REPLIES 6

Nicolai
Moderator
Moderator
Partner    VIP   

Try nbdelete with the -force option. Likely there is entries in the EMM table where NBU track images it can't delete 

DOC link : https://www.veritas.com/support/en_US/doc/15263389-127350397-0/v14669226-127350397

If there is fragment left on the storage device and you use the force option, the fragment will remain on the storage device forever, as NBU no longer have any idea that they exist. So use option carefully.

I can confirm that nbdelete worked, the record is gone. That is good.

However how can I verify information about the fragment itself? I would like to know when it is supposed to expire or amount of space that it occupies on the storage? Is this possible?

Thank you, Nicolai.

You could run the nbdelete command with the "-list" option :

~snip~
-list
Lists imported and unimported snapshots or NetBackup (tar) images eligible for deletion.
~snip~

Kind regards
Christopher

Hi @Tomas_Pospichal 

The answer partly depends on the type of storage the fragment is/was on and whether you can examine the catalog on the storage server. For MSDP, the command to use is catdbutil (resides under the pdde folder) and requires the client name and policy name to enable to see what pieces the pool still contains. The command would be something like this: "catdbutil --list --dbpath /<path to msdp>/databases/catalog/2/<client>/<policy>". Then search the output for the backupid string.

For other storage servers you'd need to ask the vendor.

The original problem though should be further investigated, as it implies to me that your retention in NetBackup is not matching the WORM retention settings (i.e. you are expiring the image in NetBackup before the image can be deleted from the WORM storage). The NetBackup retention should be longer than the minimum retention time of the WORM storage. In theory, the error would have gone away by itself once that minimum retention time expired, and NetBackup was able to tell the WORM storage to delete the piece.

Cheers
David

Tomas_Pospichal
Level 4

Hi all,

we do use the Dell DataDomain as a target storage (ddboost).
Is there any way how to verify there that the fragments are still on the storage? I would like to somehow check it, however not sure how, I have only fragments which appears to be just some records in the database.

 

Thank you all.
Tomas

Hi @Tomas_Pospichal 

Firstly, I'm no DD expert, and this is based on limited research and what I expect to see.

I believe the command is something like "ddboost storage-unit show  <mtree name>". Using the mtree name where you backup images go. 

This should provide the listing of backup files. If it appears as I think, then the listing will contain files which start with the backupid of the backup image (appened with copy number and frag number bits - but that's not critical). 

With backup ID you can tell if the image is still in the NetBackup catalog. What to do if you find extra files/images on the DD I do not know.

If all else fails - talk to the vendor - Dell - and ask them.

David