"Problems" with Image Cleanup since upgrade to 6.5.4

[Andy_Welburn]

Am having the follwoing issues with the automatic Image Cleanup process & wonder if anyone could shed some light on them:

1) Reported in the "Problems Report"

21/07/2009 16:37:13 0 master Error 97 bpexpdate Could not deassign media-id 300320, host ndmpmedia: requested media id is in use, cannot process request (97)
21/07/2009 16:37:13 0 master Error 97 bpexpdate Could not deassign media-id 300424, host ndmpmedia: requested media id is in use, cannot process request (97)
21/07/2009 16:37:14 0 master Error 97 bpexpdate Could not deassign media-id 300425, host ndmpmedia: requested media id is in use, cannot process request (97)

I've put ndmpmedia in the above output for want of a better term - this is the device host for the ndmp tape drives. I can confirm that all three tapes are currently loaded in tape drives and are currently being written to as 'normal' NDMP saves. Why is the process trying to deassign them?

2) When the Image Cleanup process has finished
i.e. 21/07/2009 16:55:51 - Info bpdbm(pid=13832) deleted 638 expired records, compressed 509, tir removed 0, deleted 0 expired copies
why am I then getting tagged on after this:
host is unreachable(47) ?

I initially thought that this was referencing an incorrectly decommissioned media server, see my post from earlier today: Decommissioning 'expired' Media Server in SSO environment.
BUT it also happened yesterday after the upgrade but before I decommissioned the media server (however, services have not been running on the latter for quite sometime now).

[Stumpr2]

Try running the NBCC utility again and correct any discrepancies.

[Andy_Welburn]

Never run one before on a minor/patch upgrade, only when we went from 5.1MP6 to 6.5GA.

Will run one when it's all quietened down on the backup front & let you know.

Cheers Bob!

[Prashant_K · Accepted solution]

Hi Andy,

Please try to run the following command nbdelete -allvolumes -force followed by bpimage -cleanup -allclients from admincmd folder and then run the image cleanup and see if that clears the errors and incase if the issue still occurs then it requires the netbackup support to create the SQL scripts to remove the stale entries in the database.

Following would be the logs that would be required by support

Logs and other evidence to collect:
1. ...\bin\nbdb_unload <path>
2. ...\bin\nbdb_backup -online <path>
3. ...\admincmd\bpimmedia.exe > bpimmedia.txt
4. nbsu -c -t (on the master server)
5. Set ...\logs\admin\ (Global Logging Level = verbose 5)
6. Set OID 111 (EMM) to DebugLevel=6, DiagnosticLevel=6
7. Run nbdelete -allvolumes -force
8. Collect Admin and 111 logs at above verbosity.


Rgds
Prashant K

[Andy_Welburn]

Issue 1 in my original post I think I've got my head around! It may be that this is something that's always happened & possibly always will it's just that with the upgrade I've been a little more attentive to the logs.

So here goes my explanation for the "error" :

I've just had this happen again with the following reported:
Could not deassign media-id 300136, host ndmpmedia: requested media id is in use, cannot process request (97)

Now there are actually 4 tapes currently loaded & being written to, but it only reported on the one detailed above.
Checking in bpmedialist output etc for all 4 tapes, it turns out that media 300136 has only just been allocated from scratch & despite the fact that it has had  copious amount of data written to it so far, the backup to this particular tape (i.e. the only backup to it so far) has not yet finished. Consequently, NetBackup sees this as an assigned tape BUT currently is not reporting any data saved to it (kilobytes, images & valid images are all zero). Presumably this is what the cleanup process is looking for so as to deassign the volume & obviously can't as it in use.
As the other 3 tapes already have images saved to them from previous backups, they are not even considered during the cleanup process & are therefore not reported.

Seems logical to me.

- phew!

[Andy_Welburn]

Regarding my second issue:

Prashant - I've marked your post as solution for this - I take it you've just had a similar issue or you work in Support (***EDIT*** I believe the latter as the files I just uploaded to support are currently 'owned' by a certain Prashant K... !), as these are the exact same steps (& I mean exact, almost word for word!!) that support asked me to do! ;)

From the output they identified 3 backup images that related to a DSSU that no longer existed, on the media server that I just 'decomissioned' - these were images that as far as I was concerned shouldn't have been on the system for at least a year & belonging to clients that only ever existed for a few weeks for testing!

For info and completeness, once identifed I was asked to run for each image:
bpexpdate -backupid <backupid> -d 0 -force
and
bpexpdate -backupid <backupid> -d 0 -force -nodelete
These failed with "no entitiy was found" - which was as expected by all parties!

Followed by:
nbdelete -allvolumes -force
and
nbdelete -backup_id <backupid> -force
for the three images - no error message was received when running this command so it was hoped that this had resolved the issue.

Still got "47:host unreachable" at the end of the image cleanup process :(. So, provided further output to support (admin & bpdbm logs, vxlogs for nbemm & nbdb_unload output) ....... it is now being excalated to back-line for the necessary sql to correct the entries.

Many thanks to you all for your input on both issues in this post (& special mention to Bob for scaring the bejeesus out of me 'cos I didn't run an NBCC in the first instance!! ;) )

[Andy_Welburn]

I have just received the necessary sql to correct the invalid entries, but I have a question for you guys (I did pass this same query back to support but as yet have had no response):

Prior to applying the EEB I have been supplied, I need to "Ensure that the only NBU service running is the NetBackup ASA service".

Now this may sound like a stupid question, but, what's the best/easiest/recommended way of achieving this?