Virtual tapes / NDMP host ? urgent problem

[posie80]

Hi all,
I have a weird issue pertaining to the VTL tapes connected to a media host (which is also a master server) right now.
It appears that since last night, it was complaining that there are no tapes available that can be used for backup :

08/02/2010 11:10:41 - Error nbjm (pid=16301) NBU status: 96, EMM status: No media is available
unable to allocate new media for backup, storage unit has none available (96)


The funny thing is,  looking at the robot and checking the list of virtual tapes that should be used for this backup, I see many scratch tapes that can be used. These tapes arent frozen (they are in Active status) and look normal, and usually they will be used for backup without any issue. This time, it seems that Netbackup is somehow 'ignoring' these tapes. The robot 108 Volumes (all virtual tapes coming from our EMC CDL) and almost half of them are scratch tapes. Half of them are currently full and assigned to the correct volume pools, and these ones can be used for backup, if I expire them. I dont understand why the other half isnt being used....

These tapes are used for NAS backups (the virtual tapes are assigned to the NDMP host), which is currently failing.

bpmedialist -h <ndmp host> only shows the tapes that can currently be used (right now they are all full). The tapes that are in scratch pool are not shown. I even tried manually to change the volume pool of one scratch tape into the correct volume pool that the backup policy is using, and triggered the backup again, in hope that Netbackup would take the assigned tape. But it didnt and the backup failed...

The barcode rule for this robot hasnt been changed. We have set a default barcode rule for this robot (1/2" cartridge tape 2)

Any ideas?? I'm also investigating myself but I thought it may be likely that its something very obvious I'm overlooking at the moment, so I appreciate if any of you can give me hints!

Netbackup 6.5.4 on Solaris 9.

Thanks
Posie

P/s : anybody else other than me experiencing issues with Symantec technotes? I'm having gateway timeouts trying to open the redirect links...

[posie80]

Oh forgot to mention, I've also tried to re-inventory the robot but it seems everything is updated and I see the same volumes/tapes assigned.

[posie80]

Ok....I have managed to resolve (?) it by deleting all the scratch tapes from the DB and re-inventory-ing the robot. The tapes came back (as Netbackup tapes and I changed them to 'scratch' pool). This time they work just fine and backup is currently going on...

I have no idea what caused it though...I have a feeling that the tapes' config were corrupt somehow but I dont quite know how to verify this for next time? Are there any steps you can take to see if media configs got corrupted?

Before I fixed this issue I ran the following command to compare between a tape that can be used for backup and a tape that cannot be used for backup :

Cannot be used :

# vmquery -m D7U50A
================================================================================
media ID:              D7U50A
media type:            1/2" cartridge tape 2 (14)
barcode:               V8D7U50A
media description:     Added by Media Manager
volume pool:           NAS_POOL (6)
robot type:            TLD - Tape Library DLT (8)
robot number:          7
robot slot:            11
robot control host:    master_server
volume group:          00_007_TLD
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               Mon Aug 02 12:05:10 2010
assigned:              Mon Aug 02 12:07:40 2010
last mounted:          Mon Aug 02 12:07:54 2010
first mount:           Mon Aug 02 12:07:54 2010
expiration date:       ---
number of mounts:      1
max mounts allowed:    ---


================================================================================

Can be used :

# vmquery -m D7U50V
================================================================================
media ID:              D7U50V
media type:            1/2" cartridge tape 2 (14)
barcode:               V8D7U50V
media description:     Added by Media Manager
volume pool:           NAS_POOL (6)
robot type:            TLD - Tape Library DLT (8)
robot number:          7
robot slot:            32
robot control host:    master_server
volume group:          00_007_TLD
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               Mon Apr 28 11:31:39 2008
assigned:              Mon Aug 02 09:21:48 2010
last mounted:          Mon Aug 02 09:22:10 2010
first mount:           Wed Jun 10 01:30:20 2009
expiration date:       ---
number of mounts:      186
max mounts allowed:    ---
status:                0x0
================================================================================


The only difference I saw was the tape that CAN be used (D7U50V) has the status info in the vmquery output and the one that couldnt be used (D7U50A) didnt have this entry at all..
I'm not sure what this entry meant, but after the deletion and re-inventory, I ran the same command and this time I could see the status info for both tapes.
All other info (density, etc) look correct before and after deletion and re-inventory.

I'm keen to find the root cause of this issue...appreciate if any of you can volunteer your input.

Thanks
Posie

[Marianne · Accepted solution]

media ID:              D7U50V
assigned:              Mon Aug 02 12:07:40 2010

Check EMM status of this tape to get info about the assigned status:

nbemmcmd -listmedia -mediaid D7U50V