Active tapes not being utilized

[Mark_Sleeper]

I have about a dozen tapes that are well below their full capacity, are not frozen and have not reached their max number of mounts that have not been mounted and written to in months.  They show a status of 'Active' or 'ActiveMultiplexed'.
We ended up getting 96 errors last night even though these tapes were available.
We're running Netbackup 6.5.3.1, on Redhat linux 4.
Media type is LTO-2 in an ADIC S-10K.
Any advice would be appreciated.
Thanks!

[Marianne]

NetBackup will append to Active media if it's in the robot, in the same volume pool and same retention level and owned by same media server (unless media sharing was enabled) as required by backup policy that needs to run.
There's a section in the NetBackup Admin Guide (probably vol II) on how media is chosen for backups.

[J_H_Is_gone]

You did check to see if the tapes were frozen or suspened, right?

[Mark_Sleeper]

Thanks for the replies!
These tapes are not frozen, I've never suspended a tape, but I'm assuming that is shows up as Suspended under Media Status instead of Active.  Their status is Active or ActiveMultiplexed with plenty of space left on them.  Several of them have not been mounted for months yet Netbackup failed a bunch of jobs last night with (96) unable to allocate new media for backup, storage unit has none available.

[Anonymous]

Try to see if there are problems in the EMM database

nbemmcmd -listmedia -conflicts

Deal with issues identified.

I recommend reading this forum post for checks or ideas to try.
https://www-secure.symantec.com/connect/forums/re-using-tapes

[Yasuhisa_Ishika · Accepted solution]

It seems like NetBackup can suspends media by some non-critial errors.
Check logs at time these media was last mounted, if remains.

# strings bptm | grep SUSPEND
SUSPENDED media id %s, could not write backup header
SUSPENDED media id %s, could not terminate correctly
SUSPENDED media id %s, could not properly terminate backup
SUSPENDED media id %s, could not write backup header for TIR
SUSPENDED media id %s, could not properly terminate TIR data
SUSPENDED media id %s, backup completed, but could not successfully terminate media
SUSPENDED media id %s, backup completed, but could not successfully terminate media
SUSPENDED media id %s, could not terminate correctly
SUSPEND BACKUP
SUSPEND RESTORE
SUSPENDED
%s SUSPENDED
ACTIVE FULL SUSPENDED FROZEN IMPORTED
SUSPENDING media id %s, Operation requested via BPTM
UNSUSPENDING media id %s, Operation requested via BPTM
skipping media id %s, either FULL or FROZE or SUSPENDED or IMPORTED
#

[Andy_Welburn]

(we should really put together a krib sheet to check for this - it comes up quite a bit in the forum!!)

Are the tapes of the correct density? (e.g. HCART, HCART3 etc)

Have the tapes physicallly expired? I don't mean data expiration, you can put a "sell-by" date on them:

e.g. have just manually put a date on our tape 300705
vmquery -m 300705
================================================================================
media ID: 300705
media type: 1/2" cartridge tape 3 (24)
barcode: 300705
media description: Added by Media Manager
volume pool: Scratch_Pool (5)
robot type: NONE - Not Robotic (0)
volume group: ---
vault name: ---
vault sent date: ---
vault return date: ---
vault slot: ---
vault session id: ---
vault container id: -
created: Tue Feb 03 09:45:03 2009
assigned: ---
last mounted: Fri Apr 03 02:03:28 2009
first mount: Tue Feb 03 15:52:03 2009
expiration date: Thu Jul 29 08:27:17 2010
number of mounts: 8
max mounts allowed: ---
================================================================================

The expiration date highlighted should be blank for the tape to utilised.

You can also use the nbemmcmd -listmedia -mediaid <mediaid> command, which give a littel more info (status, density etc etc)

[Andy_Welburn]

May be worth looking through this TechNote:

In-depth Troubleshooting Guide for Exit Status Code 96 in NetBackup Server (tm) / NetBackup Enterpri...

[stu1]

NBU would always attempt to re-use active tapes.

it will ask EMM for an active tape first before getting a scratch tape.  When it ask EMM, it will read the policy/schedule for storage unit info, density, and volume pool.  EMM will search its database for a matching tape.  If it could not find one that is available to mount immediately, you may get a 96.

If you utilize more than 1 storage unit, density, and even volume pools be certain, you're not using different policies where each request tapes from 2 different volume pool.  ACTIVE tapes would not be re-assign to SCRATCH pool, but will be assigned to the volume pool of the policy that first uses it.  Only other policy/schedule who would ask for tapes from the same volume pool can access it's ACTIVE tapes to append to.

Then there are also density and retention levels.  NBU would also by default, append to tapes only of the same retention level.  The retention level is assigned by the first job that wrote data to that tape.

It could be true that you've met all those requirements, so then check nbrb -dump; and see if the mediaID is stuck in allocated status.  This might mean some process was killed before this mediaID was released.  Being that it might be stuck as allocated, it would make NBRB think the tape is currently allocated for another job.  So it's not logically available for use.  If you have a chance for downtime where no jobs are running you could do, nbrbutil -resetAll to clear all allocation -- but never do that while there's any job running.

In the same subject with nbrb allocation, you could also schedule downtime on each media server that could use those tapes, and STOP NBU.  goto /usr/openv/netbackup/db/media/tpreq/ and see if there are any left over files.  This directly is a temp work directly by tpreq.  tpreq function should automatically clean its work file after it's done, so there should not be any leftover after NBU is stopped cleanly.  If there is, it would mean there was a time where NBU was killed before it could cleanup.  Clean up any files inside that directory on each media server.  Restart.

[xiazhen]

hi,guys :

   the below is explain for your question.

   when you create a volume , you can set maximum  number of times that media manager should be mount volume.when a volume reaches this mount limit .the volume can be read ,but not written. specifying 0 indicate unlimit mounts .

   to specify maximum number of times

   consult your vendor documention for information on the expected life of the volumes.


  the below is status code 96


  NetBackup status code: 96
Message: unable to allocate new media for backup, storage unit has none
available Explanation: The tape manager (bptm) did not allocate a new volume for backups. This error indicates that the storage unit has no more volumes available in the volume pool for this backup. Note that NetBackup does not change storage units during the backup.

Recommended action: Check the NetBackup Problems report to determine the storage unit that is out of media.
1 If the storage unit is a robot with empty slots, add more volumes (remember to specify the correct volume pool).
■
If there are no empty slots, move some media to nonrobotic and then add new volumes.
■
If you have difficulty keeping track of your available volumes, try the available_mediascript: On UNIX, this script is in:
/usr/openv/netbackup/bin/goodies/available_media
On Windows, the script is in:
install_path\NetBackup\bin\goodies\available_media.cm
d
This script lists all volumes in the volume configuration, and augments that list with information on the volumes currently assigned to NetBackup.
2 Set up a scratch volume pool as a reserve of unassigned tapes. If NetBackup needs a new tape and none are available in the current volume pool, it does the following: moves a tape from the scratch pool into the volume pool that the backup uses.
3 If the storage unit and volume pool appear to have media, verify the following:
■
Volume is not FROZEN or SUSPENDED. Check for this condition by using the NetBackup Media List report. If the volume is frozen or suspended, use the bpmedia command to unfreeze or unsuspend it (if that is wanted).
■
Volume has not expired or exceeded its maximum number of mounts.
■
The EMM database host name for the device is correct.
If you change the EMM database host name, stop and restart the following: the Media Manager device daemon, ltid, (if the server is UNIX) or the NetBackup Device Manager service (if the server is a Windows system).
■
The correct host is specified for the storage unit in the NetBackup configuration. The host connection should be the server (master or media) with drives connected to it.
■
The Media and Device Management volume configuration has media in the correct volume pool. Unassigned or active media is available at the required retention level.

Use the NetBackup Media List report to show the retention levels, volume pools, and status (active and so on) for all volumes. Use the NetBackup Media Summary report to check for active volumes at the correct retention levels.
4 The NetBackup bptm process is rejected when it requests media from the vmd process (UNIX) or the NetBackup Volume Manager service (Windows). The cause of this problem is that the process or service cannot determine the name of the host that makes the request. This error can be due to an incorrect network configuration that involves the following:
■
Multiple network interfaces
■
/etc/resolv.conf on those UNIX systems that use it
■
Running DNS with reverse addressing not configured
5 Create bptm and vmd debug log directories and retry the operation.
6 Examine the bptm debug log to verify that bptm connects to the correct system. If an error is logged, examine the vmd log. On UNIX, the vmd log is: /usr/openv/volmgr/debug/daemon/log.xxxxxx On Windows, the vmd log is: install_path\Volmgr\debug\daemon\xxxxxx.log
7 If this storage unit is new and this attempt to use it is the first, stop and restart NetBackup on the master server.

   
   

[Mark_Sleeper]

These are some great responses!  Here's an update with what I know and what I've done:

The tapes are NOT frozen or suspended. 

We don't have a limit on the number of mounts. 

The tapes have not expired. 

The command "nbemmcmd -listmedia -conflicts" returns "Command completed successfully." and nothing else. 

There are no files present in "/usr/openv/netbackup/db/media/tpreq/"

All tapes in the library are LTO-2, so there are no media type conflicts.

I have run "nbrbutil -resetALL" ... we'll see if that works.

Here is the output of both "nbemmcmd" and "vmquery"  for one of the tapes.
As you can see from the output, this LTO-2 tape has about 125GB of data on it, so it's not full.  The number of mounts is only 7, so even if we had 'max mounts' set, it would be much higher than that.  The retention period is 8, which is one year... which is what all of our full tapes have for retention, so that shouldn't be an issue.  It's listed as "Active" under "media status" in the NB console... yet it hasn't been mounted since January!

nbemmcmd -listmedia -mediaid A00818
NBEMMCMD, Version:6.5.3.1
====================================================================
Media GUID:                     674c0050-6408-1dd5-8023-a11a837f0b45
Media ID:                       A00818
Partner:                        -
Media Type:                     HCART2
Volume Group:                   000_00000_TLM
Application:                    Netbackup
Media Flags:                    1
Description:                    ---
Barcode:                        A00818
Partner Barcode:                --------
Last Write Host:                correct.hostname.is.here
Created:                        06/13/2007 16:13
Time Assigned:                  01/23/2009 17:30
First Mount:                    06/21/2007 03:54
Last Mount:                     01/24/2009 09:10
Volume Expiration:              -
Data Expiration:                01/23/2010 17:30
Last Written:                   01/23/2009 17:30
Last Read:                      -
Robot Type:                     TLM
Robot Control Host:             NONE
Robot Number:                   0
Slot:                           -
Side/Face:                      -
Cleanings Remaining:            -
Number of Mounts:               7
Maximum Mounts Allowed:         0
Media Status:                   ACTIVE
Kilobytes:                      125655059
Images:                         2
Valid Images:                   2
Retention Period:               8
Number of Restores:             0
Optical Header Size Bytes:      1024
Optical Sector Size Bytes:      0
Optical Partition Size Bytes:   0
Last Header Offset:             1994532
Adamm Guid:                     00000000-0000-0000-0000-000000000000
Rsm Guid:                       00000000-0000-0000-0000-000000000000
Origin Host:                    NONE
Master Host:                   correct.hostname.is.here
Server Group:                   NO_SHARING_GROUP
Upgrade Conflicts Flag:
Pool Number:                    5
Volume Pool:                    Full
Previous Pool Name:             Scratch
Vault Flags:                    -
Vault Container:                -
Vault Name:                     -
Vault Slot:                     -
Session ID:                     -
Date Vaulted:                   -
Return Date:                    -
====================================================================
Command completed successfully.



vmquery -m A00818
================================================================================
media ID:              A00818
media type:            1/2" cartridge tape 2 (14)
barcode:               A00818
media description:     ---
volume pool:           Full (5)
robot type:            TLM - Tape Library Multimedia (13)
robot number:          0
robot slot:            0
robot control host:    NONE
volume group:          000_00000_TLM
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               Wed 13 Jun 2007 04:13:21 PM MDT
assigned:              Fri 23 Jan 2009 05:30:21 PM MST
last mounted:          Sat 24 Jan 2009 09:10:23 AM MST
first mount:           Thu 21 Jun 2007 03:54:32 AM MDT
expiration date:       ---
number of mounts:      7
max mounts allowed:    ---
status:                0x0
================================================================================


Thanks again for all of your help!
Mark

[lknishad]

Hi,
       If the media is allready having images and the tape is active and you are getting the 96 status code.
please check the optioin multiple retention is enable or not.

for example->

if the tape having images of retention level 2 and your new policy have diffrent retention, you will get the status code 96 if multiple retention is not enabled.


Thanks & Regards
L.K.Nishad.