NetBackup console is asking different tape (which is not in the tape drive)

[Guruh_Sulistiyo]

Hi,

I have several sites in NetBackup. Each site will use either single tape drive or TL2000 to backup the image to the tape. For some of the sites using single tape drive. I notice there are 1 or 2 site having problem with the backup image to the tape (D2T).

Let say about California site. For this site we have 4 tapes which are LTO5 tape. I soft label the tape with name as CA0001, CA0002, CA0003 and CA0004. I created new volume pools (named it as CA_Volume_Pool) for this site and move all those 4 tapes to this volume pool.

What I don't understand is, sometime in the blue moon. The D2T process is not working properly for this site. It's stuck in the process of writing and asking different tape altough there is a tape (which is less kilobytes written to the tape) inside the tape drive. Let say CA0002 is inside the tape drive. NetBackup console is keep asking for CA0001. Even after I denied the request or resubmit the request in the device monitor, it's still asking for CA0001.

Why NetBackup is asking for different tape altough there is a tape inside the tape drive (and still have much space)? Is there a way to force NetBackup to proceed with the D2T process with the tape we have inside the tape drive?

Due to this problem. Our tape rotation is a bit mess up. Because sometime NetBackup console is asking previous tape which we already sent out off site. 

We have around 15 sites using single tape drive. Rest of the site ares okey. NetBackup console never ask for pending tape to be inserted in the device monitor. It always write to the tape which is inside the tape drive, and the backup is always running smoothly everytime our field service do the rotation of the tape. But I am not sure why only these 2 sites having this problem.

Any advice will be much appreciate.

[Nicolai]

Netbackup does not mix retension per defautl. So if backup A is 4 weeks, but backup B is 3 weeks (e.g due to typo) Netbackup will ask for another tape.

Also check volume pool names - diffrent volume pool names will also result in a tape mount request.

[Marianne]

It is possible that CA0001 was previously used, is not full, fits all criteria for the current job. NBU now chooses this tape to append to.

If you don't want to append to previously written tape, suspend the tape as soon as you eject it from the drive.

Just a quick clarification - does 'soft label' mean you are pre-labelling tapes using bplabel?

One more thing - NBU is an enterprise product. Although it supports standalone tape, it really does not play well with it and requires way too much manual intervention.
There is a section in NBU Admin Guide II about tape selection and management with standalone drives. You may want to look it up to see best practice recommendations?

[Guruh_Sulistiyo]

The D2T request coming from SLP. The rention period for the tape is 4 weeks. I even put a fresh new empty tape into the tape drive, to see whether the NetBackup will try to write into the tape existing tape in tape drive or not. But it still asking for another tape (which is not in the tape drive)

I double checked the volume pool already. I put all the tapes (CA0001 -CA0004) into one Volume pool which is CA_Volume_Pool.

[Mike_Gavrilov]

Guruh, it's very strange. NetBackup asks for resource in the beginning, then it gets drive and tape its writes data on it. If he thinks that the media is full it asks for the next  available but I haven't seen NBU asking for particular media for backup (if we're not talking about classic Synthetic backup). Could you share JobDetails with this strange request?

[Guruh_Sulistiyo]

Hi Marianne,

Yes, I notice sometime when we eject CA0001 from the tape drive. The tape is not full yet. But we need to do the tape rotation to make sure we secure the data and send it offsite right. Why NBU want to append the data? Shouldn't be it work like what ever tape we put into the tape drive, NBU will write into that tape? Or is there away to force NBU to write the data into the tape with what ever tape inside the tape drive?

Yes correct. I did the softlabel with bplabel command. The full command is bplabel -m CA0001 -d hcart2 -o -p CA_Volume_Pool -u 0.

I undestand your concern about the standalone tape drive. But this is what we have for now, and I don't think management is going to buy Tape Library for this. For now, we just need to bear with this condition and solve the problem... :(

[Guruh_Sulistiyo]

Hi Mike,

Yes, it very weird issue since only 1 or 2 site (out of 15 sites) having this problem.

Here is the Job Details from Activity Monitor. From he it self, it's clearly asking for CA0001 instead of the tape which already in the tape drive (CA0002)

08/03/2016 01:12:33 - begin Duplicate
08/03/2016 01:12:33 - requesting resource LCM_cabv140-hcart2
08/03/2016 01:12:33 - granted resource  LCM_cabv140-hcart2
08/03/2016 01:12:33 - started process RUNCMD (pid=15861)
08/03/2016 01:12:34 - ended process 0 (pid=15861)
08/03/2016 01:12:34 - requesting resource cabv140-hcart2
08/03/2016 01:12:34 - requesting resource @aaaa6
08/03/2016 01:12:34 - reserving resource @aaaa6
08/03/2016 01:12:34 - awaiting resource cabv140-hcart2.
          A pending request has been generated for this resource request.
          Operator action may be required. Pending Action: No action,
          Media ID: CA0001, Barcode: --------, Density: hcart2, Access Mode: Write,
          Action Drive Name: N/A, Action Media Server: N/A, Robot Type(Number): 0(N/A),
          Volume Group: ---, Action Acs: N/A, Action Lsm: N/A

[Genericus]

I would recommend killing the duplication to tape job at your console, and letting the SLP retry.

You should inventory the robot and make sure it knows that ca1 is no longer there.

When the retry starts, it should ask for a different tape.

Only reason it should ask for a specific tape is if it was already using that, or it thinks that is the next available tape.

If you are manually ejecting tapes, you need to inventory after so NetBackup knows which tapes remain.

[Guruh_Sulistiyo]

Hi Genericus,

I killed all the SLP process. But after 10 mins, the SLP retry the process and still asking for tape which is not in the tape drive.

As I mentioned in my 1st post. This is not a tape library, it's a standalone tape drive. We couldn't inventory the tape for a standalone tape drive and the tape will be showing automatically in the media and device management - device monitor.

[Marianne]

We need to find out why NBU will not use the tape in the drive.
When pending request is generated, run commands such as
bpmedialist -m (media-id)
vmquery -m (media-id)
bptm log on this media server may also tell us why NBU won't use the tape in the drive.

There are also nbemmcmd settings that can be changed for robotic-type media selection or for standalone media selection that is explained in Admin Guide II. (I do not have access to manuals right now.)

[Guruh_Sulistiyo]

Hi Marianne,

Let me try this command when the problem appear again. Right now, I just put the tape which is requested by NBU.

I am new with NetBackup. Would you please let me know how to check the btpm log in media server? Our media server is Windows 2008.

[Marianne]

bptm log folder must be created under netbackup\logs.
Log file with matching date stamp can be opened with Notepad or other text editor.

[Guruh_Sulistiyo]

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpmedialist -m CA0002
Server Host = cabv140.specialties.int.huntsman.com

 id     rl  images   allocated        last updated      density  kbytes restores
           vimages   expiration       last read         <------- STATUS ------->
--------------------------------------------------------------------------------
CA0002   3     51   08/09/2016 09:00  08/09/2016 19:47  hcart2   625094119     0
               51   09/09/2016 19:07        N/A

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpmedialist -m CA0004
Server Host = cabv140.specialties.int.huntsman.com

 id     rl  images   allocated        last updated      density  kbytes restores
           vimages   expiration       last read         <------- STATUS ------->
--------------------------------------------------------------------------------
CA0004   0      0   08/10/2016 09:27        N/A         hcart2           0     0
                0         N/A               N/A

C:\Program Files\Veritas\Volmgr\bin>vmquery -m CA0002
================================================================================
media ID:              CA0002
media type:            1/2" cartridge tape 2 (14)
barcode:               --------
media description:     ------
volume pool:           CA_Volume_Pool (7)
robot type:            NONE - Not Robotic (0)
volume group:          ---
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               8/1/2016 2:41:01 PM
assigned:              8/9/2016 9:00:10 AM
last mounted:          8/9/2016 7:08:11 PM
first mount:           8/2/2016 4:04:01 PM
expiration date:       ---
number of mounts:      3
max mounts allowed:    ---
status:                0x0
================================================================================

C:\Program Files\Veritas\Volmgr\bin>vmquery -m CA0004
================================================================================
media ID:              CA0004
media type:            1/2" cartridge tape 2 (14)
barcode:               --------
media description:     ------
volume pool:           CA_Volume_Pool (7)
robot type:            NONE - Not Robotic (0)
volume group:          ---
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               8/1/2016 2:41:08 PM
assigned:              8/10/2016 9:27:31 AM
last mounted:          ---
first mount:           ---
expiration date:       ---
number of mounts:      0
max mounts allowed:    ---
status:                0x0
================================================================================

Here is the bptm log from media server when it's asking for a tape.

08:47:53.395 [900.4728] <2> bptm: INITIATING (VERBOSE = 0): -rptdrv -jobid -1465491638 -jm
08:47:53.395 [900.4728] <2> main: Sending [EXIT STATUS 0] to NBJM
08:47:53.395 [900.4728] <2> bptm: EXITING with status 0 <----------
08:57:52.247 [4760.3872] <2> bptm: INITIATING (VERBOSE = 0): -rptdrv -jobid -1465491643 -jm
08:57:52.263 [4760.3872] <2> main: Sending [EXIT STATUS 0] to NBJM
08:57:52.263 [4760.3872] <2> bptm: EXITING with status 0 <----------
09:27:31.694 [3680.3684] <2> bptm: INITIATING (VERBOSE = 0): -rptdrv -jobid -1465491653 -jm
09:27:31.694 [3680.3684] <2> main: Sending [EXIT STATUS 0] to NBJM
09:27:31.694 [3680.3684] <2> bptm: EXITING with status 0 <----------
09:27:52.036 [3908.3912] <2> bptm: INITIATING (VERBOSE = 0): -rptdrv -jobid -1465491658 -jm
09:27:52.036 [3908.3912] <2> main: Sending [EXIT STATUS 0] to NBJM
09:27:52.036 [3908.3912] <2> bptm: EXITING with status 0 <----------

[Guruh_Sulistiyo]

From above result, could you please advice why NBU is asking CA0004 instead of CA0002 which already inside the tape?

[Marianne]

Are you sure that the bptm log is from media server cabv140?
There is no indication of any backup attempt or tape request.

So, this time you have 2-day old tape CA0002 (with a bunch of valid images at level 3) in the drive on media server cabv140 but NBU is instead asking for new tape CA0004?
Is the duplication part of this particular SLP set to Ret 3 as well?

We need to see correct bptm log...
Please double check location of the log?
It may be a good idea to increase logging level - use Host Properties -> media server to select relevant server and increase logging level for bptm to 3.

[Guruh_Sulistiyo]

Yes, that is the bptm log from the media server cabv140.

Yes, it's asking for CA0004 which is a new tape and empty. How to check the duplication part of this particular SLP set to Ret 3? From what I can tell, I create the policy and use the same SLP only. So by right the setting should be same right?

I just increased the logging level to 3 and trigger the backup again. Will update the status of the bptm log later.

[Marianne]

If all requirements (media server, pool, retention, status, etc) match the tape that is already in the drive, then NBU should by default select the tape in the drive and not request another tape.

Can you confirm that the tape in the drive is picked up by this command on the media server?
vmoprcmd -d

I am hoping to see more in media server bptm log. That log did not look like there was any backup or duplication running.

[Marianne]

Please also check output of this command for problematic media server - we need to see if the default standalone drive behaviour is in place here:

nbemmcmd -listsettings -machinename <media-server-name>

I am looking for DISABLE_STANDALONE_DRIVE_EXTENSIONS setting. Explanation of this setting:

DISABLE_STANDALONE_DRIVE_EXTENSIONS 0|1|no|yes

Disables non-robotic drive operations. During a backup, NetBackup does not automatically try to use whatever labeled or unlabeled media it finds in a non-robotic drive. Default condition is that stand-alone drive extensions are enabled.

Please also read through this section in NBU Admin Guide II:

About selecting media in standalone drives

[Guruh_Sulistiyo · Accepted solution]

Hi All,

Altough I am not able to fix this issue permanently. But I have some workaround to force the NBU use the existing tape which is inside the tape.

What I did is, once new tape is inside the tape. I cancel all the pending D2T job for the current SLP which is having problem in Activity Monitor. I also denied pending job from Device Monitor. After that, I immediatelly reboot the media server (where the tape drive connected). When the media server is up, the re-run of the D2T will kicks in. This new process won't ask for different tape.

These steps did the trick. Hope it will help anyone who are having the same problem like me.