Jobs hang awaiting resources even though tapes are in robot

[aljack3]

We have two 6.5.5 media servers attached to one 6.5.5 master.

Media Server 1 works perfectly on all counts.

Media Server 2, which was recently added, allows robots to be added, configured and inventoried.  Everything goes smoothly during the setup but when media in a robot attached to Media Server 2 is required for a job, the job hangs at "Operator action may be required.  Pending Action: No action."

See output below,

4/26/2012 6:24:17 PM - begin Duplicate

4/26/2012 6:24:17 PM - requesting resource mediaserver2-hcart-robot-tld-5

4/26/2012 6:24:17 PM - requesting resource A12345

4/26/2012 6:24:17 PM - reserving resource A12345

4/26/2012 6:24:17 PM - reserving resource A67890

4/26/2012 6:24:17 PM - awaiting resource A12345 A pending request has been generated for this resource request.

 Operator action may be required. Pending Action: No action.,

 Media ID: A12345, Barcode: A12345, Density: dlt3, Access Mode: Read,

 Action Drive Name: N/A, Action Media Server: N/A, Robot Number: N/A, Robot Type: NONE,

 Volume Group: 000_00008_TLD, Action Acs: N/A, Action Lsm: N/A

The two media servers and the robots attached to them all appear to be setup exactley the same.  Only robots attached to one works (Media Server 1) and robots attached to one do not (Media Server 2).  The density of the drive in the robot matches the density of the tape (DLT3) and the drive lists TLD for control.  The density of the storage unit for the robot is also DLT3.  Running robtest on Media Server 2 allows for the loading and unloading of tapes from the drives.

Now if a robots is moved from Media Server 2 to Media Server 1, after the device configuration for both are updated, the robot starts to work normally, so the problem is with Media Server 2.

One more observation.  Media Server 2 has several DLT3 robots and 1 HCART robot.  Catalog backups can be written to the HCART attached to Media Server 2 without a problem.

Is there something we are missing in the confguration of Media Server 2?

[watsons]

What is the state of your media server 2?

Run this from master:

/usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts -verbose

Look for the media server2 details, is its state "Active for Tape & Disk"? If it's only "active for disk", your tape backup won't work. If that is the case, use this technote to set it to tape active , or reset the master server connectivity:

http://www.symantec.com/docs/TECH168379

[Marianne]

Double-check densities as well as tape location.  Is A12345 in tld(5)? 
From what I can see, there is probably a mismatch between the STU, drive type and media density:

mediaserver2-hcart-robot-tld-5

tells me that the STU density (and probably drives as well) is set to hcart.

This says to me that the media density is dlt3:

Media ID: A12345, Barcode: A12345, Density: dlt3,

In NBU, ALL densities MUST match - STU, drive, media.
NBU will never assign dlt3 media to hcart STU or mount it in hcart drive.

Please share output of the following commands:

On master (commands are in ...netbackup/bin/admincmd:
nbemmcmd -listmedia -mediaid A12345
bpstulist -label mediaserver2-hcart-robot-tld-5 -L

On media server (commands are in ....volmgr/bin):
tpconfig -l
vmoprcmd -d
 

[aljack_half]

Active for Tape & Disk

[aljack_half]

If you are using a robot for source tapes only and will never write to that robot does it require an STU entry?

[aljack_half]

Thanks for taking a crack at this very frustrating problem.  I've since added another new media server and drives attached to this server also suffer from the same problem.

The above output is from a duplication attempt.  We are duplicating SDLT2 (DLT3) to LTO4 (HCART).

1, mediaserver2-hcart-robot-tld-5 is the destination robot for the duplication which is HCART.  TLD(8) is the source SDLT600 robot which is DLT3.

2. Running the above 4 commands return DLT3 density everytime.

We have crawled some of the log files and it looks like the request for the media is not finding it's way from the master the media server.  The media server BPTM log looks like this,

10:48:12.484 [3532.3372] <2> bptm: INITIATING (VERBOSE = 0): -rptdrv -jobid -1336183153 -jm 10:48:12.484 [3532.3372] <2> main: Sending [EXIT STATUS 0] to NBJM10:48:12.484 [3532.3372] <2> bptm: EXITING with status 0 <----------

If you move the tape to the working media server then perform the duplication you see something like this in the BPTM log of the media server,

05:28:08.703 [1220.528] <2> report_drives: DRIVE = QUANTUM.SDLT600.001 LOCK = TRUE CURTIME = 1336393688

05:28:08.703 [1220.528] <2> drivename_close: Called for file QUANTUM.SDLT600.001

05:28:08.703 [1220.528] <2> report_drives: MODE = 0

05:28:08.703 [1220.528] <2> report_drives: TIME = 1336384850

05:28:08.703 [1220.528] <2> report_drives: MASTER = masterserver

05:28:08.703 [1220.528] <2> report_drives: SR_KEY = 0 1

05:28:08.703 [1220.528] <2> report_drives: PATH = {1,0,6,0}

05:28:08.703 [1220.528] <2> report_drives: MEDIA = A12345

05:28:08.703 [1220.528] <2> report_drives: REQID = 477

05:28:08.703 [1220.528] <2> report_drives: ALOCID = 227936

05:28:08.703 [1220.528] <2> report_drives: RBID = {A2AA932E-2788-4887-8AAD-689778803549}

05:28:08.703 [1220.528] <2> report_drives: PID = 3648

05:28:08.703 [1220.528] <2> report_drives: FILE = C:\Program Files\Veritas\NetBackup\db\media\tpreq\drive_QUANTUM.SDLT600.001

05:28:08.703 [1220.528] <2> drivename_open: Called with Create 0, file QUANTUM.SDLT600.002

05:28:08.703 [1220.528] <2> drivename_checklock: Called

05:28:08.703 [1220.528] <2> drivename_checklock: File is locked

Stuff like that never shows up in the not working media servers.

[aljack_half]

Turns out the media was still assigned to another media server.  Assigning the media to the new media server solves the issue.