Robotic library is down on server, Media Server

[john10]

Hello All,

Currently we are using NBU 7.0.1 with 2008 as Master/Media server, and backup are not running fine in job details it was showing as "Robotic library is down on server, Media Server" and when i executed "vmoprcmd -d" the drive status are showing as in "AVR" mode , i recycled NBU services couple of time even though same AVR and i tried to load the media by clicking on media but is has give an error like "error connecting to Oprd on "X". network protocol error(39)" , and after some time im able to load the media's, and some outputs are below, hope any one can help.

Note:-Rebooted the Library twice.

Stuck job screen shot:-

Stuck Job details:-

1/10/2014 1:31:23 PM - requesting resource ko20w1000-hcart2-robot-tld-0
1/10/2014 1:31:23 PM - requesting resource ko20w1000.NBU_CLIENT.MAXJOBS.ko20w1000
1/10/2014 1:31:23 PM - requesting resource ko20w1000.NBU_POLICY.MAXJOBS.FULL_KO20NT100
1/10/2014 1:31:23 PM - awaiting resource ko20w1000-hcart2-robot-tld-0 Reason: Tape media server is not active, Media Server: ko20w1000,
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
     Volume Pool: DataStore, Storage Unit: ko20w1000-hcart2-robot-tld-0, Drive Scan Host: N/A
    
1/10/2014 1:37:07 PM - awaiting resource ko20w1000-hcart2-robot-tld-0 Reason: Robotic library is down on server, Media Server: ko20w1000,
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,

Vmoprcmd -d Output:-

C:\Program Files\Veritas\Volmgr\bin>vmoprcmd.exe -d

                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart2   AVR                -                     No       -         0
  1 hcart2   AVR                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 IBM.ULT3580-HH5.000   No       -
  1 IBM.ULT3580-HH5.001   No       -
 

[RiaanBadenhorst]

Hi,

The media server has lost connection to the robotic arm (medium changer). You can check if the its still listed in device manager but on Windows even if it is, it might be bogus. Reboot your media server and then check with vmoprcmd if the drive control status changes from AVR to TLD.

Make sure you see the medium changer in the device manager after reboot. If the vmoprcmd doesn't show TLD, try to rerun the storage device conifiguration wizard.

[RamNagalla]

i guess you are using the TLD robot.

what is the libraty model .

show us the output of scan from the robot control host

and also check if you are able to perfrom the robtest and list drives from robtest or not.

[Marianne]

In support of above excellent posts, Windows Device Manager is the first place to confirm that the OS can see the robotic arm.

To confirm that the robot responds to scsi commands, use the 'scan' command from cmd (Remember to right-click, Run as Administrator).
Command is in <install-path>\veritas\volmgr\bin.

[john10]

HI Both,

Thanks for the reply, robtest and scan are also not working and also Inventry, they are not giving any output just in execution state for too long time, and i did previosuly only for robtest it shown as cant open robotic utily and for Inventry also error and for scan below are the outputs results what i got yesterday., for todays its still not shown any error till now.

SCAN Output:-

[john10]

Hi and im able to detect them in Os and when i tried to delete the Drives and Robot from NBU end tried to configure from Wizard at that time its showing Both the drives and Robot also.

[RamNagalla · Accepted solution]

its looking like an libraary issue..

if you have webconsole access to the library please check in the there if it is showing any alerts...

if not try to check the robot functionlaty from the webconsole by mounthing the tapes into drive.

you may need to check with library vendor.

reboot of library also worth trying...

[Marianne]

I am battling to understand what you are saying here:

Hi and im able to detect them in Os and when i tried to delete the Drives and Robot from NBU end tried to configure from Wizard at that time its showing Both the drives and Robot also.

Does this mean that the OS can see the devices?

Is the wizard now seeing the devices?

If your devices are SAN attached, use the HBA tools for the Fibre card (e.g. SanSurfer for QLogic hba) to enable Persistent Binding. 
If no Persistent Binding is in place, OS device paths may change when server is rebooted, resulting in NBU no longer able to see/access devices at previously defined paths.

[john10]

Hello Marianne,

Yes i'm able to detect in OS end also in NBU end in Wizard while trying to reconfigure(as detect 2 tape drives and 1robot).

and also we rebooted the Library also, but when we run the Jobs its showing error as "robotic library is down".

Thanks in Advance.

[Marianne · Accepted solution]

I am wondering about your screenshot showing 'I/O device error' when you ran scan command. 

Have you done anything to fix this? Like rebooting the robot and the server?
Is scan output now showing correct robot and drive output?

Did you complete the device wizard config? Including restart of NBU Device Manager?

Please show us config and status after device wizard has completed (including restart):

tpconfig -l 

vmoprcmd -d

[revarooo · Accepted solution]

Agree with Marianne above. Scan is nothing to do with NetBackup, so if you get I/O errors there is a problem at hardware/connectivity/OS level.

[Nicolai · Accepted solution]

The device may very well be there, but it is not possible to perform any operation.

Check the robot, I bet some red LED is flashing in the service required panel.

• Check robot path for obstructions
• Check the robot is fully initialized (door opened ?)

[mph999]

If scan does not show the robot (and /or) drives, you have a problem outside of NetBackup.

scan simply sends some scsi commands to the library/ drives to ask them to provide information about themselves.  Apart from a bit of formatting (to make the putput look nice) what you 'should' see in the scan command output is SENT from the hardware device.

Therefore, if it doesn't work, with there is a connectivity issue with the path to the device, or the device has a fault.

I appreciate scan is provided by Symantec, but it is not a NetBackup command - it simply sends scsi commands to the devices that are configured in the Operating System.

[john10]

Hello All,

Thanks for your suggestions.

and Marianne below are the out puts, and we rebooted the Library couple of time and, as the server is prodution we are going to Reboot it today and i will update the status once it was done.

Tpconfig _L output:-

and when itried to configure the tape drives and Robot from Wizard below is the error i faced.

[Marianne]

tpconfig -l shows that a robot is configured with no tape drives. 

Please show us the screen at the back?

We can see something saying '.... or drives removed from robot'.

Please use Device Monitor to rescan OS for devices, then run 'scan' again and show us the output. 
(Please copy the text in the output  - easier to post than a screenshot...)

Have you checked OS Event Viewer logs for device errors? 
Check System and Application logs.

PS:
Is this server running other applications as well?
Your robot control host should ideally not run any applications other than NBU. 
The reason for this recommendation is because of what you are experiencing right now:
Device errors need to be checked and fixed at OS level. Windows servers often need a reboot to fix such issues. If NBU is the only application on the server, a reboot should not be a problem.

[john10]

Hi All,

Sorry for the delay, i was on Leave.

The Masrer/Media server and Tape Library was reboted still the issue was not fixed, when i excute the Robtest and scan its showing error, and when i try to delete the Storage devices and try to configure from Wizard its taking very Long time to scan the devices.

Below are the error messages.

C:\Program Files\Veritas\Volmgr\bin>vmoprcmd.exe -d

                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart2   AVR                -                     No       -         0
  1 hcart2   AVR                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 IBM.ULT3580-HH5.000   No       -
  1 IBM.ULT3580-HH5.001   No       -

C:\Program Files\Veritas\Volmgr\bin>tpconfig.exe -l
Device Robot Drive       Robot                    Drive                Device
Type     Num Index  Type DrNum Status  Comment    Name                 Path
robot      0    -    TLD    -       -  -          -                    {8,0,3,1}
drive      -    0 hcart2    -      UP  -          IBM.ULT3580-HH5.000  {8,0,3,0}
drive      -    1 hcart2    -      UP  -          IBM.ULT3580-HH5.001  {8,0,4,0}

C:\Program Files\Veritas\Volmgr\bin>




C:\Program Files\Veritas\Volmgr\bin>scan
************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
************************************************************
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
DeviceIoControl() error on bus 0, target 3, lun 1 (1117:The request could not be performed because of an I/O device error. )
------------------------------------------------------------
Device Name  : "Tape0"
Passthru Name: "Tape0"
Volume Header: ""
Port: 8; Bus: 0; Target: 3; LUN: 0
Inquiry    : "IBM     ULT3580-HH5     C7R3"
Vendor ID  : "IBM     "
Product ID : "ULT3580-HH5     "
Product Rev: "C7R3"
Serial Number: "1068082526"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-HH5     1068082526"
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : ""
Passthru Name: ""
Volume Header: ""
Port: 8; Bus: 0; Target: 3; LUN: 1
Inquiry    : "IBM     3573-TL         B.20"
Vendor ID  : "IBM     "
Product ID : "3573-TL         "
Product Rev: "B.20"
Serial Number: "00X2U78Z0500_LL0"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     3573-TL         00X2U78Z0500_LL0"
Device Type    : SDT_CHANGER
NetBackup Robot Type: 8
Removable      : Yes
Device Supports: SCSI-5
Number of Drives : 0
Number of Slots  : 0
Number of Media Access Ports: 0
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape1"
Passthru Name: "Tape1"
Volume Header: ""
Port: 8; Bus: 0; Target: 4; LUN: 0
Inquiry    : "IBM     ULT3580-HH5     C7R3"
Vendor ID  : "IBM     "
Product ID : "ULT3580-HH5     "
Product Rev: "C7R3"
Serial Number: "1068082530"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-HH5     1068082530"
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0

C:\Program Files\Veritas\Volmgr\bin>
 

[Marianne]

Repeat of my question asked on 13 January:

Have you checked OS Event Viewer logs for device errors? 
Check System and Application logs.

Looking at scan output, the robot seems to be faulty. 
The robot itself does not know about any tape drives or slots:

Inquiry    : "IBM     3573-TL         B.20"

Vendor ID  : "IBM     "
Product ID : "3573-TL         "
Product Rev: "B.20"
Serial Number: "00X2U78Z0500_LL0"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     3573-TL         00X2U78Z0500_LL0"
Device Type    : SDT_CHANGER
NetBackup Robot Type: 8
Removable      : Yes
Device Supports: SCSI-5
Number of Drives : 0
Number of Slots  : 0
Number of Media Access Ports: 0
Flags : 0x0
Reason: 0x0

You need to log a call with the robot vendor.

[john10]

Hi Marianne,

Below are the error mesg found in Eventvwr

TLD(0) mode_sense ioctl() failed: The operation completed successfully.

[Marianne]

More evidence that you need to log a call with your hardware vendor...

[john10 · Accepted solution]

Hello All,

We had a call with hardware vendor, and got replace the SAS cable and now the issue was fixed, thanks for your support and suggestions.