TLD(0) going to DOWN state, status: Unable to sense robotic device

[knox890]

Team,

Need your help on below.

I recently upgraded my NBU from 6.5 to 6.5.6 to fix the error "nbemm stopping"

After that, the master server cannot detect my TLD 0 device. See more information on below.

after running dmesg

TLD(0) drive read_element_status error

TLD(0) key = 0x2, asc = 0x4, ascq = 0x3, LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED

TLD(0) going to DOWN state, status: Unable to sense robotic device

after running robtest

MODE_SENSE complete

Enter tld commands (? returns help information)

init

Initiating INITIALIZE_ELEMENT_STATUS

initialize_element_status failed

sense key = 0x2, asc = 0x4, ascq = 0x3, LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED

after running /usr/openv/volmgr/bin/scan

************************************************************

*********************** SDT_TAPE    ************************

*********************** SDT_CHANGER ************************

*********************** SDT_OPTICAL ************************

************************************************************

------------------------------------------------------------

Device Name  : "/dev/rmt/6cbn"

Passthru Name: "/dev/sg/c0tw500104f0007f1079l0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  M21S"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "M21S"

Serial Number: "HU106475F7"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/7cbn"

Passthru Name: "/dev/sg/c0tw500104f0007f107cl0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  M21S"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "M21S"

Serial Number: "HU10645M50"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/sg/c0tw2103001b32653b82l0"

Passthru Name: "/dev/sg/c0tw2103001b32653b82l0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "STK     SL500           0.40"

Vendor ID  : "STK     "

Product ID : "SL500           "

Product Rev: "0.40"

Serial Number: "0TBSW00200"

WWN          : ""

WWN Id Type  : 0

Device Identifier: "STK     SL500           0TBSW00200  "

Device Type    : SDT_CHANGER

NetBackup Robot Type: 8

Removable      : Yes

Device Supports: SCSI-3

Number of Drives : 6

Number of Slots  : 575

Number of Media Access Ports: 15

Drive 1 Serial Number      : "0TBSW00201"

Drive 2 Serial Number      : "0TBSW00202"

Drive 3 Serial Number      : "0TBSW00203"

Drive 4 Serial Number      : "0TBSW00204"

Drive 5 Serial Number      : "0TBSW00205"

Drive 6 Serial Number      : "0TBSW00206"

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/11cbn"

Passthru Name: "/dev/sg/c0tw2103001b32653b82l1"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  G27D"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "G27D"

Serial Number: "0TBSW00201"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/10cbn"

Passthru Name: "/dev/sg/c0tw2103001b32653b82l2"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  G27D"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "G27D"

Serial Number: "0TBSW00202"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/9cbn"

Passthru Name: "/dev/sg/c0tw2103001b32653b82l3"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  G27D"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "G27D"

Serial Number: "0TBSW00203"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/8cbn"

Passthru Name: "/dev/sg/c0tw2103001b32653b82l4"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  G27D"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "G27D"

Serial Number: "0TBSW00204"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/5cbn"

Passthru Name: "/dev/sg/c0tw2103001b32653b82l5"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  G27D"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "G27D"

Serial Number: "0TBSW00205"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/4cbn"

Passthru Name: "/dev/sg/c0tw2103001b32653b82l6"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  G27D"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "G27D"

Serial Number: "0TBSW00206"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/2cbn"

Passthru Name: "/dev/sg/c0tw500104f0007f1085l0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  M63S"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "M63S"

Serial Number: "HU10751WBN"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/sg/c1tw500104f0007f106fl0"

Passthru Name: "/dev/sg/c1tw500104f0007f106fl0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "STK     SL500           1126"

Vendor ID  : "STK     "

Product ID : "SL500           "

Product Rev: "1126"

Serial Number: "557000102288"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_CHANGER

NetBackup Robot Type: 8

Removable      : Yes

Device Supports: SCSI-5

Number of Drives : 6

Number of Slots  : 143

Number of Media Access Ports: 15

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/0cbn"

Passthru Name: "/dev/sg/c1tw500104f0007f107fl0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  M63S"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "M63S"

Serial Number: "HU10750BWA"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/1cbn"

Passthru Name: "/dev/sg/c1tw500104f0007f1082l0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

i tried updating also my device mappings to the recent but to no avail, still errors.

help!

Inquiry    : "HP      Ultrium 3-SCSI  M63S"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "M63S"

Serial Number: "HU10747E3D"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: 16

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

[Marianne]

The error is not related to NBU upgrade/patch.

This error is coming from the robot, not NBU:

TLD(0) key = 0x2, asc = 0x4, ascq = 0x3, LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED

Start by physically inspecting the robot - check for robot door or CAP not properly closed.
Try to powercycle the robot.

If the above does not help, log a call with your hardware support team.

[Omar_Villa]

Scan command shows the sl8500 maybe the needs an update run: vmoprcmd -devconfig -l and check if path is thr same as: "/dev/sg/c0tw2103001b32653b82l0"

[mph999]

No, no update required.

MODE_SENSE complete

Enter tld commands (? returns help information)

init

Initiating INITIALIZE_ELEMENT_STATUS

initialize_element_status failed

sense key = 0x2, asc = 0x4, ascq = 0x3, LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED

robtest connected we see the mode_sense complete.  Therefore the path is correct.  Providing this happens, evertything else that happens, is outside NetBackup.

We see an ASC// ASCQ error - as explained in  http://www.symantec.com/docs/TECH169477

it can be seen the asc /ascq errors are sent from the hardware.  

Additionally, robtest is not really a NetBackup command.  Like scan, it just send scsi commands to the library.  The only 'interaction' with NBU is that fact that in the config the device file must be correct (so that rtobtest can get to the library).  All output seen in robtest when you run commands (eg. s s (show slots)) is SENT from the library - if it is wrong, then it is a library issue.

As Marianne has explained, this issue is related to a problem with the library.  Any 'investoigation' in NBU is a waste of time.

Martin

[bharatgangawane]

Hi All

I agree with Marianne is this not related to NBU.Kindly check on Harware level.

I just want to add one more thing to check physically.

Check if any Media is stucked in Robotic unit if Yes remove it from Robotic unit and try to do robtest again.

May be it will helpful.

Thanks

Bharat

[knox890]

Thanks guys for your replies!!

After I restart the tape library, the drive is now detected but the robot was not sensed at all.

I even tried to eject tapes but it cannot be ejected.

[Marianne · Accepted solution]

Please log a support call with your hardware vendor.

[bharatgangawane]

Hi all,

This issue will come due to Bad Media are stuck in Robotic arm,Try to eject Bad Medias from library.

May be this erro will not come in Future..

Bharat

[knox890]

hello,

so I contacted the vendor and was able to fix the SL500. However, I cannot seem to fix this one.

root@ATPBACKUPSP01 # /usr/openv/volmgr/bin/vmoprcmd -devconfig -l

Device Robot Drive       Robot                    Drive                 Device                                                                      Second

Type     Num Index  Type DrNum Status  Comment    Name                  Path                                                                        Device Path

robot      0    -    TLD    -       -  -          -                     /dev/sg/c0tw500104f0007f106fl0

  drive    -    0 hcart3    1      UP  -          HP.ULTRIUM3-SCSI.000  /dev/rmt/6cbn                                                             

  drive    -    1 hcart3    2      UP  -          HP.ULTRIUM3-SCSI.001  /dev/rmt/7cbn                                                             

  drive    -    2 hcart3    6      UP  -          HP.ULTRIUM3-SCSI.002  MISSING_PATH:/dev/rmt/3cbn:HU107                                          

  drive    -    3 hcart3    4    DOWN  -          HP.ULTRIUM3-SCSI.003  /dev/rmt/13cbn                                                            

  drive    -    3 hcart3    4    DOWN  -          HP.ULTRIUM3-SCSI.003  /dev/rmt/1cbn                                                             

  drive    -    5 hcart3    3    DOWN  -          HP.ULTRIUM3-SCSI.005  /dev/rmt/12cbn                                                            

  drive    -    5 hcart3    3      UP  -          HP.ULTRIUM3-SCSI.005  /dev/rmt/0cbn                                                             

  drive    -    5 hcart3    3      UP  -          HP.ULTRIUM3-SCSI.005  /dev/rmt/3cbn                                                             

  drive    -   12 hcart3    5      UP  -          HP.ULTRIUM3-SCSI.004  /dev/rmt/2cbn                                                             

robot      1    -    TLD    -       -  -          -                     /dev/sg/c0tw2103001b32653b82l0

  drive    -    4 hcart3    1      UP  -          HP.ULTRIUM3-SCSI.006  /dev/rmt/11cbn                                                            

  drive    -    6 hcart3    2      UP  -          HP.ULTRIUM3-SCSI.007  /dev/rmt/10cbn                                                            

  drive    -    7 hcart3    3      UP  -          HP.ULTRIUM3-SCSI.008  /dev/rmt/9cbn                                                             

  drive    -    8 hcart3    4      UP  -          HP.ULTRIUM3-SCSI.009  /dev/rmt/8cbn                                                             

  drive    -    9 hcart3    5      UP  -          HP.ULTRIUM3-SCSI.010  /dev/rmt/5cbn                                                             

  drive    -   10 hcart3    6      UP  -          HP.ULTRIUM3-SCSI.011  /dev/rmt/4cbn  

How to update the drive manually? I used the Config wizard but still the drive indicates DOWN status.

also, look at the dmesg info,

Apr  6 10:58:15 ATPBACKUPSP01 avrd[25419]: [ID 415999 daemon.notice] Unable to determine sg name for /dev/rmt/13cbn, using open() for scanning

Apr  6 10:58:15 ATPBACKUPSP01 avrd[25419]: [ID 239823 daemon.error] Fatal open error on HP.ULTRIUM3-SCSI.003 (device 3, /dev/rmt/13cbn), errno = 2 (No such file or directory), DOWN'ing it

Apr  6 11:00:30 ATPBACKUPSP01 last message repeated 1 time

Apr  6 11:05:44 ATPBACKUPSP01 bptm[26189]: [ID 498531 daemon.error] user scsi ioctl() failed, may be timeout, errno = 5, I/O error

Apr  6 11:06:16 ATPBACKUPSP01 tldd[26195]: [ID 536676 daemon.error] ioctl error on HP.ULTRIUM3-SCSI.005 (device 5, /devices/pci@1f,700000/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0/sg@w500104f0007f1088,0:raw) thru sg driver, I/O error

Apr  6 11:07:31 ATPBACKUPSP01 last message repeated 5 times

Apr  6 11:07:46 ATPBACKUPSP01 tldd[26195]: [ID 536676 daemon.error] ioctl error on HP.ULTRIUM3-SCSI.005 (device 5, /devices/pci@1f,700000/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0/sg@w500104f0007f1088,0:raw) thru sg driver, I/O error

Apr  6 11:09:31 ATPBACKUPSP01 last message repeated 7 times

Apr  6 11:09:31 ATPBACKUPSP01 avrd[25419]: [ID 239823 daemon.error] Fatal open error on HP.ULTRIUM3-SCSI.003 (device 3, /dev/rmt/13cbn), errno = 2 (No such file or directory), DOWN'ing it

maybe it needs to update some drive but I don't know how to. only been handling NBU for 6 months

thanks!

[knox890]

Thanks!please mark this as closed.