SL8500 - ACS - Drive issue

[Tekkali]

Hi All

Configured SL8500 - ACS - 5 drives in Solaries media server. One drive is Down state. tryied manuvally but it's again down.

Verified messages logs shows below Fatel error..

Jul 29 04:09:49 ssbkupmm016v su: [ID 366847 auth.notice] 'su root' succeeded for d705297 on /dev/pts/3
Jul 29 04:11:56 ssbkupmm016v ltid[21107]: [ID 402974 daemon.notice] Operator/EMM server has UP'ed drive 01000110_lto3 (device 4)
Jul 29 04:12:06 ssbkupmm016v avrd[21473]: [ID 124861 daemon.error] Fatal open error on 01000110_lto3 (device 4, /dev/rmt/4cbn), errno = 2 (No such file or directory), DOWN'ing it

Kindly suggest on this

[RamNagalla · Accepted solution]

it looks like Drive path has been changed or Drive is not visiable in OS level

what is the output of below commands, 

mt -f /dev/rmt/4cbn status

tpautoconf -t

tpconfig -d

you need to compare the outputs and make sure that all paths are visiable in tpconfig -d is also visible in tpautoconf -t

if they are not you need to update the configurations..

[Marianne]

I agree with Nagalla.

Looks like OS has lost connectivity to the tape drive.

Troubleshooting attempts should be at OS level.

Important commands now are:

mt -f /dev/rmt/4cbn status

and

/usr/openv/volmgr/bin/scan

This will show us what the OS sees.

[Tekkali]

Hi

# date
Tue Jul 29 05:38:33 EST 2014
# uname -a
SunOS Meidaserver 5.10 Generic_147440-19 sun4u sparc SUNW,Sun-Fire-V490
#  mt -f /dev/rmt/4cbn  status
/dev/rmt/4cbn: No such file or directory
# tpautoconf -t
TPAC60 HP      Ultrium 3-SCSI  L6HS MXP1004KAN -1 -1 -1 -1 /dev/rmt/3cbn - -
TPAC60 HP      Ultrium 3-SCSI  L6HS HU10715JNR -1 -1 -1 -1 /dev/rmt/1cbn - -
TPAC60 HP      Ultrium 3-SCSI  L6HS HU10708N21 -1 -1 -1 -1 /dev/rmt/2cbn - -
TPAC60 HP      Ultrium 3-SCSI  L6HS MXP0901BAF -1 -1 -1 -1 /dev/rmt/0cbn - -
# tpconfig -d
Id  DriveName           Type   Residence
      Drive Path                                                       Status
****************************************************************************
0   01020105_lto3        hcart3 ACS(2)  ACS=1, LSM=2, PANEL=1, DRIVE=5
      /dev/rmt/0cbn                                                    UP
1   01000105_lto3        hcart3 ACS(2)  ACS=1, LSM=0, PANEL=1, DRIVE=5
      /dev/rmt/1cbn                                                    UP
2   01010110_lto3        hcart3 ACS(2)  ACS=1, LSM=1, PANEL=1, DRIVE=10
      /dev/rmt/2cbn                                                    UP
3   01010105_lto3        hcart3 ACS(2)  ACS=1, LSM=1, PANEL=1, DRIVE=5
      /dev/rmt/3cbn                                                    UP
4   01000110_lto3        hcart3 ACS(2)  ACS=1, LSM=0, PANEL=1, DRIVE=10
      /dev/rmt/4cbn                                                    DOWN
 
Currently defined robotics are:
  ACS(2)     ACSLS server = xxxxx
 
EMM Server = xxxx

FYI

Other outputs provided below..

bash-3.00# tpconfig -l
Device Robot Drive       Robot                    Drive          Device         Second
Type     Num Index  Type DrNum Status  Comment    Name           Path           Device Path
robot      2    -    ACS    -       -  -          -              xxxxxxxx
  drive    -    0 hcart3    -      UP  -          01020105_lto3  /dev/rmt/0cbn  ACS=1, LSM=2, PANEL=1, DRIVE=5
  drive    -    1 hcart3    -      UP  -          01000105_lto3  /dev/rmt/1cbn  ACS=1, LSM=0, PANEL=1, DRIVE=5
  drive    -    2 hcart3    -      UP  -          01010110_lto3  /dev/rmt/2cbn  ACS=1, LSM=1, PANEL=1, DRIVE=10
  drive    -    3 hcart3    -      UP  -          01010105_lto3  /dev/rmt/3cbn  ACS=1, LSM=1, PANEL=1, DRIVE=5
  drive    -    4 hcart3    -    DOWN  -          01000110_lto3  /dev/rmt/4cbn  ACS=1, LSM=0, PANEL=1, DRIVE=10  =======================> Issue drive
bash-3.00#
--------------------------------------------------------------------------------------------------------------------------------
 
bash-3.00# ls -ltr /dev/rmt/*cbn
lrwxrwxrwx   1 root     root          73 Jul 16  2007 /dev/rmt/0cbn -> ../../devices/pci@8,600000/SUNW,emlxs@1/fp@0,0/st@w500104f000904e13,0:cbn
lrwxrwxrwx   1 root     root          73 Jul 16  2007 /dev/rmt/1cbn -> ../../devices/pci@8,600000/SUNW,emlxs@1/fp@0,0/st@w500104f000904e73,0:cbn
lrwxrwxrwx   1 root     root          73 Jul 16  2007 /dev/rmt/2cbn -> ../../devices/pci@8,600000/SUNW,emlxs@1/fp@0,0/st@w500104f000904e34,0:cbn
lrwxrwxrwx   1 root     root          73 Jul 16  2007 /dev/rmt/3cbn -> ../../devices/pci@8,700000/SUNW,emlxs@3/fp@0,0/st@w500104f000904e43,0:cbn
lrwxrwxrwx   1 root     root          73 Jul 16  2007 /dev/rmt/4cbn -> ../../devices/pci@8,700000/SUNW,emlxs@3/fp@0,0/st@w500104f000904e64,0:cbn =======> Issue drive
--------------------------------------------------------------------------------------------------------------------------------
Could not find the one of the tape controller from below output. (500104f000904e64) , Based on the below output we see only 4 controler for tape hence 4 drives are working.
 
bash-3.00# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             scsi-bus     connected    configured   unknown
c0::dsk/c0t0d0                 CD-ROM       connected    configured   unknown
c1                             fc-private   connected    configured   unknown
c1::500000e011f259c1           disk         connected    configured   unknown
c1::500000e011f25b31           disk         connected    configured   unknown
c2                             fc-fabric    connected    configured   unknown
c2::500104f000904e43           tape         connected    configured   unknown
c3                             fc           connected    unconfigured unknown
c4                             fc           connected    unconfigured unknown
c5                             fc-fabric    connected    configured   unknown
c5::500104f000904e13           tape         connected    configured   unknown
c5::500104f000904e34           tape         connected    configured   unknown
c5::500104f000904e73           tape         connected    configured   unknown
usb0/1                         unknown      empty        unconfigured ok
usb0/2                         unknown      empty        unconfigured ok
usb0/3                         unknown      empty        unconfigured ok
usb0/4                         unknown      empty        unconfigured ok
bash-3.00#  
--------------------------------------------------------------------------------------------------------------------------------
 
bash-3.00# luxadm -e port
/devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl                     CONNECTED
/devices/pci@8,700000/SUNW,emlxs@3/fp@0,0:devctl                   CONNECTED   ===========> Drive 3 and 4 are using this port but Drive 4 is only NOT working.
/devices/pci@8,700000/SUNW,emlxs@4/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@8,700000/SUNW,emlxs@5/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@8,600000/SUNW,emlxs@1/fp@0,0:devctl                   CONNECTED
bash-3.00#

[Tekkali]

FYI..

# /usr/openv/volmgr/bin/scan
************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
************************************************************
------------------------------------------------------------
Device Name  : "/dev/rmt/3cbn"
Passthru Name: "/dev/sg/c0tw500104f000904e43l0"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 3-SCSI  L6HS"
Vendor ID  : "HP      "
Product ID : "Ultrium 3-SCSI  "
Product Rev: "L6HS"
Serial Number: "MXP1004KAN"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "/dev/rmt/1cbn"
Passthru Name: "/dev/sg/c0tw500104f000904e73l0"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 3-SCSI  L6HS"
Vendor ID  : "HP      "
Product ID : "Ultrium 3-SCSI  "
Product Rev: "L6HS"
Serial Number: "HU10715JNR"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "/dev/rmt/2cbn"
Passthru Name: "/dev/sg/c0tw500104f000904e34l0"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 3-SCSI  L6HS"
Vendor ID  : "HP      "
Product ID : "Ultrium 3-SCSI  "
Product Rev: "L6HS"
Serial Number: "HU10708N21"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "/dev/rmt/0cbn"
Passthru Name: "/dev/sg/c0tw500104f000904e13l0"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 3-SCSI  L6HS"
Vendor ID  : "HP      "
Product ID : "Ultrium 3-SCSI  "
Product Rev: "L6HS"
Serial Number: "MXP0901BAF"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
#

[Tekkali]

OS Level not avialable. Thats why i'm requesting you to please do the cable trace by coordinating with hardware team so that we can get the switch port info where drive10_lto3 is connected to. any other why to collect switch port info..

what is next step and suggest me...

[mph999]

I'm not quite sure how we remotely trace a drive connection ... You know the WWN, so if it's zoned on the SAN via WWN the details can be found from the SAN. It woud be quite good if this was all contained in the system documentation, so that all you would of had to do is look it up.

[Marianne]

I do not quite understand what you are asking here from a backup software community:

...  i'm requesting you to please do the cable trace by coordinating with hardware team so that we can get the switch port info where drive10_lto3 is connected to. any other why to collect switch port info..

It seems you are aware of the fact that there is a hardware issue and that nothing from NetBackup side can be done to fix it.

Coordinating with hardware/SAN management team and tracing cables is something only you can do.

Good luck!

[Tekkali]

Hey Marianne

Thanks

I am working with SAN Team on this. After investigating we fond it's cabling issue. .

And library end vendor conformed every thing fine. Now I am working with SAN Team for tracing cables...