Forum Discussion

lovethatcheese's avatar
lovethatcheese
Level 5
12 years ago

Robot issue - cannot sense robotic device (202)

Hello. Having some issues with my tape drive (sun storagetek sl 48). Master is on Solaris 10, running NB 7.5.0.3. Media server is on same box.

My attached tape drive has been throwing 2009 errors..after running a couple commands earlier, I rebooted my library through the Sun admin gui...ever since doing so, I can no longer connect to this robot through NB - get the error stated above.

 

The tape drive is ready and online. I restarted ltid (usr/openv/volmgr/bin/ltid), and still am having issues. This happened after running tpautoconf -t, tpautoconf -r, and scan -changer. Rebooted library (software).

 

So, what on earth did I do to my tape drive? Why can't NB see it anymore - I've rebooted the library on my other environemnt with no issues.

Per my /var/adm/messages:

olmasterback emlxs: [ID 349649 kern.info] [ 5.04B4]emlxs1: NOTICE: 720: Link up. (4Gb, loop, initiator)
Jan  8 19:15:04 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.0554]emlxs1: NOTICE: 730: Link reset.
Jan  8 19:15:04 nbolmasterback tldcd[3163]: [ID 832037 daemon.error] scsi command failed, may be timeout, scsi_pkt.us_reason = 3
Jan  8 19:15:04 nbolmasterback tldcd[3163]: [ID 415613 daemon.error] TLD(0) mode_sense ioctl() failed: Error 0
Jan  8 19:15:04 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.02D7]emlxs1: NOTICE: 710: Link down.
Jan  8 19:15:07 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.04B4]emlxs1: NOTICE: 720: Link up. (4Gb, loop, initiator)
Jan  8 19:15:09 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.0554]emlxs1: NOTICE: 730: Link reset.
Jan  8 19:15:09 nbolmasterback tldcd[3163]: [ID 832037 daemon.error] scsi command failed, may be timeout, scsi_pkt.us_reason = 3
Jan  8 19:15:09 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.02D7]emlxs1: NOTICE: 710: Link down.
Jan  8 19:15:12 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.04B4]emlxs1: NOTICE: 720: Link up. (4Gb, loop, initiator)
Jan  8 19:15:14 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.0554]emlxs1: NOTICE: 730: Link reset.
Jan  8 19:15:14 nbolmasterback tldcd[3163]: [ID 832037 daemon.error] scsi command failed, may be timeout, scsi_pkt.us_reason = 3
Jan  8 19:15:14 nbolmasterback tldcd[3163]: [ID 415613 daemon.error] TLD(0) mode_sense ioctl() failed: Error 0
Jan  8 19:15:14 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.02D7]emlxs1: NOTICE: 710: Link down.
Jan  8 19:15:15 nbolmasterback tldd[3159]: [ID 320639 daemon.error] TLD(0) unavailable: initialization failed: Unable to sense robotic device
Jan  8 19:15:17 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.04B4]emlxs1: NOTICE: 720: Link up. (4Gb, loop, initiator)
Jan  8 19:17:17 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.0554]emlxs1: NOTICE: 730: Link reset.
Jan  8 19:17:17 nbolmasterback tldcd[3163]: [ID 832037 daemon.error] scsi command failed, may be timeout, scsi_pkt.us_reason = 3
Jan  8 19:17:17 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.02D7]emlxs1: NOTICE: 710: Link down.
Jan  8 19:17:19 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.04B4]emlxs1: NOTICE: 720: Link up. (4Gb, loop, initiator)
Jan  8 19:17:21 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.0554]emlxs1: NOTICE: 730: Link reset.
Jan  8 19:17:21 nbolmasterback tldcd[3163]: [ID 832037 daemon.error] scsi command failed, may be timeout, scsi_pkt.us_reason = 3
Jan  8 19:17:21 nbolmasterback tldcd[3163]: [ID 415613 daemon.error] TLD(0) mode_sense ioctl() failed: Error 0
Jan  8 19:17:21 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.02D7]emlxs1: NOTICE: 710: Link down.
Jan  8 19:17:24 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.04B4]emlxs1: NOTICE: 720: Link up. (4Gb, loop, init

_________________

 

tpautoconf and scan -changer no longer bring anything up, as the robot dissappeared. Per robtest:

ash-3.00# ./robtest
Configured robots with local control supporting test utilities:
  TLD(0)     robotic path = /dev/sg/c0tw500110a0008c72fal1

Robot Selection
---------------
  1)  TLD 0
  2)  none/quit
Enter choice: 1

Robot selected: TLD(0)   robotic path = /dev/sg/c0tw500110a0008c72fal1

Invoking robotic test utility:
/usr/openv/volmgr/bin/tldtest -rn 0 -r /dev/sg/c0tw500110a0008c72fal1

Opening /dev/sg/c0tw500110a0008c72fal1
user command failed, may be timeout, scsi_pkt.us_reason = 3
user command failed, may be timeout, scsi_pkt.us_reason = 3
mode_sense ioctl() failed: Error 0

____

 

anyone, please help?!?!

 

Thanks,

Scott

7.5
Backup and Recovery
NetBackup
  • Marianne's avatar
    Marianne
    12 years ago

    Try these steps:

     

    # cd /usr/openv/volmgr/bin/driver
    # /usr/openv/volmgr/bin/sg.build all 

    Install the new sg driver configuration:

    # /usr/bin/rm -f /kernel/drv/sg.conf
    # /usr/openv/volmgr/bin/driver/sg.install

    Check/verify config:

    # /usr/openv/volmgr/bin/sgscan

     

    (I am aware that Martin's method is different, but this has always worked for me.)

17 Replies

Sort By
  • mph999's avatar
    mph999
    Level 6
    12 years ago

    Hi Scott,

    It is very late here so this will be brief based on first impressions.

    This :

     

    Opening /dev/sg/c0tw500110a0008c72fal1
    user command failed, may be timeout, scsi_pkt.us_reason = 3
    user command failed, may be timeout, scsi_pkt.us_reason = 3
    mode_sense ioctl() failed: Error 0

     
    Shows that that there is an issue connecting to the robot on path /dev/sg/c0tw500110a0008c72fal1, either a connectivity issue, or the robot is not responding:
     
    Is it possible the path has changed somehow ?
     
    This is almost certainly not a NBU issue, though, if the path has changed reconfig on NBU would be required.
     
    Step to troubleshoot.
     
    1.  Prove that the OS can see the library.  
     
    cfadm -al -o show_FCP_dev
     
    Do you see the changer listed ?
     
    (Sometimes cfgadm won'y show anything, if so you'll need to find another command to show the devices that are san connected.  HBA vendors usually provide a utility, luxadm is another solaris command that can be used).
     
    If not, this proves the issue is between the library and the OS.
     
    If you can see the library in this step, then scan should see it and a rebuild of the SG driver will fix (*) 
     
    This should work ...
     
    modunload -i $(echo $(modinfo |grep "sg (SCSA" |awk '{print $1}'))
    mv /kernel/drv/sg.conf /kernel/drv/sg.conf.old
    cd /usr/openv/volmgr/bin/driver
    mv sg.links sg.links.safe
    mv sg.conf sg.conf.safe
    ../sg.build all
    /usr/openv/volmgr/bin/driver/sg.install
     
    (*)  If it still doesn't work, looks like a library issue, it is perfectly possible to get visibility from the OS, but  the library fails to respond to other commands.
     
    Note. When you run scan, providing the sg driver is correct all that it does is send a scsi command to the library.  If this fails, the issue will be between the library and the OS - NBU is not involved (scan does not run any NBU commands).  I've seen san issues cause symptoms like this, faulty HBAs or HBA drivers/ firmware and robot faults.  I've never seen NBU cause this.
     
    A good test is this :
     
    /usr/openv/volmgr/bin/scsi_command -d <path to library>
     
    So
     
    /usr/openv/volmgr/bin/scsi_command -d /dev/sg/c0tw500110a0008c72fal1
     
    This should return the model.
     
    Again, we provide the scsi_command, but this is only sending a scsi_command, absolutley no NBU involved which helps to show where the issue is.
     
    Hope this helps,
     
    Martin
     
     
     
     
     
  • lovethatcheese's avatar
    lovethatcheese
    Level 5
    12 years ago

    Thank you for the response:

    cfadm command:

    fga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    cfga_msg: NULL msgp
    Ap_Id                          Type         Receptacle   Occupant     Condition
    c8                             fc           connected    unconfigured unknown
    c9                             fc-private   connected    configured   unknown
    c9::500110a0008c72fa           med-changer  connected    configured   failed
    c10                            fc           connected    unconfigured unknown
    c11                            fc-private   connected    configured   unknown
    c11::500110a0008c7300          tape         connected    configured   failed

    ______________________

    I see the changer listed, so I will proceed.

    When it comes to this command:

    modunload -i $(echo $(modinfo |grep "sg (SCSA" |awk '{print $1}')) 

    Looks like I got the syntax wrong or something, keeps coming back with :

    usage:  modunload -i <module_id> [-e <exec_file>]

    I'm not very skilled when it comes to coding, so I copied your command verbatim. Was that what you were implying?

     

     

     

  • Yasuhisa_Ishika's avatar
    Yasuhisa_Ishika
    Level 6
    12 years ago

    Did you run cfgadm with "-al -o show_FCP_dev" argments(cfgadm  -al -o show_FCP_dev)? No lun# is displayed in your output, and device condition is displayed as "failed".

    Here is an output of "cfgadm -al -o show_FCP_dev" in my working environment.

    <pre>
    # cfgadm -al -o show_FCP_dev
    Ap_Id                          Type         Receptacle   Occupant     Condition
    c2                             fc-fabric    connected    configured   unknown
    c2::5001a4bcf8d18000,0         med-changer  connected    configured   unknown
    c2::5001a4bcf8d18000,1         tape         connected    configured   unknown
    c2::5001a4bcf8d18000,2         tape         connected    configured   unknown
    c3                             fc           connected    unconfigured unknown
    </pre>

    BTW, somehow link of emlxs1 continue to go up and down. You should check HBA hardware, HBA driver, SAN swithch or so first.

  • Yasuhisa_Ishika's avatar
    Yasuhisa_Ishika
    Level 6
    12 years ago

    modunload -i $(echo $(modinfo |grep "sg (SCSA" |awk '{print $1}')) 

    Looks like I got the syntax wrong or something, keeps coming back with :

    usage:  modunload -i <module_id> [-e <exec_file>]

    Use "modunload -i `modinfo| awk '$6=="sg"{print $1}'`" instead.

  • lovethatcheese's avatar
    lovethatcheese
    Level 5
    12 years ago

    Yasuhisa,

    I did use this command:

    /usr/openv/volmgr/bin
    bash-3.00# cfgadm -al -o show_FCP_dev  

    and I see what you mean with my results:

    Ap_Id                                          Type         Receptacle   Occupant     Condition
    c8                                                fc                connected    unconfigured unknown
    c9                                                fc-private    connected    configured   unknown
    c9::500110a0008c72fa               med-changer  connected    configured   failed
    c10                                              fc               connected    unconfigured unknown
    c11                                              fc-private   connected    configured   unknown
    c11::500110a0008c7300             tape         connected    configured   failed

     

     

    This robot should have two drives. I see both in your example, with LUN included. Mine, not so much. Interesting indeed. I'm on Solaris 10, not sure if that matters?

  • lovethatcheese's avatar
    lovethatcheese
    Level 5
    12 years ago

    Also, as an fyi, your tip of

    Use "modunload -i `modinfo| awk '$6=="sg"{print $1}'`" instead

    did work. I ran that, then proceeded with Martin's suggestion earlier:

    modunload -i $(echo $(modinfo |grep "sg (SCSA" |awk '{print $1}'))
    mv /kernel/drv/sg.conf /kernel/drv/sg.conf.old
    cd /usr/openv/volmgr/bin/driver
    mv sg.links sg.links.safe
    mv sg.conf sg.conf.safe
    ../sg.build all
    /usr/openv/volmgr/bin/driver/sg.install
    _______
     
    first step threw this:
     
    bash-3.00# mv /kernel/drv/sg.conf /kernal/drv/sg.conf.old
    mv: cannot rename /kernel/drv/sg.conf to /kernal/drv/sg.conf.old: No such file or directory
    ________
    bash-3.00# mv sg.links sg.links.safe
    bash-3.00# mv sg.conf sg.conf.safe
    bash-3.00# ../sg.build all
    The file ./st.conf should be appended to /kernel/drv/st.conf.
    A reboot may be necessary to create any new device files.
    Created file ./sg.conf.
    Created file ./sg.links.
    bash-3.00# ./sg.install

            Copied files to /kernel/drv/amd64.

    Doing add_drv of the sg driver

    Leaving existing sg configuration.
    If you wish to update the configuration you need to
            rm -f /kernel/drv/sg.conf
    and rerun /usr/openv/volmgr/bin/driver/sg.install.
     _______________
     
    So, I'm assuming I don't have to re-run the last commands about 'updating the configuration', correct?
    Re-running cfgadm -al -o show_FCP_dev after this step still shows changer and tape in 'failed' condition.
    Is this looking like a hardware issue, aka my robot died? Maybe it's a coincidence a week of 2009 errors / software reboot led to it? It worked, although not at optimal levels, before software reboot.
     
     
     
     
     
     
     
  • lovethatcheese's avatar
    lovethatcheese
    Level 5
    12 years ago

    please ignore my last post, as I didn't spell 'kernel' right.

    anywho, after fixing my mistake:

    bash-3.00# modunload -i `modinfo| awk '$6=="sg"{print $1}'`
    bash-3.00# mv /kernel/drv/sg.conf /kernel/drv/sg.conf.old
    bash-3.00# mv sg.links sg.links.safe
    bash-3.00# mv sg.conf sg.conf.safe
    bash-3.00# ../sg.build all
    The file ./st.conf should be appended to /kernel/drv/st.conf.
    A reboot may be necessary to create any new device files.
    Created file ./sg.conf.
    Created file ./sg.links.
    bash-3.00# ./sg.install

            Copied files to /kernel/drv/amd64.

    Doing add_drv of the sg driver

    devfsadm: driver failed to attach: sg
    Warning: Driver (sg) successfully added to system but failed to attach

    WARNING: /usr/sbin/add_drv failed.
             There may be no SCSI devices on this machine
     

  • Marianne's avatar
    Marianne
    Level 6
    12 years ago

    Forget about NBU re-config until OS status is correct.

    cfgadm output is still showing 'failed':

    c9::500110a0008c72fa               med-changer  connected    configured   failed
    ...
    c11::500110a0008c7300             tape         connected    configured   failed

    As per Yasuhisa's post above - you need to check HBA as we see these entries being repeated in messages file:

    Jan  8 19:15:09 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.02D7]emlxs1: NOTICE: 710: Link down.
    Jan  8 19:15:12 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.04B4]emlxs1: NOTICE: 720: Link up. (4Gb, loop, initiator)
    Jan  8 19:15:14 nbolmasterback emlxs: [ID 349649 kern.info] [ 5.0554]emlxs1: NOTICE: 730: Link reset.

    A server reboot will reset hba, 
    If these messages continue after reboot, log a call with your server support team.

  • mph999's avatar
    mph999
    Level 6
    12 years ago

    c9::500110a0008c72fa           med-changer  connected    configured   failed

    Yes, typically I managed not to cover what to do in the state 'failed', there are many possibilities, as you will understand you can only cover some, hence my description of the most common.

    Failed state looks to me like a library issue - there is a level of communication (you can see it) but it is not responding correctly.

    I would recommend to speak with the hardware vendor and perhaps os support, but hardware vendor first.

    This :

    modunload -i $(echo $(modinfo |grep "sg (SCSA" |awk '{print $1}'))

     
    should work, but for whatever reason is not working for you, so this is an explnation of what it should do.
     

    It uses the modinfo command to get the instance number of the SG driver and ‘pastes’ this into the modunload command, which ‘stops’ the driver.

    For example :

    root@womble netbackup $ modinfo |grep "sg (SCSA"

    189 7b7e0000   37a8 338   1  sg (SCSA Generic Revision: 3.7)

    We only want the instance number (1st field) so the same command with the awk bit gives :

    root@womble netbackup $ modinfo |grep "sg (SCSA" |awk '{print $1}'

    189

    To unload this we could do :

    modunload –i 189

    But is you put a command inside $(   )  then it will use only the output of the command :

    Easiest shown with echo ...

    root@womble netbackup $ echo date

    date

    But use $() and it will not echo date, but will echo the output of date ...

    root@womble netbackup $ echo $(date)

    Fri Jan 4 12:08:03 GMT 2013

    Hence ...

    modunload -i $(echo $(modinfo |grep "sg (SCSA" |awk '{print $1}')

    So to do this manually, this command will give the instance number of the driver

    modinfo |grep "sg (SCSA" |awk '{print $1}'

    This will unload it :

    modunload –i <instance number >

    Martin

     

  • lovethatcheese's avatar
    lovethatcheese
    Level 5
    12 years ago

    Hello. Been working on Marianne's suggestion to resolve 'condition' failed status - rebooted the server and power cycled the tape drive. No longer says 'condition failed':

        Ap_Id                                  Type         Receptacle   Occupant     Condition
    c8                                              fc           connected    unconfigured unknown
    c9                                             fc-private   connected    configured   unknown
    c9::500110a0008c72fa,0         tape         connected    configured   unknown
    c9::500110a0008c72fa,1         med-changer  connected    configured   unknown
    c10                                           fc           connected    unconfigured unknown
    c11                                           fc-private   connected    configured   unknown
    c11::500110a0008c7300,0        tape         connected    configured   unknown

    However, in /var/adm/messages, I am getting this now:

    Jan  9 15:03:12 nbolmasterback vmd[717]: [ID 617826 daemon.notice] ready for connections
    Jan  9 15:03:22 nbolmasterback avrd[741]: [ID 748556 daemon.notice] st.conf configuration for HP.ULTRIUM4-SCSI.000 (device 0), name [HP Ultrium LTO 4], vid [HP      Ultrium 4*], type 0x3b, block size 0, options 0x18619 (see st(7D) man page)Jan  9 15:03:45 nbolmasterback mac: [ID 736570 kern.info] NOTICE: e1000g1 unregistered
    Jan  9 15:03:45 nbolmasterback mac: [ID 736570 kern.info] NOTICE: e1000g2 unregistered
    Jan  9 15:03:45 nbolmasterback mac: [ID 736570 kern.info] NOTICE: e1000g3 unregistered
    Jan  9 15:03:58 nbolmasterback pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
    Jan  9 15:03:58 nbolmasterback genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
    Jan  9 15:04:41 nbolmasterback pseudo: [ID 129642 kern.info] pseudo-device: fcsm0
    Jan  9 15:04:41 nbolmasterback genunix: [ID 936769 kern.info] fcsm0 is /pseudo/fcsm@0
    Jan  9 15:04:50 nbolmasterback scsi: [ID 799468 kern.info] st1 at fp5: name w500110a0008c7300,0, bus address 2
    Jan  9 15:04:50 nbolmasterback genunix: [ID 936769 kern.info] st1 is /pci@3,0/pci1022,7458@a/pci10df,fc12@1,1/fp@0,0/tape@w500110a0008c7300,0
    Jan  9 15:04:50 nbolmasterback scsi: [ID 365881 kern.info] /pci@3,0/pci1022,7458@a/pci10df,fc12@1,1/fp@0,0/tape@w500110a0008c7300,0 (st1):
    Jan  9 15:04:50 nbolmasterback  <HP Ultrium LTO 4>
    Jan  9 15:04:50 nbolmasterback avrd[741]: [ID 990698 daemon.notice] st.conf configuration for HP.ULTRIUM4-SCSI.001 (device 1), name [HP Ultrium LTO 4], vid [HP      Ultrium 4*], type 0x3b, block size 0, options 0x18619 (see st(7D) man page)Jan  9 15:04:54 nbolmasterback tldcd[748]: [ID 295976 daemon.error] TLD(0) [748] robotic path /dev/sg/c0tw500110a0008c72fal1 does not exist
    Jan  9 15:04:54 nbolmasterback last message repeated 1 time
    Jan  9 15:04:54 nbolmasterback tldd[727]: [ID 795118 daemon.error] TLD(0) unavailable: initialization failed: Unable to open robotic path
    Jan  9 15:06:56 nbolmasterback tldcd[748]: [ID 295976 daemon.error] TLD(0) [748] robotic path /dev/sg/c0tw500110a0008c72fal1 does not exist
    Jan  9 15:06:56 nbolmasterback last message repeated 1 time
    Jan  9 15:06:56 nbolmasterback tldd[727]: [ID 795118 daemon.error] TLD(0) unavailable: initialization failed: Unable to open robotic path
    Jan  9 15:08:58 nbolmasterback tldcd[748]: [ID 295976 daemon.error] TLD(0) [748] robotic path /dev/sg/c0tw500110a0008c72fal1 does not exist
    Jan  9 15:27:19 nbolmasterback sg: [ID 266374 kern.notice] Symantec SCSA Generic Revision: 3.7
    ____________

    I see the drive listed here, however, saying the robotic path doesn't exist is interesting. Again, I only started looking at the tape drive after a week of 2009 errors. After a software reboot, I am having these issues. How to reconfigure the path? Even though nothing changed, is this simply refreshing / reseting this?