problem with qlogic (dual port ) HBA card

[Albatross_]

Hi everybody,

we were running netbackup 6.5 on RHEL 4.0 from long back and has never faced any issues.. Recently I am stuck up with the hardware part.

we have qlogic dual port hba card on the server which was connected to HP MSL G3 Series robotic library. It was working fine untill couple of weeks back.

All of sudden netbackup is unable to find the devices and says missing path error for the drive.

So we tried to replace the HBA with  other HBA ( other vendor with single port ) and it worked fine we ran the backup successfully, So concluded that there should be problem with qlogic card and replaced with new qlogic card but still the problem persists..

I could not understand this peculiar behaviour of the hba card / OS / Netbackup application

find the o/p of vmoprcmd

Drive Name               Label   Ready  RecMID  ExtMID  Wr.Enbl.  Type
    Host                       DrivePath                            Status
=============================================================================
HP.ULTRIUM4-SCSI.000     No      No                     No        dlt3
    computenode4               MISSING_PATH:4:0:0:0:HU18504EHD      DOWN

HP.ULTRIUM4-SCSI.001     No      No                     No        dlt3
    computenode4               MISSING_PATH:3:0:0:0:HU18514FNG      DOWN

No signs of library device in the dev folder.... ( /dev/snt0 or /dev/nst1 )

I am really stucked up ...

Any help would be really apreciated...

Cheers

Albatross

[mph999]

I'm struggling to understand it as well - odd.

Firstly, it's not NBU - if scan doesn't 'see' the devices, neither does the OS, check by looking in what is found in /proc/scsi/scsi file.

At this time, qlogic is not working but 'an.other' card is, which nicely narrows down the issue.

I'd install the qlogic software, sansurfer - this look straight down the san as displays what devices the card can see - it removes any doubt as to if the card is having an issue - if it doesn't see the devices there is an issue with the card/ san , if it does there is an ssue between the card and the OS.

[Albatross_]

Hi mph999,

o/p of cat /proc/scsi/scsi

Attached devices:
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: LSILOGIC Model: Logical Volume   Rev: 3000
  Type:   Direct-Access                    ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: HP       Model: Ultrium 4-SCSI   Rev: H44W
  Type:   Sequential-Access                ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 01
  Vendor: HP       Model: MSL G3 Series    Rev: F.30
  Type:   Medium Changer                   ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: HP       Model: Ultrium 4-SCSI   Rev: H44W
  Type:   Sequential-Access                ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: HP       Model: Ultrium 4-SCSI   Rev: H44W
  Type:   Sequential-Access                ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 01
  Vendor: HP       Model: MSL G3 Series    Rev: F.30
  Type:   Medium Changer                   ANSI SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 00
  Vendor: HP       Model: Ultrium 4-SCSI   Rev: H44W
  Type:   Sequential-Access                ANSI SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
  Vendor: AMI      Model: Virtual CDROM    Rev: 1.00
  Type:   CD-ROM                           ANSI SCSI revision: 02
Host: scsi6 Channel: 00 Id: 00 Lun: 00
  Vendor: AMI      Model: Virtual Floppy   Rev: 1.00
  Type:   Direct-Access                    ANSI SCSI revision: 02

o/p of lspci

lspci -v| grep -i ql

02:01.0 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to PCI-X HBA (rev 02)
    Subsystem: QLogic Corp.: Unknown device 0141
02:01.1 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to PCI-X HBA (rev 02)
    Subsystem: QLogic Corp.: Unknown device 0141
06:01.0 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to PCI-X HBA (rev 02)
    Subsystem: QLogic Corp.: Unknown device 0141
06:01.1 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to PCI-X HBA (rev 02)
    Subsystem: QLogic Corp.: Unknown device 0141

can we make anything of above outputs

--

Cheers

Albatross

[mph999]

OK, so /proc/scsi/scsi shows some devices - are these the 'missing' devices - I'll presume they are.

But, just noticed something : 

vmoprcmd shows a couple of drives (ok missing but still, shows what should be there)

/proc/scsi/scsi shows x4 drives and x2 changers - how many should there be ?

Are you using multipathing, if so disable it at the OS level - seee if things improve.  If you are not using multip pathing, why are there twice as many devices as it appears there should be ?

Do you have /dev/nst files for the drives, you mentioned they are missing for the robot, but didn;t mention the drives.

Do you have mt on the system - can this talk to the devices :

mt -f <device file> status 

Need to get the OS seeing the devices, with the devices responding to commands.  IF no mt commnd (think it has to be installed as a package) use /usr/openv/volmgr/scsi_command.

/usr/openv/volmgr/bin/scsi_command -d <device file>

This should retun the device inquiry string, it doesn't use NBU and so is a good enough tool as using mt command.

I'm not sure if the unknown device in lspci output is relevant - haven;t got a machine to hand to check - perhaps a llinux forum could answer this in the meantime (or ask qlogic).

Other thing to consider is force the recreation of the device files - just Google for the commands , see if that makes a difference.

[Marianne]

I am also curious to see output of /usr/openv/volmgr/bin/scan.

If 'scan' sees the devices it means that device paths have changed. Simply delete existing devices in NBU and re-run device config wizard.

PS: 
As a matter of interest - the library uses a /dev/sg path name.
Only tape drives use /dev/nst path names.

See NBU Device Config Guide: http://www.symantec.com/docs/TECH52800
 

[Albatross_]

Hi

we have 2 drives and one changer.

No multipathing is configured

There are no /dev/nst files in the devices list in OS level, drive shows down and missing path in netbackup level.

I have mt on system but cannot talk to the devices.

Sooner I will post the o/p of /usr/openv/volmgr/bin/scsi_command as of now the system is down as there is some hardware check going on.

Cheers

Albatross

[Marianne]

The above output indicates that 2 paths to the devices have been configured. Probably both hba's zoned to all devices?

While hardware checks are happening... PLEASE consider upgrade to newer, supported OS and NBU version.

If this server is still 32-bit architechture, please use current issues and OS/NBU support issues as motivation to migrate to 64-bit hardware and newer OS which will enable NBU upgrade to supported version.

[Albatross_]

Hi Marianne,

Find the o/p of scan command

[root@computenode4 bin]# scan
************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
*********************** SDT_OPTICAL ************************
************************************************************
------------------------------------------------------------
Device Name  : "/dev/sg5"
Passthru Name: "/dev/sg5"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      MSL G3 Series   F.30"
Vendor ID  : "HP      "
Product ID : "MSL G3 Series   "
Product Rev: "F.30"
Serial Number: "0906BR0034"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "HP      MSL G3 Series   0906BR0034"
Device Type    : SDT_CHANGER
NetBackup Robot Type: 8
Removable      : Yes
Device Supports: SCSI-5
Number of Drives : 2
Number of Slots  : 48
Number of Media Access Ports: 0
Drive 1 Serial Number      : "HU18504EHD"
Drive 2 Serial Number      : "HU18514FNG"
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "-"
Passthru Name: "/dev/sg6"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 4-SCSI  H44W"
Vendor ID  : "HP      "
Product ID : "Ultrium 4-SCSI  "
Product Rev: "H44W"
Serial Number: "HU18504EHD"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-5
Flags : 0x0
Reason: 0x10000
------------------------------------------------------------
Device Name  : "-"
Passthru Name: "/dev/sg4"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 4-SCSI  H44W"
Vendor ID  : "HP      "
Product ID : "Ultrium 4-SCSI  "
Product Rev: "H44W"
Serial Number: "HU18514FNG"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-5
Flags : 0x0
Reason: 0x10000
------------------------------------------------------------
Device Name  : "-"
Passthru Name: "/dev/sg3"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 4-SCSI  H44W"
Vendor ID  : "HP      "
Product ID : "Ultrium 4-SCSI  "
Product Rev: "H44W"
Serial Number: "HU18504EHD"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-5
Flags : 0x0
Reason: 0x10000
------------------------------------------------------------
Device Name  : "/dev/sg2"
Passthru Name: "/dev/sg2"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      MSL G3 Series   F.30"
Vendor ID  : "HP      "
Product ID : "MSL G3 Series   "
Product Rev: "F.30"
Serial Number: "0906BR0034"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "HP      MSL G3 Series   0906BR0034"
Device Type    : SDT_CHANGER
NetBackup Robot Type: 8
Removable      : Yes
Device Supports: SCSI-5
Number of Drives : 2
Number of Slots  : 48
Number of Media Access Ports: 0
Drive 1 Serial Number      : "HU18504EHD"
Drive 2 Serial Number      : "HU18514FNG"
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "-"
Passthru Name: "/dev/sg1"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 4-SCSI  H44W"
Vendor ID  : "HP      "
Product ID : "Ultrium 4-SCSI  "
Product Rev: "H44W"
Serial Number: "HU18514FNG"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-5
Flags : 0x0
Reason: 0x10000

I believe the OS itself is not detecting the paths.. How come the NB detects them ??

I tried to delete the old config and tried to configure the device config in NB but it shows as in attachment.

I dont even get any output with mt command

Thanks

Albatross

[mph999 · Accepted solution]

Device name is missing for some of the devices.

Eg.

Device Name  : "-"
Passthru Name: "/dev/sg1"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 4-SCSI  H44W"
Vendor ID  : "HP      "
Product ID : "Ultrium 4-SCSI  "

There is a /dev/sg device which is why scan shows the device, it communicates using /dev/sg.

Note, NetBackup is not seeing these devices, scan is, which is not NetBackup.

What you should see above is a /dev/nstxx device file.

[Marianne]

Please work through the Device Config Guide - link in my previous post.

You will see info such as:

For tape drive device files, NetBackup uses only the /dev/nstx files (n indicates the no rewind device file). The Linux driver should create the device files automatically. If the device files do not exist, see the Linux documentation for information about how to create them.

[Albatross_]

Hi Everybody,

The problem got resolved....I am able to configure devices in netbackup

Solution: loaded st drivers into the kernel manually and booooom the devices got detected.

Thanks for the replies...

Cheers

Albatross