Scan Command

[H_Sharma]

Hi Experts,

We have one master server windows 2008 and 8 drives.

8 drives are visible on OS level and working fine.

We are experincing issue with the Drives 1 or 2 drives are taking too much time to mount tapes. Its an intermittent issue. Sometimes all 8 are working perfectly fine. We observed its not only a particular drive but if all 7 drives are working 1 of them is not mounting tape due to which backups are in queue and after half an hour it mounts the tape and all drives start working.

In scan -changer command on master server it shows all the 8 drives with serial number.

However scan command also shows all 8 drives with serial number and on specific each drives there are only 5 drives. Is there any issue?

or if scan -changer shows all the 8 drives is it ok? if its ok then why scan command is showing 5 drives( Indivudual details of the drives ) after showing all 8 drives ? 

[Marianne]

scan -changer will show what the robot sees via the internal control path.

This is what the robot uses to know where to mount and dismounts tapes. 
Again this is control path only. No data can flow through this path.

scan -tape is checking the data path at OS level. This is the path that is used for data flow. 
This is the path that needs to acknowledge the tape mount and tell NBU that the tape is mounted, positioned and ready to receive data.

If the robot mounts the tape in a drive, but the data path is missing, the OS cannot see and acknowledge the tape mount and cannot transfer any data.

If connection between OS and the data path is lost, you need to troubleshoot at OS and SAN level.

I would expect to see errors in the OS System logs - Event Viewer System log on Windows and /var/adm/messages on Solaris.

[mph999]

So, there are x8 drives in total, and scan -changer shows all 8 drives in the chager - ok good,.

Do  you mean, when you run scan, you see the details for x8 drives, but it you run it again, you only see x5 drives ?

If so, I would suggest you have an intermittant connectivity issue to the drives.  Look in system messages file, do you see the drives disappearing then reappearing ?

[mph999]

... as always, Marianne answerd whilst I was typing my answer out - however, it seems that we are saying more or less the same thing ...

[Marianne]

Issue with drives have been going on for some time now:

 Drive Issue 

Cluster Failover 

You need to check drives on each server separately.

It is quite possible that one media server can lose connectivity to a drive (showing DOWN on this media server) but working perfectly fine on another media server.

Here you need to check each component in the connection path on the problematic media server (or master node): HBA, HBA driver, cable, gbic on both sides, switch port, Switch config (zoning), persistent binding between OS and HBA, etc.. etc...

[H_Sharma]

Hi Martin,

8 no. of drives are always same those are shown in bold and italic below....... This is the same output every time.

But Device name say Tape 7, Tape 1 are sometimes missing. These keep chaning every time....

 Sometimes it shows 5 Device name sometimes 6 etc not all the 8 ... this is the concern over here...

I believe Scan command should show 8 device names as well say tape 1 , tape 2 etc... Is it alarming ?

************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
************************************************************
------------------------------------------------------------
Device Name  : "Tape7"
Passthru Name: "Tape7"
Volume Header: ""
Port: 1; Bus: 0; Target: 1; LUN: 0
Inquiry    : ""
Vendor ID  : ""
Product ID : " "
Product Rev: ""
Serial Number: ""
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : ""
Passthru Name: ""
Volume Header: ""
Port: 1; Bus: 0; Target: 1; LUN: 1
Inquiry    : ""
Vendor ID  : "  "
Product ID : "     "
Product Rev: "0225"
Serial Number: ""
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_CHANGER
NetBackup Robot Type: 8
Removable      : Yes
Device Supports: SCSI-5
Number of Drives : 8
Number of Slots  : 110
Number of Media Access Ports: 4
Drive 1 Serial Number      : "1"
Drive 2 Serial Number      : "2"
Drive 3 Serial Number      : "3"
Drive 4 Serial Number      : "4"
Drive 5 Serial Number      : "5"
Drive 6 Serial Number      : "6"
Drive 7 Serial Number      : "7"
Drive 8 Serial Number      : "8"
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape1"
Passthru Name: "Tape1"
Volume Header: ""
Port: 1; Bus: 0; Target: 2; LUN: 0
Inquiry    : ""
Vendor ID  : ""
Product ID : ""
Product Rev: ""
Serial Number: ""
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape0"
Passthru Name: "Tape0"
Volume Header: ""
Port: 2; Bus: 0; Target: 1; LUN: 0
Inquiry    : ""
Vendor ID  : " "
Product ID : " "
Product Rev: ""
Serial Number: ""
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape3"
Passthru Name: "Tape3"
Volume Header: ""
Port: 3; Bus: 0; Target: 0; LUN: 0
Inquiry    : ""
Vendor ID  : "     "
Product ID : "  "
Product Rev: ""
Serial Number: ""
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape6"
Passthru Name: "Tape6"
Volume Header: ""
Port: 4; Bus: 0; Target: 1; LUN: 0
Inquiry    : ""
Vendor ID  : "HP      "
Product ID : "  "
Product Rev: ""
Serial Number: ""
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0

[Marianne]

This particular server only sees 4 tape drives. As per previous post, you need to troubleshoot at OS and SAN level to see why the other 4 drives cannot be seen.

[RonCaplinger]

In your other post that Marianne referenced below, you stated:

     I did the below troubleshooting.

     1:- SCAN can see all the drives on master and 2 media servers.

But here in this thread, you are saying SCAN *doesn't* see all 8 drives on the master and media servers. 

If the output here is correct, and you keep seeing drives diasappear and then reappear in the SCAN output, then I would guess that you have multiple paths to the tape drives through your SAN fabric, and either:

• one or more SANs does not have all drives and the robot zoned to all three of the NBU servers,
• or you might have a cabling or interference issue somewhere between the SANs and the media servers, or the SANs and the tape drives.

Can you have your SAN admin check for any errors in their logs?  Can they confirm that the drives are correctly zoned to the master and media servers?

[RonCaplinger · Accepted solution]

^^^ What she said!!!!

And when you look at which drives are seen by each NBU server, COMPARE THE SERIAL #'S & WWNs, not the "Tape0", etc.  One media server may call a drive Tape0, and the other called it Tape5 when it was added.  Ignore those names.

This will help to track down which drives are the ones causing problems.

Once you can see all the drives from all the servers, make sure you have persistent binding enabled on your HBAs (done through some utilitiy for your HBA's, not NetBackup) so that the paths between the tape drives and the ports on your tape drives will stay the same after a server reboot.  This has caused all kinds of problems for us in the past, with tape drives disappearing from one media server but working fine on the others.

[H_Sharma]

Thanks RON,

We have total 3 hosts per drive configured 1 master and 2 media server.

So what do you see is there any problem? Because I guess we must see all 8 drives in the scan output.

and now these are apperianing and disappering every time....

[Marianne]

So what do you see is there any problem?

YES! There is a problem.

If all 8 drives have been zoned to master and all media servers and configured in NBU, then YES, there is a problem.

The media server where you have collected output has lost connectivity to 4 tape drives.

As per previous advice, you need to troubleshoot at OS and SAN level.
Please go through all attempts to help you in previous posts.... 

[H_Sharma]

Hi Marianne,

I checked the scan output on master it shows drives coming up and disappering. I checked the scan output on media servers in one of the media server it shows all 8 and in one 4 and disappering.

Ok this is what i had seen.

Interestingly. 4 days ago we had got our drives firmware upgraded. In which we had to gracecfully restart the netbackup services on master and we did power cycle of our library.

This drive issue is disappered and all the 8 drives are working perfectly fine.

However we have scan output that is still same after the firmware updgrade and power cycle and netbackup up/down.

Let me tell you we have taken the output of the Scan command in .txt file as suggest by the TSE.

Master:- Scan still shows 4-5 drives and disappering.

Media 1:- Scan still shows all the 8 drives.

Media 2:- Scan still shows 4-5 drives and rest disappering.

One more interesting point here. Which i missed earlier. We have a cluster server in which there is no such issue of drive and all the drives were running and scan output is correct for all.

So we can rule out the possibility of firmware upgrade.Its nothing to do with firmware.

So I firmly believe that refresing the netbackup services had resolved the issue but again interesting thing is still SCAN output is not correct on the problemtic server however drives started working perfectly fine

Pls share whats your stake on this one :) 

[mph999]

NBU doesn't use scan as such for job operations, so it is possible that if the drives are configured, scan can give incorrect output, but jobs still run ok.  I've seen this once on Solaris in the past.

scan also works independantly of NBU, let me show you :

Here I have stopped NBU on my master server:

bpps -x shows no processes.

NB Processes
------------

MM Processes
------------

Shared Symantec Processes
-------------------------
    root 14964     1   1   Mar 05 ?           4:40 /opt/VRTSpbx/bin/pbx_exchange

root@womble $ scan

However, scan still works.

************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
************************************************************
------------------------------------------------------------
Device Name  : "/dev/rmt/0cbn"
Passthru Name: "/dev/sg/c1t3l0"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 1-SCSI  E38W"
Vendor ID  : "HP      "
Product ID : "Ultrium 1-SCSI  "
Product Rev: "E38W"
Serial Number: "HU74D01591"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "HP      Ultrium 1-SCSI  HU74D01591"
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
root@womble DataCollect $

So, providing a job doesn't need the drive to do what ever scan makes it do, it'll work.  If you deleted the drives however, and tried to use the wizard to readd them, you'll probably have an issue,as the device wizard does something similiar  to scan -all

[Marianne]

We have a cluster server in which there is no such issue of drive and all the drives were running and scan output is correct for all.

I agree - issue is most likely not with drive firmware.

If you look at my post over here  you will see that I have suggested other possibilities - 

HBA, HBA driver, cable, gbic on both sides, switch port, Switch config (zoning), persistent binding between OS and HBA, etc.. etc...

Have you checked hardware config on both master server nodes?
Are the HBAs the same make/model? 
Are the same drivers and firmware used for HBAs?
Are disk and tape zoned to different HBAs?

Have you checked that correct HBA settings are used for tape?
Old TN for Emulex as an example: http://www.symantec.com/docs/TECH22464 

Same OS patches and hotfixes?
(I have in the past seen that outdated Microsoft Storport drivers caused similar issues.)

Have you confirmed that all devices are zoned to both nodes and media servers?

Have you checked Windows Event Viewer logs for errors when the devices are losing connectivity?