Not all Netbackup Tape Drives are running except single/one

[Zahid_Haseeb]

Environment

OS = windows 2008 Server

Netbackup Server version = 7.1

Media Multiplexing (Under Policy => Schedule Tab => Schedule) = 1

Allow Multiple data strem (Under Attributes tab) = no (unchecked)

Problem

We have six drives and three are down but remaining three are ok. Although only one is running at a time (Means from the three drives sometimes Drive1 runs and sometimes drive2 and sometimes drive3)

Few results are for reference:

Detail status of Activity Monitor

5/8/2012 2:47:53 PM - awaiting resource netbkup-hcart-robot-tld-0 - No drives are available

Result of tpconfig

C:\Program Files\Veritas\Volmgr\bin>tpconfig.exe -d
Id  DriveName           Type   Residence
      SCSI coordinates/Path                                            Status
****************************************************************************
0   IBM.ULT3580-TD5.000  hcart  TLD(0)  DRIVE=3
      {3,0,2,0}                                                        DOWN
      {4,0,2,0}                                                        DOWN
1   IBM.ULT3580-TD5.001  hcart  TLD(0)  DRIVE=4
      {3,0,3,0}                                                        DOWN
      {4,0,3,0}                                                        UP
2   IBM.ULT3580-TD5.002  hcart  TLD(0)  DRIVE=5
      {3,0,4,0}                                                        UP
      {4,0,4,0}                                                        UP
3   IBM.ULT3580-TD5.003  hcart  TLD(0)  DRIVE=6
      {3,0,5,0}                                                        DOWN
      {4,0,5,0}                                                        DOWN
4   IBM.ULT3580-TD5.004  hcart  TLD(0)  DRIVE=1
      {4,0,6,0}                                                        DOWN
      {3,0,6,0}                                                        DOWN
5   IBM.ULT3580-TD5.005  hcart  TLD(0)  DRIVE=2
      {4,0,7,0}                                                        UP
      {3,0,7,0}                                                        UP

Currently defined robotics are:
  TLD(0)     SCSI coordinates = {3,0,5,1}

EMM Server = netbkup

[mph999 · Accepted solution]

I presume you have chcked that the number of write drives is set >1 on the stu ?

1. Good idea to fix the down drives

2.  When no jobs are running, run from master :

nbrbutil -resetall

Does this resolve the issue ?

Martin

[Zahid_Haseeb]

Thanks mph for your quick reply

max concurrent writes allowed = 6

In 15minutes let me run this command nbrbutil -resetall

[Marianne]

If you have running backups and cannot do 'nbrbutil -resetall', run
nbrbutil -dump and post MDS allocations in the bottom part of the output. (netbackup\bin\admincmd)

Please also post output of vmoprcmd -d. (volmgr\bin)

If we compare output of these 2 commands, we can see if 'orphaned media and/or drive allocations are causing this.

*** EDIT ***

Similar issue caused by orphaned media allocation: https://www-secure.symantec.com/connect/forums/idle-tape-drives-duplication-jobs-queueing

[Zahid_Haseeb]

@ Marianne Find both results

MdsAllocation: allocationKey=151578 jobType=1 mediaKey=4000115 mediaId=0
008L5 driveKey=2000346 driveName=IBM.ULT3580-TD5.005 drivePath={4,0,7,0} stuName
=netbkup-hcart-robot-tld-0 masterServerName=netbkup mediaServerN
ame=netbkup ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fat
PipeKey=0 scsiResType=1 serverStateFlags=1

===================================================================

C:\Program Files\Veritas\Volmgr\bin>vmoprcmd.exe -d

                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart  DOWN-TLD             -                     No       -         0
  0 hcart  DOWN-TLD             -                     No       -         0
  1 hcart  DOWN-TLD             -                     No       -         0
  1 hcart    TLD                -                     No       -         0
  2 hcart    TLD               Yes   2025L4  2025L4   Yes     Yes        0
  2 hcart    TLD               Yes   2025L4  2025L4   Yes     Yes        0
  3 hcart  DOWN-TLD             -                     No       -         0
  3 hcart  DOWN-TLD             -                     No       -         0
  4 hcart  DOWN-TLD             -                     No       -         0
  4 hcart  DOWN-TLD             -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 IBM.ULT3580-TD5.000   No       -
  0 IBM.ULT3580-TD5.000   No       -
  1 IBM.ULT3580-TD5.001   No       -
  1 IBM.ULT3580-TD5.001   No       -
  2 IBM.ULT3580-TD5.002   No       netbkup
  2 IBM.ULT3580-TD5.002   No       netbkup
  3 IBM.ULT3580-TD5.003   No       -
  3 IBM.ULT3580-TD5.003   No       -
  4 IBM.ULT3580-TD5.004   No       -
  4 IBM.ULT3580-TD5.004   No       -
  5 IBM.ULT3580-TD5.005   No       -
  5 IBM.ULT3580-TD5.005   No       -

C:\Program Files\Veritas\Volmgr\bin>

[Marianne]

No orphaned media or drive allocation, but nbrbutil and vmoprcmd output do not match:

nbrbutil:
Drive: IBM.ULT3580-TD5.005  Media Id: 0008L5

vmoprcmd:
Drive: LT3580-TD5.002 Media Id: 2025L4

There is no resource allocation for media id 2025L4 that is currently loaded in drive LT3580-TD5.002 and the drive and media that is allocated in nbrb are not mounted....

Something 'weird going on here and probably causing drives to be DOWN'ed.

Is you device mapping 100% correct? Incorrect device mapping will end up with media mounted in 'wrong' drives, media that cannot be unmounted from drives, etc...
Device mapping go 'wrong' when no persistent binding is in place. This will cause OS device name to change after a reboot, causing NBU device mapping between OS device and drive postion to be incorrect.

As a start, check that Device Mappings are correct. If not, check Persistent Binding in your HBA management utility. Delete drives and re-run NBU device config wizard.

If all of this checks out fine, please post the text in Details tab of queued job(s).

[Zahid_Haseeb]

Ahh

I ran the command and all three drives started taking backup's

[revarooo]

Thanks for letting us know. Can you mark mph999 post which suggested running resetall as the solution.

[Zahid_Haseeb]

Why not sure. I always like to share the resolution, giving vote and Marking the Solution too :)

[mph999]

Excellent ...

When you have symptoms like this, and, if you are able to run the resetall command, it is often worth giving this a go as the (possible ) quickest solution.  Providing no backups are running, if it does not work, then no harm is done.

However.

If the issue keeps happenig, then there is something more 'serious ' wrong, and more details investigation is required.

Regards,

Martin

[mph999]

Well, really the solution should go to the person who gave the correct answer.  There are then  awarded some points for this.  this is the way the forum should work.

If you mark the solution yourself, it kind of doesn't work as no points are awarded (you are not 'allowed' to assign yourself points), and the person who assisted you misses out.

Martin

[Zahid_Haseeb]

emm yes. But Actually What worths me is that the person should be motivated who is giving us the time to think the possible solutions/TIP

[mph999]

I'm not quite following you there ...

I think if somebody gives up their time and expertise to assist someone with an issue in their environment, it is only fair that they receive the marked solution and points.

I see you have 66 solutions against your profile, this is very good, and I am sure you are proud of this achievement.  I am sure you would be a little 'upset' if noboby had marked you correct ansewers, and instead taken the 'solution' themselves, not really very fair.

As you will know, as you build up points you can receive awards - yes I will be honest, I partly do this forum to help people and because I enjoy NetBackup, but, I also partly do it to receive points - I put many many hours of my own time into this, and it is only fair you receive a little token of appreciation.

Martin

[revarooo]

100% agree Martin, you should have received the "Solution" marked against your post.

[Zahid_Haseeb]

First I am not upset.Second I think we have to leave commenting on each other apart from real work. I am sorry if you hurt:)

[Marianne]

Let us give credit where it is due.

Martin has given you the exact command over here: https://www-secure.symantec.com/connect/forums/not-all-netbackup-tape-drives-are-running-except-singleone#comment-7093151

You have typed in exact command that Martin suggested.

Therefore the Solution was rightfully earned by Martin.

I have moved the Solution.

[Zahid_Haseeb]

I have moved the Solution.

Whom I marked the solution ? I marked the solution for mph999/martin. is not it ?

[Marianne]

No. You marked your own post as Solution.

I have cleared your Solution and marked Martin's post.

[Zahid_Haseeb]

Ohh. I apologized. I really allologized.

[Zahid_Haseeb]

One last question:

This command actually resets all nbrb allocations, requests, and persisted states.

This problem actually relates with the Tape Library who's(Tape Library) media may got stuck in the drives or etc ?