Drives are idle while backup jobs are queued

[anasgr]

Hi,

I have storage unit with 5 drives and multiplexing value as 4 on one HP-UX media server.

Storage Unit Type:    Media Manager
Host Connection:      server-name
Number of Drives:     5
On Demand Only:       no
Density:              hcart3 (20)
Robot Type/Number:    TLD (8) / 1
Max Fragment Size:    1048576
Max MPX/drive:        4

While only two drives are being used other jobs are queued with log message "Required drives are in use  Media Server: ionian"

 0 hcart3   TLD               Yes   OBJ197  OBJ197   Yes     Yes        0  
  1 hcart3   TLD               Yes   OBF637  OBF637   Yes     Yes        0  
  2 hcart3   TLD                -                     No       -         0  
  3 hcart3   TLD                -                     No       -         0  
  4 hcart3   TLD                -                     No       -         0 

Any suggestions??

Thanks

[Mark_Solutions]

A little more information please ...

Is it backups or duplications that are queued?

What version of NetBackup is this?

Are the other drives shown as being UP?

If duplications is it disk staging or lifecycle polcies?

If created the output of the admin log from the Master Server or bptm log from the Media Server would be helpful

Thanks

[RiaanBadenhorst]

Hi,

Check if you dont have orphaned MDS allocations. You can use the nbrbutil -dump command to list all the allocations. Push it to a text file and look  at the bottom of the file whether the MDS allocations are listed.

You should see allocation for each active job with the media id and storage unit/media server listed. If you find allocation that are not really active, you can release them with -releaseMDS [ID].

Just make sure there are no active restore or duplication that you've not factored in.

Details in the note

http://www.symantec.com/docs/TECH46807

[Marianne]

Please post the 'MDS allocation' section at the bottom of 'nbrbutil -dump' output.

[Mark_Solutions]

Can you tell us more please ....

Are these backup jobs or duplications that are queued?

How many bpbrm and bptm processes are running on the Media Server involved?

Can you post the full text of the detailed status of the queued job and also the admin log from the Master Server please.

If the backup jobs are for the same client what is you max jobs per client set to on the Master Server Host properties?

Thanks

[Marianne]

Please post output of 'vmdareq' on master.

Has anything changed in your environment recently?

[anasgr]

thanks

[anasgr]

backup jobs are queued, NB version is 6.5.3.1, all drives are UP:-

# vmoprcmd -d                                                                                                                                                                                     

                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart3   TLD               Yes   OBJ197  OBJ197   Yes     Yes        0  
  1 hcart3   TLD               Yes   OBF637  OBF637   Yes     Yes        0  
  2 hcart3   TLD                -                     No       -         0  
  3 hcart3   TLD                -                     No       -         0  
  4 hcart3   TLD                -                     No       -         0  

Thanks

[anasgr]

Earlier when I faced the issue I got the output of the nbrbutil with respect to the unsued drives :-

#nbrbutil  -releaseDrive  sj12-px720-3-20
No allocation for drive sj12-px720-3-20 found
#  nbrbutil  -releaseDrive  sj12-px720-3-19
No allocation for drive sj12-px720-3-19 found
#  nbrbutil  -releaseDrive  sj12-px720-3-17
No allocation for drive sj12-px720-3-17 found

Right now, MDS allocation showed the allocation for the other drives in the same domain ( different media server) :

MdsAllocation allocationKey=118445 jobType=1 mediaKey=4084307 mediaId=OBJ411 driveKey=2000018 driveName=sj12-vtl-7-d3 drivePath=/dev/rmt/c9t4d2BESTnb serverStateFlags=1
        MdsAllocation allocationKey=118491 jobType=1 mediaKey=4080427 mediaId=OBD771 driveKey=2000020 driveName=sj12-vtl-7-d5 drivePath=/dev/rmt/c10t4d0BESTnb serverStateFlags=1
        MdsAllocation allocationKey=118497 jobType=1 mediaKey=4081672 mediaId=OBF776 driveKey=2000019 driveName=sj12-vtl-7-d4 drivePath=/dev/rmt/c9t5d1BESTnb serverStateFlags=1
        MdsAllocation allocationKey=118498 jobType=1 mediaKey=4081904 mediaId=OBE359 driveKey=2000027 driveName=sj12-vtl-7-d12 drivePath=/dev/rmt/c10t5d2BESTnb serverStateFlags=1
        MdsAllocation allocationKey=118500 jobType=1 mediaKey=4078934 mediaId=OBA748 driveKey=2000024 driveName=sj12-vtl-7-d9 drivePath=/dev/rmt/c9t4d3BESTnb serverStateFlags=1

Thanks

[Marianne]

Is this the version on Master and Media Server(s)?

Has this particular media server worked fine before?

We have seen something similar at a particular client some time ago. Even logged a Support call which turned into 2 frustrating weeks of sending large amount of highest level verbose logs....

If we stopped NBU on master and media servers including PBX (in a busy 24x7 environment), all will start out fine after restarting and gradually deteriorate over time where not even half the devices were utilized.

My colleague at the time found lots of enhancements for nbrb in a README for a particular 6.5 patch (cannot remember exact details). All problems disappeared after patching master. Media servers were updated later.

Please consider installing 6.5.6 or even upgrading to 7.x... LOTS of improvements to the resource broker (nbrb).

[Omar_Villa]

What version of NBU do you have? NBRB haves tons of timing bugs go to /usr/openv/logs/ and check the msecs of nbrb: grep -i msec *118*

share the latest piece of code you have, if the environment is large can be nbrb is getting hammer, a quick solution that I dont like is just bouncing the master just to give it a fresh start.

I see you have 2 jobs running or drives in use, what is the number of concurrent drives for that STU? is it 2? that will explain the queue.

Are this shared drives if they are share the vmdareq command output will help to identify who else is using those drives and if you have any SCSI reservations.

please let us know.

regards.

[anasgr]

Everything seemed to be working fine on this version earlier. There was no change as far as the master/media server configuration is concerned.

# vmdareq
-------------------
---------------------
drive-15 - RESERVED on Mon Oct 31 00:16:48 2011
     media-server RESERVED SCAN_HOST UP
drive-16 - RESERVED on Sun Oct 30 23:09:28 2011
     media-server RESERVED SCAN_HOST UP
drive-17 - AVAILABLE
     media-server SCAN_HOST UP
drive-19 - AVAILABLE
     media-server SCAN_HOST UP
drive-20 - AVAILABLE
     media-server SCAN_HOST UP
------------------
----------------

I will consider upgrading the master/media servers.

Thanks for all the suggestions!

-Anand

[Omar_Villa]

share with use your bpstulist -L command, if the drives are not getting assign is because something is holding it and is between NBRB or the STU's limitations, upgrading can help, dont remember you posted your NBU version.

Good luck.