10-25-2011 03:06 AM
Hi,
I have storage unit with 5 drives and multiplexing value as 4 on one HP-UX media server.
Storage Unit Type: Media Manager
Host Connection: server-name
Number of Drives: 5
On Demand Only: no
Density: hcart3 (20)
Robot Type/Number: TLD (8) / 1
Max Fragment Size: 1048576
Max MPX/drive: 4
While only two drives are being used other jobs are queued with log message "Required drives are in use Media Server: ionian"
0 hcart3 TLD Yes OBJ197 OBJ197 Yes Yes 0
1 hcart3 TLD Yes OBF637 OBF637 Yes Yes 0
2 hcart3 TLD - No - 0
3 hcart3 TLD - No - 0
4 hcart3 TLD - No - 0
Any suggestions??
Thanks
10-25-2011 03:20 AM
A little more information please ...
Is it backups or duplications that are queued?
What version of NetBackup is this?
Are the other drives shown as being UP?
If duplications is it disk staging or lifecycle polcies?
If created the output of the admin log from the Master Server or bptm log from the Media Server would be helpful
Thanks
10-25-2011 03:26 AM
Hi,
Check if you dont have orphaned MDS allocations. You can use the nbrbutil -dump command to list all the allocations. Push it to a text file and look at the bottom of the file whether the MDS allocations are listed.
You should see allocation for each active job with the media id and storage unit/media server listed. If you find allocation that are not really active, you can release them with -releaseMDS [ID].
Just make sure there are no active restore or duplication that you've not factored in.
Details in the note
http://www.symantec.com/docs/TECH46807
10-25-2011 04:27 AM
Please post the 'MDS allocation' section at the bottom of 'nbrbutil -dump' output.
10-25-2011 09:21 AM
Can you tell us more please ....
Are these backup jobs or duplications that are queued?
How many bpbrm and bptm processes are running on the Media Server involved?
Can you post the full text of the detailed status of the queued job and also the admin log from the Master Server please.
If the backup jobs are for the same client what is you max jobs per client set to on the Master Server Host properties?
Thanks
10-25-2011 10:28 AM
Please post output of 'vmdareq' on master.
Has anything changed in your environment recently?
10-25-2011 09:56 PM
thanks
10-25-2011 09:57 PM
backup jobs are queued, NB version is 6.5.3.1, all drives are UP:-
# vmoprcmd -d
PENDING REQUESTS
<NONE>
DRIVE STATUS
Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId
0 hcart3 TLD Yes OBJ197 OBJ197 Yes Yes 0
1 hcart3 TLD Yes OBF637 OBF637 Yes Yes 0
2 hcart3 TLD - No - 0
3 hcart3 TLD - No - 0
4 hcart3 TLD - No - 0
Thanks
10-25-2011 09:59 PM
Earlier when I faced the issue I got the output of the nbrbutil with respect to the unsued drives :-
#nbrbutil -releaseDrive sj12-px720-3-20
No allocation for drive sj12-px720-3-20 found
# nbrbutil -releaseDrive sj12-px720-3-19
No allocation for drive sj12-px720-3-19 found
# nbrbutil -releaseDrive sj12-px720-3-17
No allocation for drive sj12-px720-3-17 found
Right now, MDS allocation showed the allocation for the other drives in the same domain ( different media server) :
MdsAllocation allocationKey=118445 jobType=1 mediaKey=4084307 mediaId=OBJ411 driveKey=2000018 driveName=sj12-vtl-7-d3 drivePath=/dev/rmt/c9t4d2BESTnb serverStateFlags=1
MdsAllocation allocationKey=118491 jobType=1 mediaKey=4080427 mediaId=OBD771 driveKey=2000020 driveName=sj12-vtl-7-d5 drivePath=/dev/rmt/c10t4d0BESTnb serverStateFlags=1
MdsAllocation allocationKey=118497 jobType=1 mediaKey=4081672 mediaId=OBF776 driveKey=2000019 driveName=sj12-vtl-7-d4 drivePath=/dev/rmt/c9t5d1BESTnb serverStateFlags=1
MdsAllocation allocationKey=118498 jobType=1 mediaKey=4081904 mediaId=OBE359 driveKey=2000027 driveName=sj12-vtl-7-d12 drivePath=/dev/rmt/c10t5d2BESTnb serverStateFlags=1
MdsAllocation allocationKey=118500 jobType=1 mediaKey=4078934 mediaId=OBA748 driveKey=2000024 driveName=sj12-vtl-7-d9 drivePath=/dev/rmt/c9t4d3BESTnb serverStateFlags=1
Thanks
10-27-2011 12:21 AM
Is this the version on Master and Media Server(s)?
Has this particular media server worked fine before?
We have seen something similar at a particular client some time ago. Even logged a Support call which turned into 2 frustrating weeks of sending large amount of highest level verbose logs....
If we stopped NBU on master and media servers including PBX (in a busy 24x7 environment), all will start out fine after restarting and gradually deteriorate over time where not even half the devices were utilized.
My colleague at the time found lots of enhancements for nbrb in a README for a particular 6.5 patch (cannot remember exact details). All problems disappeared after patching master. Media servers were updated later.
Please consider installing 6.5.6 or even upgrading to 7.x... LOTS of improvements to the resource broker (nbrb).
10-27-2011 04:08 AM
What version of NBU do you have? NBRB haves tons of timing bugs go to /usr/openv/logs/ and check the msecs of nbrb: grep -i msec *118*
share the latest piece of code you have, if the environment is large can be nbrb is getting hammer, a quick solution that I dont like is just bouncing the master just to give it a fresh start.
I see you have 2 jobs running or drives in use, what is the number of concurrent drives for that STU? is it 2? that will explain the queue.
Are this shared drives if they are share the vmdareq command output will help to identify who else is using those drives and if you have any SCSI reservations.
please let us know.
regards.
10-31-2011 12:46 AM
Everything seemed to be working fine on this version earlier. There was no change as far as the master/media server configuration is concerned.
# vmdareq
-------------------
---------------------
drive-15 - RESERVED on Mon Oct 31 00:16:48 2011
media-server RESERVED SCAN_HOST UP
drive-16 - RESERVED on Sun Oct 30 23:09:28 2011
media-server RESERVED SCAN_HOST UP
drive-17 - AVAILABLE
media-server SCAN_HOST UP
drive-19 - AVAILABLE
media-server SCAN_HOST UP
drive-20 - AVAILABLE
media-server SCAN_HOST UP
------------------
----------------
I will consider upgrading the master/media servers.
Thanks for all the suggestions!
-Anand
10-31-2011 02:44 AM
share with use your bpstulist -L command, if the drives are not getting assign is because something is holding it and is between NBRB or the STU's limitations, upgrading can help, dont remember you posted your NBU version.
Good luck.