06-16-2013 01:00 PM
Hi Everyone,
This was an issue which i have come acrossed while i was doing the regular health check.
There were a a few jobs which were in Queue for around 14 hrs and are not being active inspite of the drives being available.
So, i had checked the library console and found a few RAS tickets which i had resolved and everything seems to be fine from the Hardware level.
Still i am observing that the Queued jobs are still in Q and not being active.
Could someone please help me out in this.
Thanks in advance
06-16-2013 01:47 PM
What is the exact reason for jobs being queued in Job Details?
Please also show us output from 'vmoprcmd -d' as well as the MDS Allocation part of 'nbrbutil -dump' output.
06-16-2013 02:19 PM
This is the output for vmoprcmd -d
1 hcart TLD - No - 0
1 hcart TLD - No - 0
1 hcart TLD - No - 0
1 hcart TLD - No - 0
1 hcart TLD - No - 0
1 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
2 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
3 hcart TLD - No - 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
4 hcart TLD Yes SG2225 SG2225 Yes Yes 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
5 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
6 hcart TLD - No - 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
7 hcart TLD Yes SG0564 SG0564 Yes Yes 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
8 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
9 hcart TLD - No - 0
ADDITIONAL DRIVE STATUS
Drv DriveName Shared Assigned Comment
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
0 IBM.ULTRIUM-TD4.009 Yes tvmwpbkp54.ad.b
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
1 IBM.ULTRIUM-TD4.008 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
2 IBM.ULTRIUM-TD4.010 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
3 IBM.ULTRIUM-TD4.011 Yes -
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
4 IBM.ULTRIUM-TD4.012 Yes tvmwpbkp55.ad.b
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
5 IBM.ULTRIUM-TD4.013 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
6 IBM.ULTRIUM-TD4.014 Yes -
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
7 IBM.ULTRIUM-TD4.015 Yes tvmwpbkp54.ad.b
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
8 IBM.ULTRIUM-TD4.016 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
9 IBM.ULTRIUM-TD4.017 Yes -
06-16-2013 02:23 PM
MDS Allocation:
MDS allocations in EMM:
MdsAllocation: allocationKey=51378 jobType=1 mediaKey=4001591 mediaId=TV
0626 driveKey=2000823 driveName=IBM.ULTRIUM-TD4.003 drivePath=/dev/nst3 stuName=
tvlxpbkp01-hcart-robot-tld-1 masterServerName=tvmwpbkp51 mediaServerName=tvlxpbk
p01 ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiRe
sType=1 serverStateFlags=1
MdsAllocation: allocationKey=51384 jobType=1 mediaKey=4010362 mediaId=SG
2220 driveKey=2000936 driveName=IBM.ULTRIUM-TD4.009 drivePath=c128t0l1 stuName=t
vmwpbkp54-hcart-robot-tld-0-SBIDPNS03DM2 masterServerName=tvmwpbkp51 mediaServer
Name=tvmwpbkp54 ndmpTapeServerName=SBIDPNS03DM2 diskVolumeKey=0 mountKey=0 linkK
ey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=1
MdsAllocation: allocationKey=51385 jobType=1 mediaKey=4009060 mediaId=SG
0564 driveKey=2000930 driveName=IBM.ULTRIUM-TD4.015 drivePath=c144t0l4 stuName=t
vmwpbkp54-hcart-robot-tld-0-SBIDPNS03DM4 masterServerName=tvmwpbkp51 mediaServer
Name=tvmwpbkp54 ndmpTapeServerName=SBIDPNS03DM4 diskVolumeKey=0 mountKey=0 linkK
ey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=1
MdsAllocation: allocationKey=51387 jobType=1 mediaKey=4010357 mediaId=SG
2225 driveKey=2000927 driveName=IBM.ULTRIUM-TD4.012 drivePath=c144t0l1 stuName=t
vmwpbkp55-hcart-robot-tld-0-SBIDPNS03DM3 masterServerName=tvmwpbkp51 mediaServer
Name=tvmwpbkp55 ndmpTapeServerName=SBIDPNS03DM3 diskVolumeKey=0 mountKey=0 linkK
ey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=1
06-16-2013 09:17 PM
You forgot to show us reason for queued jobs in Activity Monitor Details tab.
Were vmoprcmd and nbrbutil commands done at more or less the same time?
vmoprcmd shows drives in use by media servers tvmwpbkp54 and tvmwpbkp55.
nbrbutil shows allocations for one more media server: tvlxpbkp01. Is there a job active for this media server?
I don't understand the multiple lines per drive in vmoprcmd output. Please tell us more about your environment:
Are all of these drives in one robot?
How many media servers? All sharing the drives?
Amount of drives and MPX per STU?
Are jobs queued for a particular STU?
08-28-2013 03:15 PM
Hi All sorry for the delayed response...
The issue has been resolved...As per Vendor's recommendation we have upgraded the library and the drive firmware after which the services have been bounced across all the master nd media servers after that the issue has been resolved..
Thanks a lot for ur help..cheers..