backups job are hanging

[syedzeeshan]

nbu 6.5

windows 2003 r2

all of sudden from my media server backup Normal duplication jobs are hanging the back drives are engaged

- begin Duplicate

2/12/2012 8:00:14 AM - requesting resource STU_HO1BAK_IBMHO1_IBMHO2

2/12/2012 8:00:14 AM - awaiting resource STU_HO1BAK_IBMHO1_IBMHO2 Reason: Media is in use, Media Server: ho1bak,

                 Robot Number: 5, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,

                 Volume Pool: Weekly_Pool, Storage Unit: STU_HO1BAK_IBMHO1_ALL_U4, Drive Scan Host: N/A

2/12/2012 8:05:37 AM - granted resource 0179L4

2/12/2012 8:05:37 AM - granted resource IBM.ULT3580-TD4.002

2/12/2012 8:05:37 AM - granted resource STU_HO1BAK_IBMHO2_ALL_U4

2/12/2012 8:05:41 AM - started process bptm (4064)

2/12/2012 8:05:48 AM - started process bpdm (3952)

2/12/2012 8:05:53 AM - started process bptm (4064)

2/12/2012 8:05:53 AM - mounting 0179L4

2/12/2012 8:05:57 AM - begin rea

[Marianne]

I'm not sure I understand your question...

Are you saying backup jobs are hanging because duplication jobs are holding on to the drives?

The Activity Monitor details posted above is for a duplication job. It initially found the media needed to be in use, but 5 minutes later it was available and the duplication started.

[mph999]

Following Mariannes excellent post, exactly what is the problem ...

The dive is reported as busy, but then becomes available after a few minutes.

"2/12/2012 8:00:14 AM - awaiting resource STU_HO1BAK_IBMHO1_IBMHO2 Reason: Media is in use, Media Server: ho1bak"

I see you are accredited in NetBackup 7 - so you should have the skills to at least explain in detail the environment.  We cannot be expected to provide a solution with  little detail.

I see you say "all of sudden from my media server backup Normal duplication jobs are hanging the back drives are engaged"

OK, from this we can presume that this desn't normally happen, so, when one of these jobs runs and is waiting for the drive to become free, what is happening on the media server, for example, you could run vmoprcmd (volmgr\bin  directory) to list the drives on the media server, we can then see if hese drives contain tapes or not.

What other jobs are running when these duplication jobs start ?, you coul use bpdbjobs to list the jobs a the command line, then save this to a file and upload it here.

Martin

[Marianne]

Is this posted related to your other post?

https://www-secure.symantec.com/connect/forums/duplication-jobs-hanging

Please help us to understand your problem by giving us more information.
We are really trying to understand but you have not given enough details.

If you are battling with English, please type as much info as possible in your home language, then paste it in Google Translate to translate to English. (not just command output without explanation, please.)
As a reseller/Partner, think of all the info that you need to supply when logging a Support call on behalf of a customer.

I have seen that Google Translate does not always do a great job with translations, but it should help us to get a good understanding of your problem.

[syedzeeshan]

hi guys,

apparently the media server jobs are starting and it is taking 7-8 hours and still writing

dead slow speed .

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbrbutil -dump

Allocation Requests
(AllocationRequestSeq )


Allocations
(AllocationSeq
         index=0 (Allocation: id={629AFDB2-07EF-46F3-9DBD-2BAF8D4F7DBA} provider
=NamedResourceProvider resourcename=hobak1.NBU_POLICY.MAXJOBS.Ho2vatran_SQL_Diff
_Policy masterserver=hobak1 groupid={00000000-0000-0000-0000-000000000000} userS
equence=-1 userid="jobid=426025" named resource allocation)
         index=1 (Allocation: id={FC042F96-EA3D-47CA-8CA9-2700FE740DE1} provider
=MPXProvider resourcename=STU_HO1BAK_VLSHO1_ALL_U3 masterserver=hobak1 groupid={
7C5B298C-59B2-4E58-B968-B68A90744602} userSequence=0 userid="jobid=426025" (Medi
a_Drive_Allocation_Record: allocationKey=410811 (Media_Drive_Record: MediaKey=40
00977 MediaId=V047L3 MediaServer=ho1bak DriveKey=2000144 DriveName=HP.ULTRIUM3-S
CSI.013 PrimaryPath={2,0,1,6} PoolName=ArchiveLogs RobotNum=3 RobotType=8 MediaT
ypeName=NetBackup HCART3 DriveTypeName=NetBackup HCART3 NdmpControlHost= Retenti
onLevel=15 PolicyType=2 JobType=1 MasterServer=hobak1) (Storage_Unit_Record: STU
=STU_HO1BAK_VLSHO1_ALL_U3 STUType=2 MasterServer=hobak1 MediaServer=ho1bak Robot
Type=8 RobotNumber=3 Density=20 OnDemandOnly=0 ConcurrentJobs=8 ActiveJobs=1 Max
Multiplexing=1 NdmpAttachHost= AbsolutePath=) (Bptm_Strings_Record: 0="MEDIADB 1
 410811 V047L3 4000977 ------ 20 1328984096 1329145667 1329404867 0 73486976 34
34 15 10 0 0 1024 0 1148321 0" 1="VOLUME 1 V047L3 4000977 HO1V047L3 ArchiveLogs
*NULL* *NULL* 24 8 3 47 0 {00000000-0000-0000-0000-000000000000} 0" 2="DRIVE 3 H
P.ULTRIUM3-SCSI.013 2000144 01JU1nj-06 {2,0,1,6} -1 -1 -1 -1 -1 -1 -1 -1 *NULL*
*NULL* *NULL* *NULL* 1 128 0 1 0 0" 3="STORAGE 1 STU_HO1BAK_VLSHO1_ALL_U3 20 104
8576 2 1 0 0 ho1bak ho1bak *NULL*" 4="DISKGROUP 0 6 *NULL* 6 *NULL* 6 *NULL*" 5=
"DISKVOLUME 0 6 *NULL* 6 *NULL* 0" 6="DISKMOUNTPOINT 0 6 *NULL*" ) TpReqFileName
=))
         index=2 (Allocation: id={7BCB4476-1AD8-4AF5-905F-AB1620EC5A96} provider
=NamedResourceProvider resourcename=hobak1.NBU_POLICY.MAXJOBS.Ho1astran_SQL_Diff
_Policy masterserver=hobak1 groupid={00000000-0000-0000-0000-000000000000} userS
equence=-1 userid="jobid=426027" named resource allocation)
         index=3 (Allocation: id={C3E2859B-01D2-4A9B-8E32-B9BC2E6350B1} provider
=MPXProvider resourcename=STU_HO1BAK_VLSHO2_ALL_U3 masterserver=hobak1 groupid={
2315CE01-145F-41B3-A209-2F33AB77C263} userSequence=0 userid="jobid=426027" (Medi
a_Drive_Allocation_Record: allocationKey=410813 (Media_Drive_Record: MediaKey=40
00836 MediaId=V508L3 MediaServer=ho1bak DriveKey=2000122 DriveName=HP.ULTRIUM3-S
CSI.001 PrimaryPath={3,0,7,2} PoolName=ArchiveLogs RobotNum=2 RobotType=8 MediaT
ypeName=NetBackup HCART3 DriveTypeName=NetBackup HCART3 NdmpControlHost= Retenti
onLevel=15 PolicyType=2 JobType=1 MasterServer=hobak1) (Storage_Unit_Record: STU
=STU_HO1BAK_VLSHO2_ALL_U3 STUType=2 MasterServer=hobak1 MediaServer=ho1bak Robot
Type=8 RobotNumber=2 Density=20 OnDemandOnly=0 ConcurrentJobs=8 ActiveJobs=0 Max
Multiplexing=1 NdmpAttachHost= AbsolutePath=) (Bptm_Strings_Record: 0="MEDIADB 1
 410813 V508L3 4000836 ------ 20 1326035552 1329145785 1329404985 0 32447104 142
 55 15 10 0 0 1024 0 507343 0" 1="VOLUME 1 V508L3 4000836 HO2V508L3 ArchiveLogs
*NULL* *NULL* 24 8 2 9 0 {00000000-0000-0000-0000-000000000000} 0" 2="DRIVE 3 HP
.ULTRIUM3-SCSI.001 2000122 01FbPTS002 {3,0,7,2} -1 -1 -1 -1 -1 -1 -1 -1 *NULL* *
NULL* *NULL* *NULL* 1 128 0 1 0 0" 3="STORAGE 1 STU_HO1BAK_VLSHO2_ALL_U3 20 1048
576 2 1 0 0 ho1bak ho1bak *NULL*" 4="DISKGROUP 0 6 *NULL* 6 *NULL* 6 *NULL*" 5="
DISKVOLUME 0 6 *NULL* 6 *NULL* 0" 6="DISKMOUNTPOINT 0 6 *NULL*" ) TpReqFileName=
))
         index=4 (Allocation: id={E376887B-3C65-4A56-99BD-AA074D2FDFD3} provider
=NamedResourceProvider resourcename=hobak1.NBU_POLICY.MAXJOBS.Ho2vatran_SQL_Log_
Policy masterserver=hobak1 groupid={00000000-0000-0000-0000-000000000000} userSe
quence=-1 userid="jobid=426020" named resource allocation)
         index=5 (Allocation: id={078AB266-F14B-44C2-820D-2CE36A99CF55} provider
=MPXProvider resourcename=STU_HO1BAK_VLSHO1_ALL_U3 masterserver=hobak1 groupid={
9CE814BD-4AE7-4DE8-A708-8D1577DD6348} userSequence=0 userid="jobid=426020" (Medi
a_Drive_Allocation_Record: allocationKey=410812 (Media_Drive_Record: MediaKey=40
00945 MediaId=V015L3 MediaServer=ho1bak DriveKey=2000142 DriveName=HP.ULTRIUM3-S
CSI.011 PrimaryPath={2,0,1,4} PoolName=ArchiveLogs RobotNum=3 RobotType=8 MediaT
ypeName=NetBackup HCART3 DriveTypeName=NetBackup HCART3 NdmpControlHost= Retenti
onLevel=15 PolicyType=2 JobType=1 MasterServer=hobak1) (Storage_Unit_Record: STU
=STU_HO1BAK_VLSHO1_ALL_U3 STUType=2 MasterServer=hobak1 MediaServer=ho1bak Robot
Type=8 RobotNumber=3 Density=20 OnDemandOnly=0 ConcurrentJobs=8 ActiveJobs=1 Max
Multiplexing=1 NdmpAttachHost= AbsolutePath=) (Bptm_Strings_Record: 0="MEDIADB 1
 410812 V015L3 4000945 ------ 20 1329127648 1329145476 1329404676 0 27703744 14
14 15 10 0 0 1024 0 432908 0" 1="VOLUME 1 V015L3 4000945 HO1V015L3 ArchiveLogs *
NULL* *NULL* 24 8 3 15 0 {00000000-0000-0000-0000-000000000000} 0" 2="DRIVE 3 HP
.ULTRIUM3-SCSI.011 2000142 01JU1nj-04 {2,0,1,4} -1 -1 -1 -1 -1 -1 -1 -1 *NULL* *
NULL* *NULL* *NULL* 1 128 0 1 0 0" 3="STORAGE 1 STU_HO1BAK_VLSHO1_ALL_U3 20 1048
576 2 1 0 0 ho1bak ho1bak *NULL*" 4="DISKGROUP 0 6 *NULL* 6 *NULL* 6 *NULL*" 5="
DISKVOLUME 0 6 *NULL* 6 *NULL* 0" 6="DISKMOUNTPOINT 0 6 *NULL*" ) TpReqFileName=
))
         index=6 (Allocation: id={42D94D80-8075-4BC3-9A02-AC33FCFD78E6} provider
=NamedResourceProvider resourcename=hobak1.NBU_CLIENT.MAXJOBS.ho2vatranvs master
server=hobak1 groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 use
rid="jobid=426020" named resource allocation)
         index=7 (Allocation: id={B0A018F2-C701-4597-AC24-10E769B520FB} provider
=NamedResourceProvider resourcename=hobak1.NBU_CLIENT.MAXJOBS.ho2vatranvs master
server=hobak1 groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 use
rid="jobid=426025" named resource allocation)
         index=8 (Allocation: id={C3D20D04-1D3F-4CBC-8141-E92C40D1E7A7} provider
=MPXProvider resourcename=anMPXGroup masterserver=hobak1 groupid={7C5B298C-59B2-
4E58-B968-B68A90744602} userSequence=-1 userid="jobid=426025" (MPXGroupAllocatio
n: groupId={7C5B298C-59B2-4E58-B968-B68A90744602}))
         index=9 (Allocation: id={A8ED4544-6EBE-43D8-9449-7771EFB28BB3} provider
=NamedResourceProvider resourcename=hobak1.NBU_CLIENT.MAXJOBS.ho1astranvs master
server=hobak1 groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 use
rid="jobid=426027" named resource allocation)
         index=10 (Allocation: id={3F289FA4-EBE0-4F94-96BF-2C4A14AF9796} provide
r=MPXProvider resourcename=anMPXGroup masterserver=hobak1 groupid={9CE814BD-4AE7
-4DE8-A708-8D1577DD6348} userSequence=-1 userid="jobid=426020" (MPXGroupAllocati
on: groupId={9CE814BD-4AE7-4DE8-A708-8D1577DD6348}))
         index=11 (Allocation: id={DC9D7083-634C-42A5-BEBD-2BA47917E0D4} provide
r=MPXProvider resourcename=anMPXGroup masterserver=hobak1 groupid={2315CE01-145F
-41B3-A209-2F33AB77C263} userSequence=-1 userid="jobid=426027" (MPXGroupAllocati
on: groupId={2315CE01-145F-41B3-A209-2F33AB77C263})))

MDS allocations in EMM:

        MdsAllocation: allocationKey=410811 jobType=1 mediaKey=4000977 mediaId=V
047L3 driveKey=2000144 driveName=HP.ULTRIUM3-SCSI.013 drivePath={2,0,1,6} stuNam
e=STU_HO1BAK_VLSHO1_ALL_U3 masterServerName=hobak1 mediaServerName=ho1bak ndmpTa
peServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=1 se
rverStateFlags=1
        MdsAllocation: allocationKey=410812 jobType=1 mediaKey=4000945 mediaId=V
015L3 driveKey=2000142 driveName=HP.ULTRIUM3-SCSI.011 drivePath={2,0,1,4} stuNam
e=STU_HO1BAK_VLSHO1_ALL_U3 masterServerName=hobak1 mediaServerName=ho1bak ndmpTa
peServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=1 se
rverStateFlags=1
        MdsAllocation: allocationKey=410813 jobType=1 mediaKey=4000836 mediaId=V
508L3 driveKey=2000122 driveName=HP.ULTRIUM3-SCSI.001 drivePath={3,0,7,2} stuNam
e=STU_HO1BAK_VLSHO2_ALL_U3 masterServerName=hobak1 mediaServerName=ho1bak ndmpTa
peServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=1 se
rverStateFlags=1

 

[Marianne]

nbrbutil output does not help to understand your issue.

Please help us to understand your duplication issue.

How is your duplication job configured - Vault? SLP? GUI? bpduplicate from cmd? DSSU?

If Vault, SLP, DSSU or GUI, please share screenshot of selection/config.
If bpduplicate, share command that is used.

Are you perhaps duplicating MPX'ed backup to non-MPX? This selection will result in DOG-slow duplication.

What are the buffer sizes in ...netbackup\db\config on ho1bak?

Please ensure that you have bptm and bpdm log folders on ho1bak.

Also let us know how devices are attached to ho1bak. Are devices (disk and tape) attached to same or different hba's?
OS on ho1bak?
NBU version on ho1bak?

Please post bptm and bpdm logs from completed duplication.

[mph999]

Do all these steps  ...

1.  Answer the questions Mariane posted above

I don't know if you are on windows or unix - but in netbackup\db\config do you have files called 

SIZE_DATA_BUFFERS

NUMBER_DATA_BUFFERS

on the media servers ???

If you do, please tell me what values they contain.

2.  What are the tape drives (VTL/ Physical)

3.  What type are the tape drives, eg LTO4 ?

4.  Has this problem happened before

5.  Is this a new setup, or a new media server

6.  Do backup jobs run quick on the media server(s)

7.  If ou do not know the answer to 6), create a temp policy and backup the media server itsef, does this go quick.

Martin