09-22-2016 07:17 AM
Hello All,
NBU Master:7.6.0.2
OS:WIN 2K8 R2
I am facing few issues with duplication jobs.Everytime they are triggered,they go in queued and job state state says "Drives are in Use" although i have drives which are active and ready for use.Secondly.they fail with 96 most of the times and sometimes with ec 191.
I need to know from experts what all steps i need to follow to proceed with troubleshooting this issue.
Thanks.
09-22-2016 11:40 AM
Hello Shekaib1,
could you provide more details
1)detail status of the duplicate job
2)how many tape libraries configured in this master server
3)if multiple libraries, does the avaliable drives belongs to same library that duplication job is looking?
4)could you show us the output of tpconfig -d and vmoprcmd -d from the media server that is haivng stroge unit allocated for this duplication jobs?
5)does backup jobs working fine on these drives?
6)what is the location of source copy? does it in tape of disk.. if tape.. does the tpae is avaliable in library ?
09-22-2016 10:00 PM
10-10-2016 12:37 PM
Please find the answers
1)10/10/2016 10:23:49 - requesting resource LCM_SUG_Tape_Duplicate
10/10/2016 10:23:50 - granted resource LCM_SUG_Tape_Duplicate
10/10/2016 10:23:50 - started process RUNCMD (2576)
10/10/2016 10:23:50 - ended process 0 (2576)
10/10/2016 10:23:50 - begin Duplicate
10/10/2016 10:23:51 - requesting resource SUG_Tape_Duplicate
10/10/2016 10:23:51 - requesting resource @aaaaj
10/10/2016 10:23:51 - reserving resource @aaaaj
10/10/2016 10:23:52 - Error nbjm(pid=7212) NBU status: 96, EMM status: No media is available
10/10/2016 10:23:52 - Error nbjm(pid=7212) NBU status: 96, EMM status: No media is available
10/10/2016 10:23:52 - Error nbjm(pid=7212) NBU status: 96, EMM status: No media is available
10/10/2016 10:23:53 - end Duplicate; elapsed time: 0:00:03
unable to allocate new media for backup, storage unit has none available(96)
2)1
4)tpcnfig -d
C:\Program Files\Veritas\Volmgr\bin>tpconfig -d
Id DriveName Type Residence
SCSI coordinates/Path Status
****************************************************************************
0 HP.ULTRIUM5-SCSI.000 hcart2 TLD(0) DRIVE=1
{1,0,5,0} UP
1 HP.ULTRIUM5-SCSI.001 hcart2 TLD(0) DRIVE=2
{1,0,6,0} UP
2 HP.ULTRIUM5-SCSI.002 hcart2 TLD(0) DRIVE=3
MISSING_PATH:{2,0,7,0}:HU1241RN9H DOWN
3 HP.ULTRIUM5-SCSI.003 hcart2 TLD(0) DRIVE=4
{1,0,4,0} UP
4 HP.ULTRIUM5-SCSI.004 hcart2 TLD(0) DRIVE=3
{1,0,3,0} UP
Currently defined robotics are:
TLD(0) SCSI coordinates = {1,0,5,1}
EMM Server = hg1-nbmas01.hug.hardygroup.co.uk
vmoprcmd -d
C:\Program Files\Veritas\Volmgr\bin>vmoprcmd -d
PENDING REQUESTS
<NONE>
DRIVE STATUS
Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId
0 hcart2 TLD - No - 0
1 hcart2 TLD - No - 0
2 hcart2 DOWN-TLD - No - 0
3 hcart2 TLD - No - 0
4 hcart2 TLD - No - 0
ADDITIONAL DRIVE STATUS
Drv DriveName Shared Assigned Comment
0 HP.ULTRIUM5-SCSI.000 Yes -
1 HP.ULTRIUM5-SCSI.001 Yes -
2 HP.ULTRIUM5-SCSI.002 Yes -
3 HP.ULTRIUM5-SCSI.003 Yes -
4 HP.ULTRIUM5-SCSI.004 Yes -
C:\Program Files\Veritas\Volmgr\bin>
5)yes backups are completing
6)Source is tape and duplication goes onto tapes only.Currently 4 tapes are showing active in media database.
we have SLP's defined for daily,weekly and monthly retentions.It seems only daily SLP gets completed but rest fail with ec 96.
10-10-2016 04:01 PM
SUG_Tape_Duplicate <== Is this your Storage Unit Group for tape storage?
Can you check if it includes storage units for robot 0 & hcart2 density type?
If it is, make sure the robot has available media (do an inventory first), make sure no other jobs are using the other 4 UP drives, run a small duplication job to see if it picks up the media.
10-10-2016 07:23 PM
10-11-2016 07:18 AM
attached is avaiable media output and SLP screenshot for reference
10-11-2016 07:34 AM
You forgot about available_media output....
Please copy the output to media.txt and upload.
PS:
You show a screenshot of Monthly SLP with Infinity retention.
Do you realize that tapes in this pool with Infinity retention will never expire? i.e. can never be overwritten?
Only tapes in this pool with same retention level can be appended to.
10-11-2016 09:59 AM
10-11-2016 12:19 PM
10-11-2016 05:31 PM
Marianne,
I do not know the reason of DBBACKUP tapes.I have been given this environment few days backup and i do not know where to start from.It seems root of problem looks like tapes only.i have also seen in past that we had scratch tapes in library but they were not being mounted on the drives.I did rob test to move medias and they did fine.
I would log a case with Support to get this checked and also get more tapes loaded to see if it makes a difference.Also would apprecite advice from experts on how to move further in this scenario.
Yes,we have scarcth pool but with zero tapes.
10-11-2016 06:03 PM
Also,we have around 30 tapes showing in media database.
C:\Program Files\Veritas\NetBackup\bin\admincmd>bpmedialist -m K08274
requested media id is not assigned to this host in the EMM database
C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listmedia -mediaid K08
274
NBEMMCMD, Version: 7.6.0.2
The function returned the following failure status:
volume does not exist in database (35)
Command did not complete successfully.
can i put this media in library?
10-12-2016 12:33 AM
You made a typo error - media id is KO8274 not K08274.
Please try nbemmcmd command again with correct media id.
Run nbemmcd command as well for a tape with DBBACKUP status - e.g. KK7204
I am curious to know what is going on here.
The DBBACKUP status happened in pre-6.x days when each media server had its own mediaDB database and volDB was on the master. When mediaDB on a media server was lost, there was this inconcistency in the db's with tapes showing Assigned in volDB without matching nbemmcmd entry.
This should not happen in versions from 6.0 onwards as all these dbs were migrated into the EMM database on the master server.
Can you run these commands and upload output of each?
bpmedialist
vmquery -a -bx
available_media is a combination of above 2 commands, so I am curious to see what they produce separately.
Best to get that NBCC running and submit to Support as a matter of urgency.
In the meantime, eject all the DBBACKUP status tapes that are currently in the robot and replace with AVAILABLE tapes.
You need to get duplications going ASAP to prevent huge SLP backlog.
And please start with a daily process to look for unassigned tapes and put them into Scratch pool.
If unassigned tapes are moved to Scratch pool, they will from there onwards go back to scratch when images on tapes expire.
10-12-2016 09:19 AM
Please find the list of medias and vmquery as requested.
C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listmedia -mediaid KK7204
NBEMMCMD, Version: 7.6.0.2
====================================================================
Media GUID: 5c31dc57-ae72-4597-8f36-221592b035f1
Media ID: KK7204
Partner: -
Media Type: HCART2
Volume Group: 000_00000_TLD
Application: Netbackup
Media Flags: 1
Description: Added by Media Manager
Barcode: KK7204L4
Partner Barcode: --------
Last Write Host: hg1-nbmed01.hug.hardygroup.co.uk
Created: 05/09/2016 15:24
Time Assigned: 29/09/2016 05:44
First Mount: 29/09/2016 08:22
Last Mount: 29/09/2016 08:22
Volume Expiration: -
Data Expiration: 13/10/2016 02:04
Last Written: 29/09/2016 13:32
Last Read: -
Robot Type: TLD
Robot Control Host: hg1-nbmed01.hug.hardygroup.co.uk
Robot Number: 0
Slot: 60
Side/Face: -
Cleanings Remaining: -
Number of Mounts: 1
Maximum Mounts Allowed: 0
Media Status: FULL
Kilobytes: 1026132906
Images: 7
Valid Images: 7
Retention Period: 1
Number of Restores: 0
Optical Header Size Bytes: 1024
Optical Sector Size Bytes: 0
Optical Partition Size Bytes: 0
Last Header Offset: 1099087
Adamm Guid: 00000000-0000-0000-0000-000000000000
Rsm Guid: 00000000-0000-0000-0000-000000000000
Origin Host: NONE
Master Host: hg1-nbmas01.hug.hardygroup.co.uk
Server Group: UNRESTRICTED_SHARING_GROUP
Upgrade Conflicts Flag:
Pool Number: 4
Volume Pool: ENCR_NetBackup
Previous Pool Name: -
Vault Flags: -
Vault Container: -
Vault Name: -
Vault Slot: -
Session ID: -
Date Vaulted: -
Return Date: -
Media on Hold: 0
====================================================================
Command completed successfully.
10-12-2016 09:20 AM
10-12-2016 10:43 AM
10-13-2016 04:58 PM
Hello Marianne,
I have checked in media DB and found few tapes to be active,no time assigned on them,not in robot.I have put those tape in library and see them as active now and they are getting assigned.
Currently we have 14 duplication jobs running and 3 are active.1drive is being used for backup.Is there any way to fasten these duplication jobs.
Also one of the drives is showing missing path.Do i have delete the drive and then re add it
10-13-2016 09:00 PM
10-19-2016 07:49 AM
yes,queued jobs had status "Drives are in Use".
But now due to shortage of tapes,mangement has decided to inactivate few SLP's.Since we have SLP for daily,weekly and monthly backups.
So i am now trying to disable monthly till we get new tapes.But when i try to disable it.
C:\Program Files\Veritas\NetBackup\bin\admincmd>nbstlutil inactive -lifecycle SUG_Dedupe_SUG_Tape_Monthly
nbstlutil: unknown operation '-lifecycle'
nbstlutil: unknown operation 'SUG_Dedupe_SUG_Tape_Monthly'
Is there something else i need to add/edit in the above command.
10-19-2016 08:01 AM
The command looks fine - just be very sure that you use a hyphen (-) and not a dash (–)
When we copy from a doc, it is sometimes a dash (–lifecycle) instead of hyphen (-lifecycle)
Also remember to open cmd with 'Run as Administrator'.