Forum Discussion

Maurice_Byrd's avatar
Maurice_Byrd
Level 4
14 years ago
Solved

Idle Tape Drives - Duplication jobs queueing

I'm running NBU 7.1 with one Master/Media and three Media servers.  All servers are running Windows Server 2008 R2 Standard with two 6-core processors and 24GB ram.  All policies except SQL and Catalog are set to write to a MSDP then to tape. My SLP's control which one of the two pools are used.  The SLP's are two step - backup to disk then duplicate to tape.  The six tape drives are all in a robotic library and shared between the four servers via fiber connection.

The backup jobs run fine to disk. However, the duplication jobs only run three at a time.  Having three idle drives is causing the duplication jobs to queue up.  All of the drives are up and available.  In fact, as one duplication job finishes a new one begins to write to one of the previously idle drives.   

I've tried enabling multiplexing on the tape STU's and creating multiple volume pools.  Neither change seems to have had an affect on the operation.

Ideas anyone?

7.1.x and Earlier
Backup and Recovery
NetBackup
  • Marianne's avatar
    Marianne
    14 years ago

    Just found something else that also caused a similar problem for another Connect member not so long ago...

    nbrbutil shows 6 tapes being allocated, but only 3 drives:

    mediaId=0008L5 (no drive)
    mediaId=0009L5 (no drive)
    mediaId=0010L5 (no drive)
    mediaId=0052L5 driveName=IBM.ULT3580-TD5.004
    mediaId=0005L5 driveName=IBM.ULT3580-TD5.002
    mediaId=0049L5 driveName=IBM.ULT3580-TD5.003

    Now what we saw in the other thread is that a media id allocation/reservation seems to 'assume' that a drive is associated with the media allocation and therefore reduces the number of drives available.

    Please release MDS allocation for these 3 media-id's and see if it resolves the queueing problem:

    nbrbutil -releaseMDS  42
    nbrbutil -releaseMDS  44
    nbrbutil -releaseMDS  52

    If you look at the allocation numbers for these 3 tapes in comparison with other allocation numbers (1118120, 2185063, 2210733, etc...) it looks like they have been 'stuck' for a long-long time....

     

18 Replies

  • Marianne's avatar
    Marianne
    Level 6
    14 years ago

    Just found something else that also caused a similar problem for another Connect member not so long ago...

    nbrbutil shows 6 tapes being allocated, but only 3 drives:

    mediaId=0008L5 (no drive)
    mediaId=0009L5 (no drive)
    mediaId=0010L5 (no drive)
    mediaId=0052L5 driveName=IBM.ULT3580-TD5.004
    mediaId=0005L5 driveName=IBM.ULT3580-TD5.002
    mediaId=0049L5 driveName=IBM.ULT3580-TD5.003

    Now what we saw in the other thread is that a media id allocation/reservation seems to 'assume' that a drive is associated with the media allocation and therefore reduces the number of drives available.

    Please release MDS allocation for these 3 media-id's and see if it resolves the queueing problem:

    nbrbutil -releaseMDS  42
    nbrbutil -releaseMDS  44
    nbrbutil -releaseMDS  52

    If you look at the allocation numbers for these 3 tapes in comparison with other allocation numbers (1118120, 2185063, 2210733, etc...) it looks like they have been 'stuck' for a long-long time....

     

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    14 years ago

    Multiplexing has no effect on the drives when it comes to duplication from disk.

    NetBackup does tend to follow some strange rules when it comes to SLP duplications, especially when groups are used as the targets

    If each media server has a storage unit that has six drives specified in its storage unit, which is then listed in the storage unit group and subsequently used in the SLP then it should really use all drives  - but i have seen it do otherwise!

    If however you have restricted each STU down to 3 drives it reflects this across to your group so you need to bring them back up to 6 if possible or make copies of them that have six drives available to put into a group and then the SLP

    I have found one thing that can help and this is to add the drive multiplyer to the LIFECYCLE_PARAMETERS files (in the netbackup\bin\ directory)

    Just add the line :

    TAPE_RESOURCE_MULTIPLIER  4

    May or may not help - groups always seem to cause issues

  • Maurice_Byrd's avatar
    Maurice_Byrd
    Level 4
    14 years ago

    Thanks Mark.  

    Since I have each tape STU set to use six drives I created the LIFECYCLE_PARAMETERS file in the ...\netbackup\db\config directory.  And added the multiplier line. After restarting services, the duplication jobs are still only utilizing three drives.

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    14 years ago

    Are the other drives up and of the same density?

    Do you have enough tapes available?

    Tell us more about your setup and configuration of everything

  • Marianne's avatar
    Marianne
    Level 6
    14 years ago

    Please also tell us what reason is listed for queued jobs in Job Details.

    Is this a new setup where all drives have never been active simultaneously or did it work before but then started 'misbehaving'?
    If it worked fine before, there is a possibility that there are orphaned device allocations in the resource broker. Bottom part of 'nbrbutil -dump' will confirm.

    As per Mark's post - we are going to need more info about your setup - STU config as well as SLP config...

  • Maurice_Byrd's avatar
    Maurice_Byrd
    Level 4
    14 years ago

    All the drives are the same type (LTO-5) and density (hcart2) with plenty of tapes available.  There is a STU configured for the master server and each media server to allow them to address the tape library. There are two MSDP's in use.

    Multiple SLP's are in use to spread the backup jobs across the media servers.  Each SLP has two steps - 1) Backup to disk, 2) Duplicate to tape.

    The queued duplication jobs have a status of Queued or Active.  The details show:

     

    Queued:  

    4/5/2012 2:51:55 PM - requesting resource LCM_atlnbumedia04-hcart2-robot-tld-0

    4/5/2012 2:51:56 PM - Info nbrb(pid=3504) Limit has been reached for the logical resource LCM_atlnbumedia04-hcart2-robot-tld-0    

     

    Active:

    4/5/2012 2:51:55 PM - requesting resource LCM_atlnbumedia04-hcart2-robot-tld-0

    4/5/2012 2:51:55 PM - Info nbrb(pid=3504) Limit has been reached for the logical resource LCM_atlnbumedia04-hcart2-robot-tld-0   

    4/9/2012 7:39:25 AM - granted resource LCM_atlnbumedia04-hcart2-robot-tld-0

    4/9/2012 7:39:25 AM - started process RUNCMD (10780)

    4/9/2012 7:39:25 AM - ended process 0 (10780)

    4/9/2012 7:39:26 AM - begin Duplicate

    4/9/2012 7:39:26 AM - requesting resource atlnbumedia04-hcart2-robot-tld-0

    4/9/2012 7:39:26 AM - requesting resource @aaaat

    4/9/2012 7:39:26 AM - reserving resource @aaaat

    4/9/2012 7:39:37 AM - awaiting resource atlnbumedia04-hcart2-robot-tld-0 - No drives are available

     

     

    Early last month I did have an issue with my drive paths showing down for some media servers. The only way I was able to resolve that issue was to remove the drives and STU's and run the wizard to add them back. The duplication backlog is recent but I can't definitely say the drive issue didn't contribute to the condition.  At the end of last month I migrated a few BE 9.x clients to my NBU environment.

     

     

  • Marianne's avatar
    Marianne
    Level 6
    14 years ago

    awaiting resource atlnbumedia04-hcart2-robot-tld-0 - No drives are available

    Please show us config for this STU:

    bpstulist -label atlnbumedia04-hcart2-robot-tld-0 -L

    as well the MDS allocation at the bottom of 'nbrbutil -dump' output.

    Please also verify that all drives are UP:

    vmoprcmd -d

  • Maurice_Byrd's avatar
    Maurice_Byrd
    Level 4
    14 years ago

     

    STU config:
     
    Label:                atlnbumedia04-hcart2-robot-tld-0
    Storage Unit Type:    Media Manager
    Host Connection:      _STU_NO_DEV_HOST_
    Number of Drives:     6
    On Demand Only:       no
    Density:              hcart2 (14)
    Robot Type/Number:    TLD (8) / 0
    Max Fragment Size:    1048576
    Max MPX/drive:        4
     
    The attached text file contains the MDS Allocations
    All drives are up.  I was not able to get the 'vmoprcmd -d' command to run.
  • Maurice_Byrd's avatar
    Maurice_Byrd
    Level 4
    14 years ago

    Is it possible to have NBU write images with the same expiration date to multiple tapes...

    1) without waiting for one tape to be full before using another tape?

    2) without mixing images with different expiration dates?

  • Marianne's avatar
    Marianne
    Level 6
    14 years ago

    Your storage unit config seems 'odd'. The name of the STU indicates a media server name but no Media Server name in 'Host Connection':

    Host Connection:      _STU_NO_DEV_HOST_

    Why is that? Seems you have at least 2 STU's with Host Connection not specified.

    We can see that 4 drives are in use. The STU names and media servers do not match because of the STU config:

    driveName=IBM.ULT3580-TD5.004 stuName=atlnbumedia02-hcart2-robot-tld-0 mediaServerName=atlnbumedia02
    driveName=IBM.ULT3580-TD5.005 stuName=atlnbumedia04-hcart2-robot-tld-0 mediaServerName=atlnbumedia02
    driveName=IBM.ULT3580-TD5.001 stuName=atlnbumedia04-hcart2-robot-tld-0 mediaServerName=atlnbumedia02
    driveName=IBM.ULT3580-TD5.003 stuName=atlnbumedia02-hcart2-robot-tld-0 mediaServerName=atlnbumedia04

    Please fix this as a start - select matching media server in Host Connection.

    Next, we need to understand how SLP's are configured - are you using storage unit groups or specific STU for duplication?
     

    Please check for DOWN drives. vmoprcmd is in <install_path>\veritas\volmgr\bin

    How NBU group jobs into batches for duplication and how it can be customized is explained in SLP Best Practice Guide: http://www.symantec.com/docs/TECH153154