A pending request has been generated for this resource reque

[kertvaros]

HI all,

I saw several posts from years ago regarding the problem I have but they did not help me find the solution.

In an nutshell: Netbackup 7.5 Master server is on Windows 2008 server and three other Madeia servers are also running on Windows 2008 server. Four LTO4 tape drives in HP MLS6000 tape library are attached to two of the Media servers, two drives per each of the two media server.

I am trying to duplicate tapes with barcode ending on L1 to tapes with barcore ending on L4 and everything looks good and normal in setup but wken I kick off duplication job, it remains wueued for ever not doing eanything, displaying the following message in a detailed report in Activity monitor:

06/20/2014 3:15:03 PM - awaiting resource 600276 A pending request has been generated for this resource request.
  Operator action may be required. Pending Action: No action.,
  Media ID: 600276, Barcode: 600276L1, Density: hcart, Access Mode: Read,
  Action Drive Name: N/A, Action Media Server: N/A, Robot Number: N/A, Robot Type: NONE,
  Volume Group: 001_00001_TLD, Action Acs: N/A, Action Lsm: N/A

Output of relevant commands on a Master server and Media server is as follows:

F:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listmedia -mediaid 600276
NBEMMCMD, Version: 7.5.0.4
====================================================================
Media GUID:                     db790df5-7581-40f3-812e-40ad4754b2ba
Media ID:                       600276
Partner:                        -
Media Type:                     HCART
Volume Group:                   001_00001_TLD
Application:                    Netbackup
Media Flags:                    1
Description:                    Added by Media Manager
Barcode:                        600276L1
Partner Barcode:                --------
Last Write Host:                prod1bak01
Created:                        09/23/2011 02:27
Time Assigned:                  09/27/2011 18:30
First Mount:                    09/27/2011 18:51
Last Mount:                     09/27/2011 18:51
Volume Expiration:              -
Data Expiration:                INFINITY
Last Written:                   09/27/2011 20:11
Last Read:                      -
Robot Type:                     TLD
Robot Control Host:             0prd1wbak04
Robot Number:                   1
Slot:                           16
Side/Face:                      -
Cleanings Remaining:            -
Number of Mounts:               1
Maximum Mounts Allowed:         0
Media Status:
Kilobytes:                      32020705
Images:                         8
Valid Images:                   8
Retention Period:               9
Number of Restores:             0
Optical Header Size Bytes:      1024
Optical Sector Size Bytes:      0
Optical Partition Size Bytes:   0
Last Header Offset:             0
Adamm Guid:                     00000000-0000-0000-0000-000000000000
Rsm Guid:                       00000000-0000-0000-0000-000000000000
Origin Host:                    NONE
Master Host:                    prod1bak01
Server Group:                   NO_SHARING_GROUP
Upgrade Conflicts Flag:
Pool Number:                    12
Volume Pool:                    ENCR_Netbackup
Previous Pool Name:             Scratch
Vault Flags:                    -
Vault Container:                -
Vault Name:                     -
Vault Slot:                     -
Session ID:                     -
Date Vaulted:                   -
Return Date:                    -
Media on Hold:                  0
====================================================================
Command completed successfully.

-------

F:\Program Files\Veritas\NetBackup\bin\admincmd>bpstulist -label bak03_hp
bak03_hp 2 prd1wbak03 8 1 6 2 0 "*NULL*" 0 1 1048576 *NULL* 0 1 0 0 0 0 *NULL* prd1wbak03 0

------------------------

M:\Program Files\Veritas\Volmgr\bin>tpconfig -l
Device Robot Drive       Robot                    Drive                 Device
Type     Num Index  Type DrNum Status  Comment    Name                  Path

robot      1    -    TLD    -       -  -          -                     prd1wbak04
  drive    -    0  hcart    3      UP  -          HP.ULTRIUM4-SCSI.002  {3,0,0,0}
  drive    -    1  hcart    4      UP  -          HP.ULTRIUM4-SCSI.003  {3,0,0,1}

M:\Program Files\Veritas\Volmgr\bin>vmoprcmd -d

                                PENDING REQUESTS

ReqId      User    RecMID  ExtMID  Density  Mode    Time   Barcode   VolGroup
726131                     600276  hcart    Read    15:24  600276L1  001_00001_
725544                     600320  hcart    Read    15:23  600320L1  001_00001_
731733                     600279  hcart    Read    15:42  600279L1  001_00001_

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart    TLD                -                     No       -         0
  1 hcart    TLD                -                     No       -         0
 

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 HP.ULTRIUM4-SCSI.002  No       -
  1 HP.ULTRIUM4-SCSI.003  No       -

Thnaks in advance for your help!

Kerti

[SymTerry]

Hello,

I would like to get a little more detail on how your doing the duplication. This thread outlines the steps to do a tape to tape duplication.

Are you keeping the new tapes in a different storage pool for this duplication? Also please post the detailed status of the job in the activity monitor as well.

[RamNagalla]

how did you tirgger the restore...?

did you select alternate read or medaia server stu for duplication or same media server which is used for backup?

show us the output of only vmoprcmd 

[kertvaros]

I basically followed the following recipe given in one of the previous postings EXCEPT: I selected the same volume pool as original one. Why duplicate tape cannot be in the same pool as original? I think we want to have thenm both in the same pool.

Here is what I have done:

in the GUI
go to Catalog
in the action pull down choose duplicate ( this means look for items that qualify for duplication)
in media id put in the id of the source tape.
in date range start back in 2000 or some date you know is before the images on the tape ( as you are only searching this one tape it will not find a lot)
now hit the search button

Once it comes back in the bottom with all the images on the tape. pick all or just the ones you want to duplicate.

right click the highlighted ones and choose duplicate

in the new screen
Say how many copies you want
check mark if you want the dup tape to become the primary
choose a storage unit
choose a volume pool - try to choose a pool OTHER than the one the source tape is in - as it might try to dup the tape to the same tape.
your choice to keep or change the retention
 if you keep multiplexing it will go faster, if you do not keep it it will take longer to dup.

-----------

Thanks!

Kerti

[kertvaros]

Thanks for you response;

Here it is:

M:\Program Files\Veritas\Volmgr\bin>vmoprcmd

                           HOST STATUS
Host Name                                  Version   Host Status
=========================================  =======   ===========

prd1wbak03                                 750400    ACTIVE
prd1wbak04                                 750600    ACTIVE
 

                                PENDING REQUESTS

                                    <NONE>

                                  DRIVE STATUS

Drive Name                     Label   Ready     RecMID  ExtMID  Wr.Enbl.  Type
    Host                                       DrivePath                                             Status
=============================================================================
HP.ULTRIUM4-SCSI.000     No      No                                      No        hcart
        prd1wbak04                         {4,0,0,0}                                              TLD

HP.ULTRIUM4-SCSI.001     No      No                                      No        hcart
        prd1wbak04                         {4,0,0,2}                                              TLD

HP.ULTRIUM4-SCSI.002     No      No                                      No        hcart
        prd1wbak03                          {3,0,0,0}                                             TLD

HP.ULTRIUM4-SCSI.003     No      No                                      No        hcart
        prd1wbak03                          {3,0,0,1}                                             TLD

-------------

Kerti

[Marianne]

L1 tapes are LTO1 tapes, right?

LTO1 tapes can only be mounted in LTO1, LTO2 or LTO3 tape drives.
Not in LTO4 drives.

[kertvaros]

Marianne,

actually they are LTO4 tapes but only have label with the barcode that ends on L1.  A week ago we were able to duplicate single tape on our first trial with no problems from the one whose barcode ends on L1 to the one whose barcode ends on L4 but hit the snag with our second attempt and few subequent attempts to duplicate. Dulication job goes to state described above and stays there forever.

Thank you for your response and interest in this matter.

It is highly appreciated!

Kerti

[Marianne]

Try to suspend all the L1 source tapes.

The problem here is that the source tape are not Full, not supended, frozen, etc, same media server, same pool, same retention.

NBU duplication uses the same selection criteria that is used for backups - if a suitable tape is found that can be appended to, it will select that. So, it can happen that the same source tape is then selected as destination (resulting in Pending Request). This is why the manual says that a different pool should be selected.

Suspending the source tapes should exclude them from selection as destination.

[Deb_Wilmot]

Check the C:\Program Files\Veritas\NetBackup\db\media\drives directory for stale files associated with these drives names.  *** Stale files are files that are not actually in use.  If you view the file(s) you will see a PID in them associated with a bptm pid that should be active on the system.  If the bptm pid is not active on the system, (doesn't show up in the process list), the file can be deleted.  

Stale files in this directory will cause allocation of resources to not occur.  The resource broker (nbrb) polls those files every 10 minutes.  If a stale file exists, we can't reallocate the drive, because we don't know the state of the drive.

[kertvaros]

I suspended the source tape, same result; I unsuspended it and froze it, still same result:

07/02/2014 3:41:54 PM - begin Duplicate
07/02/2014 3:41:54 PM - requesting resource bak04_hp
07/02/2014 3:41:54 PM - requesting resource 600279
07/02/2014 3:41:54 PM - reserving resource 600279
07/02/2014 3:41:55 PM - awaiting resource 600279 A pending request has been generated for this resource request.
  Operator action may be required. Pending Action: No action.,
  Media ID: 600279, Barcode: 600279L1, Density: hcart, Access Mode: Read,
  Action Drive Name: N/A, Action Media Server: N/A, Robot Number: N/A, Robot Type: NONE,
  Volume Group: 001_00001_TLD, Action Acs: N/A, Action Lsm: N/A

Thanks!

Kerti

[kertvaros]

Hi Deb,

it seems that there are no stale files in C:\Program Files\Veritas\NetBackup\db\media\drives.

The only file there is:

drive_HP.ULTRIUM-SCSI.002    11/27/2013 12:23 PM  002 file

If I try to open this file I get a message saying that Windows cannot open that file.

Thanks!

Kerti

[Marianne]

I am now reading all posts again.... some of Nagalla's questions on 28 June are still unanswered. Like, how was the restore triggered, did you specify Altread host to read the tape? Which STU as destination? These questions are very relevant, as nbemmcmd output shows that media server prod1bak01 wrote the backups on this tape. We do not see this media server in vmoprcmd output. What happened to this media server? This is the media server for which the pending request is generated.

[kertvaros]

Marianne,

I am just trying to duplicate the tape that was written in 2011, not to initiate restore off of that tape.

To do that, in Activity Monitor on Master server prod1bak01 I click on Catalog, and fiollow the procedure outlined in one of my previous postings. Destination storage unit is HP tape library, or more precise, Storage group  that comprises of two sets of two tape drives in that tape library. Robot in a library is under the control of wbak04 media server.

prod1bak01 is current Master server but is not Media server for particular HP library any more (it may have been in 2011 in old configuration that was changed before I came on board); at present time there are two media servers attached to that library, wbak03 and wbak04 respectfully, each of them controlling two tape drives in HP library.

I gather from your post that Media server for duplication needs to be changed  to be either wbak03 or wbak04, one of the current Media servers for duplication to occur. How and where do I specify that in duplication setup?

I hope I answered all questions asked; if there is still some ambiguity, please don't hesitate to ask.

Thank you,

Kerti

[Marianne · Accepted solution]

LOL! You seem to miss the fact that a duplication is in fact a restore and backup. The source tape is read similar to a restore and data sent to the STU selected as destination. Without specifying Altreadhost, the read request is sent to the media server that did the backup. Altreadhost cannot be specified in the GUI, only when you run bpduplicate from cmd. In this instance, your master is no longer a media server and should have been decommissioned as media server by changing all media ownership to one of the current media servers. You can do so with this command: bpimage -oldserver master-name -newserver media-server-name The read request will now be sent to the new media owner. Preferably the same media server that you select as STU to prevent data being sent across the network. Hope this helps.

[Deb_Wilmot]

I see your note about the file in db\media\drives.  That is actually a stale file.  Note the date on the file:

drive_HP.ULTRIUM-SCSI.002    11/27/2013 12:23 PM  002 file

*** it's 2013.  I doubt you have had a backup running since 2013. :)

Remove that file.

Deb

[kertvaros]

Actually prod1bak01 is both Master server for entire site and media server for another library with LTO5 tape drives. I believe if I execute bpimage command as suggested above, media ownership will change not only for desrred media but for undesired too, those that need to stay owned by prod1bak01.

My best bet seems to be defining Altreadhost on  command line as an argument to bpduplicate command.

Netbackup documentation is overhelming. Where can more information on bpduplicate be found?

Thank you!

Kerti

[Marianne]

You can change ownership for individual pieces of media with bpmedia command: bpmedia -m (media-id) -newserver (media-server-name) More about all NBU commands in the Command Reference Guide. Links to all NBU manuals in 'Handy NBU Links' in my signature.

[kertvaros]

Thank you, Deb!

Since there are four drives in a tape library that has not been used a lot lately, it was not noticed that one of the tape drives may not be taking any jobs.

I deleted the file and will test the library soon.

Thanks again!

Kerti

[kertvaros]

It seems like -movedb is missing in command above; According to Command Reference Guide (thanks for the reference!) it should read:

> bpmedia -movedb -m (media-id) -newserver (media-server-name)

This command works; without -movedb it does not.

Thanks a lot!

Kerti

[Marianne]

You are 100% correct!

Just sometimes I am brave enough to post from my mobile phone without double-checking manuals....