Odd issues with restoring data from tape

[Toddman214]

Netbackup 7.1.0.4, Windows 2008R2, Dell ML6000 series libraries.

We have a disaster recovery rehearsal this weekend at our recovery site, and I am having an odd issue with performing a tape restore from the library at that site.

History ->  I located some test data, and wrote it to tape media ID 2904L3 (barcode 002904L3) in our primary tape library. We always use the last 6 characters as the media ID. Once I confirmed the needed data was written to this tape, I took the tape to our recovery site and loaded it into the library there, and inventoried it. The tape showed up in there as media ID 002904. Since the catalog should know this data is on media ID 2904L3, I access the tape library interface, and changed the labeling setting from "standard" to "plus 6". Once I applied that setting, and reinventoried the library at our recovery site, now the tape ID shows properly as 2904L3 (barcode 002904L3).

Here is the problem. When I set this tape as the primary copy, and kicked off a restore, this job just hangs there, and tells me that "A pending action is associated with the resource request. User intervention may be required", which generally means "load the tape, dummy." I noticed that the library did not load the tape into a drive, so I again accessed the library interface, and "moved" the tape into a drive. Then, the message changed to "this media ID is already in use." It's as if Netbackup now doesnt realize that this is in fact the same tape.

Any clues as to what is going on here? I'll provide more info as needed. Its getting to be crunch time. :(

Thank you!

Todd

[revarooo]

What is the output of 

tpconfig -d

vmquery -m 2904L3

(for the record, I would use the first 6 characters. What would happen if you had tapes 002904L3 and 012904L3  - you'd hit a duplicate tape label issue)

[leonard_boyle]

Also how is the tape drive defined?

Does it have the same drive type as the lto3 tape cart.  That is are they both HCART or HCART2 or ...

How about you storage units, volume pools.

Do you have bptm logging set to 5. If so what is the log have for this mount request.

[Toddman214]

@revaroo

tpconfig -d just shows my master server

E:\>tpconfig -d
Id  DriveName           Type   Residence
      SCSI coordinates/Path                                            Status
****************************************************************************

Currently defined robotics are:

EMM Server = pdc00nbua801w.ohlogistics.com

E:\>

vmquery - 2904L3 gives the following results

E:\>vmquery -m 2904L3
================================================================================
media ID:              2904L3
media type:            1/2" cartridge tape 3 (24)
barcode:               002904L3
media description:     Added by Media Manager
volume pool:           Test (7)
robot type:            NONE - Not Robotic (0)
volume group:          ---
vault name:            ---
vault sent date:       ---
vault return date:     ---
vault slot:            ---
vault session id:      ---
vault container id:    -
created:               8/15/2011 11:02:36 AM
assigned:              9/10/2012 10:48:28 AM
last mounted:          9/10/2012 10:49:08 AM
first mount:           8/16/2011 12:46:14 AM
expiration date:       ---
number of mounts:      35
max mounts allowed:    ---
status:                0x0
================================================================================

E:\>

@Leonard Boyle

The drive itself that Im trying to restore the data with is an LTO4. The tape is LTO3 and was written by an LTO3 drive. Whats odd is that in the activity monitor under the drives tab at the bottom, all of my other drives are showing as HCART3. The drives in the library at the recovery site that I am trying to restore from are showing as HCART.

As for my storage units for that media server/library, there are two....one listed as HCART and one as HCART3.

As for the logging level, it is set to verbose 5, but I'm not sure I understand what you are asking there. Im still learning this stuff. Just barely over half a year under my belt.

[leonard_boyle]

In general you can look in bptm log for the details of the mount request for the tape. If there is a problem, the details of the problem should be in the log.

In this case the first thing to fix things so  that the hcart3 setting for the tape cart  matchs that of the tape drive and the storage unit for the tape drive.

There are multiple ways to do this, but you should be able to do it in the devices section of the netbackup admin gui.

[Marianne]

Len is 100% right - you need to config drives at DR site as hcart3.
Although LTO3 tapes can be used in LTO4 drives, densities in NBU must match.
NBU will never load hcart3 tape in hcart drive.

[Toddman214]

Ok, after noticing that differences once Leonard led me in that direction (thank you Leonard), I changed all of the HCART drives to HCART3 drives. I'm still having the same issues that I opened this posting about, BUT, this did resolve an open ticket I have with our support vendor (silver lining). :) Ive had one LTO4 drive that would NEVER allocate a tape. As soon as I saved the changes, it instantly allocated a tape and started writing.

Still have to knock out this issue though. ---->

9/12/2012 2:45:31 PM - awaiting resource 2904L3 A pending request has been generated for this resource request.
  Operator action may be required. Pending Action: No action.,
  Media ID: 2904L3, Barcode: 002904L3, Density: hcart3, Access Mode: Read,
  Action Drive Name: N/A, Action Media Server: N/A, Robot Number: N/A, Robot Type: NONE,
  Volume Group: ---, Action Acs: N/A, Action Lsm: N/A

I wonder if maybe  a tape pool or something is not being shared with the library at the recovery site.

[mph999]

Not sure which server the vmquery output is from ...

media description:     Added by Media Manager
volume pool:           Test (7)
robot type:            NONE - Not Robotic (0)
volume group:          ---

...but it's showing as non-robotic 

No matter ... just confirm that on the DR site there are drives set the same density as the media and it is showing in vmquery that it is in the robot and has a slot number.

Then check in robtest (s s) command that it really is in the slot.

Apologies that it is back to basics, but sometimes this shows up something that was missed.

Martin

[Marianne]

Have another look at vmquery output: Media id 2904L3 is not in the robot:

robot type:            NONE - Not Robotic (0)

[Toddman214]

Under device monitor, where it shows pending requests, there is 1 pending. When I try to drag it up to one of the drives, it says "The request, request ID 14336351, could not be assigned to the selected tape drive: IBM.Ultrium-TD4.001. Unknown drive name (297)."

Marriane, thats interesting, since I'm looking at that media ID in the tape library. Its like somewhere, its media ID isnt matching up?? 

Oh, I ran the vmquery from the master server, which is ONLY a master server and not a library control host. Am I supposd to run that from the media server controling that library?

[Marianne]

Did you run vmquery from production or DR master?

If DR master, the master and media server share the same EMM database that is by default located on the master server. So, it is correct to run vmquery on master server.
tpconfig must be run on media server.

Something seems to be wrong on your DR site. 

Did you run full catalog recovery on DR master?
If so, did you re-config devices (to update with DR-specific details)?
Did you inventory the robot after re-config of devices?

[Toddman214]

Ok, got off the phone with our support vendor, and was trying to follow and type at the same time, so this may not make any sense. We made some adjustments within the media host override under Host properties>general server on the master. There was a backup server and a restore server that were the same so we removed that. We changed the recovery site media server. It was pointing to itself,  and was picking the first one in the list as the default. We then went into VM access hosts section on the master server and added the recovery site host media server there as well.  This allowed the recovery site media server to properly mount the tape. Then, we reran the job, and it has allocated media, and is now restoring. :)