Help with Disk Staging and Error 191

[jwmcf1]

I am in the process of moving from using tape to using external hard drives for my offsite copies of back ups.  I have several Oracle jobs that are backed up to a Disk Storage Unit.  Disk staging is enabled on this DSU and has been working great to duplicate the jobs to tape for our offsite copies.

Last week I switched from tape to USB external disk.  What I did was edit the staging location for the DSU of the jobs.  In addition I pointed the DSU to a new folder location and copied the backups from the origional location to this new location which has a new name.

After doing so disk staging is no longer working correctly.  The staging starts on schedule but I now get 50 or so job ID's for duplication that are all failed with status of 191. The duplicate jobs all start within seconds of each other. THis is then followed by hundreds of image cleanup jobs (which are succesful).  After this another batch of 50 or so duplication jobs will start and fail with 191 followed by another hundred or so image cleanup jobs.

Before the switch we would have 3 activty monitor entries 1. the start of the DSSU disk staging job, 2 and 3 were duplication jobs.  That was it.

This is a snippet of the duplication logs.

7/31/2012 2:00:58 AM - Critical bpdm(pid=8232) image open failed: error 2060013: no more entries    
7/31/2012 2:01:15 AM - Error bpduplicate(pid=10200) host nbuhost backup id oracle1_1343189809 read failed, media open error (83). 
7/31/2012 2:01:15 AM - Error bpduplicate(pid=10200) host nbuhost backupid oracle1_1343189809 write failed, termination requested by administrator (150). 
7/31/2012 2:01:17 AM - Error bpduplicate(pid=10200) Duplicate of backupid oracle1_1343189809 failed, termination requested by administrator (150).  
7/31/2012 2:01:17 AM - Error bpduplicate(pid=10200) Status = no images were successfully processed.     
7/31/2012 2:01:17 AM - end Duplicate; elapsed time: 00:20:57

I have a feeling pointing the DSU to a new location is what has caused this, but could use some assitance.

thanks

[mph999]

If I have followed your explanation correctly, you have moved the backup images, but the backup headers are now pointing to the old location.

Try this :  Take one backup id that it cannot find.

nbcatsync -backupid <backupid> -dryrun

Hopefully, this will correct the location, we will see from the output if it is able to make a change

If you run bpimagelist -backupid <backid> this will show the fragement lines and the location can be seen.

Before making any changes run:

nbrb_backup -online <dir>    Create <dir> before you run it.  I think it is in netbackup\bin

You need to do this with NO backups running, else if you have to restore the EMM backup, you will be out of sync with the image DB, which will give more issues.

Martin

[jwmcf1]

I will try your suggestions now.  As you responded I was writting this which explains what I did in detail:

*******************

Ahh jeez, Ok, so in my research it is looking like I made a mistake by modifing the DSU directory pathname and then moving the images by simply copying them to the new location using windows copy. 

NBU and DSU's are on a windows server

This is what I did: 

1. Created a new folder on the windows system

2. Opened the DSU in the NBU gui and changed the "Absolute pathname to directory" to the new folder

3. Copied all of the NBU images from the old location to the new location using windows copy

4. lastly I deleted the old folder location

5. The next backup that ran caused the 191 issues

6. I then created a new DSU pointing to the "absolute pathname .." I had just created.  Enabled disk staging for this new DSU

7. Changed the backup job to use this new DSU

8.  Same issue is occuring

I'm guessing this was a mistake to do - Correct?  How can I correct the situation?  Should I expire all images and then start new?  I don't want to loose the current backups though.

thanks

**************************************

As I stated I will go trhrough your suggestions and report back.  If this new info provides more insight please let me know

thanks

[jwmcf1]

It appears that I do not have nbcatsync on my netbackup server. It is not in nebackup\bin or bin\admincmd.

Searched server and nothing is found.

Netbackup version is 7.0

[mph999]

Hmm, if you run a backup to this 'new' location, forget staging, does the backup work.

I was thinking it is the staging of the previously backed up images that fail.

If it is the duplication that is failing, try my caommand - as -dryrun, it will only report what it wants to do.

Martin

[mph999]

The mistake was coping the original backup images.  It would have been better to duplicate them.

I have to go out, I'll check back later.  Marianne will prbably be along in a bit and she or one of the others will help out in my absense.

Martin

[jwmcf1]

Yes the backups do run and there are new image files created in the new location.  I will run another job with duplication turned off to verify, but looking at last nights job there are many image files that were created as they normally would.

Also if I check my DSSU staging location (which is my USB drive) there are infact img files that were written but many more than would be expected.  The total size of the DSU files is 20GB and the DSSU has 90GB of data.

Also, I responded above - nbcatsync does not work and I can't find it on my system.

[jwmcf1]

ok thanks

[jwmcf1]

Unfortuneately nbcatsync is nowhere to be found on the system.  I don't know how or why, but it's not there.  Not in Nebackup\bin  did a search of all drives and windows can not find the file.

[mph999]

It should be there somewhere ...

From the google 'man nbcatsync'

http://www.symantec.com/docs/HOWTO43774

Name

nbcatsync — run a utility that resyncs the disk media IDs in the image catalog after catalog recovery operations complete

SYNOPSIS

nbcatsync -backupid catalog_backup_id [-prune_catalog] [-no_sync_slp] [-dryrun] [-keepgoing]

nbcatsync -sync_dr_file dr_file_path [-copy number] [-dryrun]

On UNIX and Linux systems, the directory path to this command is /usr/openv/netbackup/bin/

On Windows systems, the directory path to this command is<install_path>NetBackup\bin\

If it really is missing, then you might have to log a call to sort this out, by manually editting the files.

Really have to dash now, running late ...

Martin

[mph999]

OK, thank you for looking - you could copy it across from another system if you have one at the same version.

One thing I am unsure of is when it came in. It's not in the 7.0 command ref guide, but it is in the 7.1 guide. It could be that this is just a documentation thing.

I suspect that the command was just not there at 7.0, I doubt it has been deleted.

Looks like a call I am afraid - sorry.

Editting catalog files is not really for the forum, it is something we'll do in support if need be, but is 'unsupported'.

Martin

[CRZ]

Sorry, I can confirm the bad news:  nbcatsync originally came out for 6.5.6 AFTER 7.0 was out.  First 7.x release was 7.0.1.

(Why are you still at 7.0GA?)

[jwmcf1]

Well rats. 

So before I call NBU support, can I ask what is Option B?   Before nbcatsync (7.0 and prior) if something like this happened what is the fix?

[Marianne]

I'm guessing this was a mistake to do - Correct? 

Correct.... Best to ask for advice before making major changes......

How can I correct the situation?  

Upgrade/patch ASAP, then use nbcatsync. 

Another option would be to expire images (-justimage) and then import.
nbcatsync will be easiest, even if it means you need to patch/upgrade first.

[mph999]

Thx Chris , that solves that little puzzle ...

Before catsync, well it would depend ...

Manual edit of the catalog files or revert the system to how it was.

If you do log a call, please show them this thread, I'm one of the UK engineers.  Don't want anyone going off in a different direction.

If it was me, I'd put the .01 patch on and go with nbcatsync.

Martin

[jwmcf1]

If I upgrade to 7.5 will my 7.0 clients (Linux clients backing up Oracle) still be operational with the new version?

thanks

[Marianne]

See Version Compatibility TN: http://www.symantec.com/docs/TECH29677

[jwmcf1]

thanks everyone

[mph999]

Your clients will be fine at 7

Are you all fixed now ?

If so, would you be generous enough to mark the post that helped you the most as the solution.

Regards,

Martin

[jwmcf1]

Not fixed yet.  My plan is to update to 7.5 and try your initial suggestion.  Was thinking I would have to delay the upgrade and would close the thread, but I may have time tomorrow to update. 

How long does a typical upgrade take for a standalone server?   I'm assuming an 1hr, but maybe its more involved than just clicking through setup.  Looking through upgrade guidance docs now.