Restore Netbackup from catastrophic failure of Master Server

[taylorm]

I am trying to recover my Netbackup and system data (catalog db and system partitions) from my tape library. My master server (UNIX) crashed and was unrecoverable. Reinstalled sytem to original partitions and need to recover from tape. I cannot figure out how to recover my Catalog DB. I am using NBU 6.0. Any help would be appriciated.

[Andy_Welburn]

Check out the NetBackup 6.0 Troubleshooting Guide for UNIX & Windows:

http://ftp.support.veritas.com/pub/support/products/NetBackup_Enterprise_Server/279295.pdf

There are various sections on 'Disaster Recovery'.

Good luck.

[Mr__Ship]

If you don't have a copy handy, download a copy of the NBU 6.0 Troubleshooting Manual from the Symantec Support Website. Chapter 7 deals with Disaster Recovery of the Master Server. The first recommended step is that you open a support call with Tech Support.

Here's a link to the Manual:

http://ftp.support.veritas.com/pub/support/products/NetBackup_Enterprise_Server/279295.pdf

[taylorm]

Andy,

Thank you for your post. I have read the documentation, and I am trying to work my way through it. Just looking for any tips or tricks that might help.

[zippy]

Make sure you have enough space in /usr or create a new mount and softlink it. 

I hope you have the DB backup tapes on hand.

You use the 'bprecover' command to restore the NetBackup database.
NetBackup can mount the tape if the tape is in a robot and enough of the
NetBackup database still exists so that NetBackup knows the tape is in the robot.
In that case, you would specify the media ID and media type (density) and
the tape would be mounted. For example:
bprecover -ev b00003 -d 8mm

If NetBackup does not know where the tape is or cannot mount it, you
would manually mount the tape and specify the device path. For example:
bprecover -tpath /dev/rmt/0cbn

Chapter 5 of the Troubleshooting Guide explains disaster recovery
procedures.

Here is some more information about the 'bprecover' command.

# which bprecover
/usr/openv/netbackup/bin/admincmd/bprecover
# bprecover -help
bprecover: -l -ev media id -d density [-v]
           -l -tpath <raw_tape_device_path> [-v]
           -l -dpath <disk_device_path> [-v]
           -l -opath <optical_device_path> [-v]
           -r [ALL|image_number] -ev media id -d density [-stdout]
[-dhost
<dest
ination_host>] [-v]
           -r [ALL|image_number] -tpath <raw_tape_device_path>
[-stdout]
[-dhost
<destination_host>] [-v]
           -r [ALL|image_number] -dpath <disk_device_path> [-stdout]
[-dhost
<de
stination_host>] [-v]
           -r [ALL|image_number] -opath <optical_device_path> [-stdout]
[-dhost
<destination_host>] [-v]
#
#
# bprecover -l -ev b00003 -d 8mm
Database Backup Information from B00003

Created:      02/06/02 18:08:30
Server:       megatron
Block size:   32768

            Path
            ----
IMAGE1      /usr/openv/netbackup/db
IMAGE2      /usr/openv/volmgr/database
IMAGE3      /usr/openv/var
#
#
# echo $MANPATH
/usr/share/man:/usr/share/man/man*:/usr/share/man/sman*:/usr/man:/usr/openwi
n/ma
n:/usr/local/man:/usr/openv/man/share/man
#
#
# man bprecover
Reformatting page.  Please Wait... done

bprecover(1M)
bprecover(1M)

NAME
      bprecover - recover selected NetBackup related catalogs

SYNOPSIS
      /usr/openv/netbackup/bin/admincmd/bprecover [-v]

      -l -ev media_id -d density [-v]

      -l -dpath disk_path [-v]

      -l -tpath tape_device_path [-v]

      -l -opath optical_device_path [-v]

      -r [all | ALL | image_number] -ev media_id -d density [-stdout]
[-dhost
           destination_host] [-v]

      -r [all | ALL | image_number] -dpath disk_device_path [-stdout]
[-dhost
           destination_host] [-v]
<snip>
#

The manpage for 'bprecover' is also in Appendix A of the System
Administrator's Guide.

[taylorm]

Ok, so I have run the bprecovery command on every single tape in my library and have not found the catalog backup. I keep getting this "media or file was not created by bpbackup db. specified media or path does not contain a valid nb database backup header" I know that one of the tapes was a catalog backup. I cannot get NBU up all of the way, I am using just a partially installed version at this point. Is there a requirement for processes that I haven't found?

[zippy]

taylor,

importing all your tapes is, well is very painfull, you should have kept track of your Database backup tapes.  I email the list to me every day, rcp a copy to other servers, every day.

I make an Ignite, jumpstart, baremetal copy of my master and know where my DB tapes are at all time.

restore the server first then restore the veritas DB.

How can you restore your enviroment if you can restore your Master Backup server, yaaaa scarry thought.

what I email to me and rcp to other servers, every day.

Frequency of DB Backup:  after each successful backup schedule

  Server: Hostname_of_Master_server_that_has_the_DB_on_it!

  Sequence # 1    Last Media Used: 000917

    Written             Allocated           Type   Density  Media

    ------------------- ------------------- ----   -------  -----

  1 04/16/2008 10:18:27 05/19/2005 16:15:33 RMedia hcart    000917

  2 03/15/2008 09:17:05 01/04/2005 18:29:18 RMedia hcart    000886

  Paths Included:

    /veritas/openv/netbackup/db

    /veritas/openv/volmgr/database

    /veritas/openv/var

[Stumpr2]

I run rsync on the images directory 3 times a day from production to DR

[zippy]

heck if it the data was on a file system that was on... say an arrary or a volume group that was configured on seperate disk system, you could re-import it....

rsync, man i haven used that in a long time, good idea though.

[taylorm]

I appriciate all of the commnets of better ways to do it, but I am stuck in my Government configuration and am not allowed to change it. Trust me I would have built the system much differently!! The bottom line is I lost my raid and my system disk and now I have a bunch of tapes and cannot seem to do anything with them. This is very frustrating not being able to find my catlog to restore from!!!!

[Stumpr2]

Did you open a case with symantec?

[taylorm]

No,

I dont have a support contract anymore. I DO love the government

[Stumpr2]

I would post to the auburn group

http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

[taylorm]

Thanks,

I just subscribed.

[Stumpr2]

you'll like it. That is the place I go to when I have questions.

[Russ_Lyons]

Do you have the Catalog Backup tapes?? IF so, recovery is relatively easy. Done this with our DR testing last month.

I can send you a brief doc on what was done.

[taylorm]

Russ,

I have found my catalog tapes. Your documentation would be very helpful if you could send it.

[Lobacevskij]

Hi Russ, I'm going to prepare a DR plan for my company and writing a procedure to restore our Master Server. If you have a draft and if you can send me a copy I will appreciate. I have a Catalog Backup Tape and our Master Server is a RedHat ES4. Thanks in advance, Luca.

[jim_dalton]

Heres what you do: rebuild a server with the same Os and patch as the dead one; layout the disks in order to accomodate the db restore (or just make one huge partition: if everything wont fit , then you need more disk: you need to know how much space was used on the original); install and patch netbackup - dont configure it; attach all your devices, robots etc;get your latest dump tape and put it in an attached tape drive and brrecover -r -tpath /dev/rmt/xxx all files and directories that came from the same server. If you failed to backup everything you have a headache; if youve got the dbs then youre mostly ok. Then trash the storage units, robots etc etc and reconfigure. Done. It works well but theres preparation needed.

Alternatively (this also works) : if youve got a ufsdump or similar, boot your server off alternative media/disk, format up your target disk(s) to receive the dump, ufsrestore the partitions, make bootable, change to be the default boot device, boot. Done. Works very well under Solaris and theres no need to apply brrecover. You will need to reconfigure your devices as youve just restored the original ones: thats a bit messy but is do-able.

Message Edited by tim walton on 07-09-2008 04:25 AM

Message Edited by tim walton on 07-09-2008 04:28 AM