Really awkward catalog recovery

[KG-Airbus]

Hi, bit of a complicated situation here that I need some assistance with, so I'll layout the setup first:

1) Old Master Server, has no access to tape devices and no longer has any media servers,

2) New Backup environment with access to tape libraries

3) Old Master loses entire /usr/openv directory (due to disk array failure), one final Catalog backup tape available,

Now, the usr/openv partition was recreated (this is a Solaris master server, if it matters) and I reinstalled Netbackup as it was previously (as near as I can tell). Now all I need to do is a Catalog recovery, but how, without access to any tape devices.

I made the old master a client of the new Master and a media server (put their names in the bp.conf as SERVER and MEDIA_SERVER entries), communication is fine. I had intended to use bprecover on the media server to do the catalogue recovery but I can't be sure that it won't just overwrite the catalog on the New Master server, which absolutely CANNOT happen, I would be fired!

I have the DR file for the old Master's catalog backup and I've edited to change references to the media server that made the backup (which no longer exists) to the new media server that I want to run the recovery, however I can only explicitely reference this DR file if I use the wizard or the GUI, neither of which will actually tell me which server they're going to write the catalog back to before they actually start. I though I might be able to use the -d <destination host> option in bprecover to do that but then you don't get the option of specifying the DR file (obviously the new backup environment has its own DR file and I don't want it using that).

So I'm reading the Troubleshooting Guide and it mentions that because the Catalog Backup is written in standard Netbackup format you can do a normal restore, you just need to select the backup type as NBU-Catalog (page 565). So I import the tape, select the old Master as the source and destination, set the backup type as NBU-Catalog and off we go, an hour later it's restored all the files in /usr/openv (that appear in the Catalog backup) successfully (I told it to overwrite everything), so I restart the services on the old Master and... nothing. It restarts without any apparent errors, it's just completely ignoring the restored Catalog and presents a vanilla installation, no media, no volume pools, nothing (the reason I need this restored is because I need to know when the media written under the old environment expires, so I can recall it from offsite storage and reuse it in the new environment).

Clearly I've missed something but I've no idea what, any ideas? I do have a final Full system backup (which includes everything on the old server) which I could restore as well but I haven't tried that yet, obviously I'd have to re-run the Catalog restore after I'd done that.

[mph999]

Hi,

Yes, you can treat the catalog tape as a normal backup, but you cannot restore the catalog in this way.  The catalog restore does more than just copy the data off the tape (there is an SQL database involved).

You can't just overwrite the /ur/openv/... folders from a catalog tape (due to the SQL db) but , you could just copy back /usr/openv/netbackup/db/images/* - you will then have browseable backups, and the retention of each image will be found from the header file, but, you will not have the bpmedialist output available (this is inthe EMM db).

You can't run a restore over the top as I've mentioend, as the DB won't start.

Martin 

[Mark_Solutions]

Have you checked that it has actually restored the files?

Anything in the images directory etc?

There is a lot more to the catalog than just the flat files (EMM Database etc.) so you may need PS help for this

[mph999]

Hi,

Yes, you can treat the catalog tape as a normal backup, but you cannot restore the catalog in this way.  The catalog restore does more than just copy the data off the tape (there is an SQL database involved).

You can't just overwrite the /ur/openv/... folders from a catalog tape (due to the SQL db) but , you could just copy back /usr/openv/netbackup/db/images/* - you will then have browseable backups, and the retention of each image will be found from the header file, but, you will not have the bpmedialist output available (this is inthe EMM db).

You can't run a restore over the top as I've mentioend, as the DB won't start.

Martin 

[KG-Airbus]

Hi,

> Have you checked that it has actually restored the files?

> Anything in the images directory etc?

Yes, sorry, should have mentioned that, there's 150Gb in there now that wasn't there before! 

> There is a lot more to the catalog than just the flat files (EMM Database etc.) so you may need PS help for this

Curses, was trying to avoid that. I'm currently importing the last Full system backup, so I might try that first and see how I get on.

[Mark_Solutions]

OK - keep us updated - and you know where we are if you need us Kristian - just give Dave H a ring.

[KG-Airbus]

Ok cheers, it's not actually Kristian though, I'm just borrowing his login because it's his name on the contract! 

[Marianne]

Please see this TN:

DOCUMENTATION: How to recover a NetBackup catalog from tape if the original media server which did the backup is not available.

http://www.symantec.com/docs/TECH48819

Please double-check and confirm the following:

NBU master server hostname must match the original hostname.

NBU must be installed in same location. The default on Solaris master is to install in /opt and create a symbolic link to /usr.

Catalog backup is made with actual file location, not the /usr/openv path.

A progress log is written during catalog recovery:

/usr/openv/netbackup/logs/user_ops/root/logs/Recover…….

Use this log to troubleshoot any errors.

[KG-Airbus]

Hi, as I mentioned, I'm not comfortable using that method as all I have available is a media server from a different Netbackup environment (ie it belongs to a different master server) and unless someone can categorically tell me that the media server will not overwrite its own Master server's catalog with that of the old master server then I just can't take that risk and attempt it unfortunately.

The hostname hasn't changed, NBU is installed in /usr/openv which is a separate partition, there's a link in /opt to /usr/openv, rather than the other way around, so the actual file location IS /usr/openv. I don't have a "Recover..." log because I didn't use bprecover, like I said, I used the standard restore procedure from the GUI and changed the backup type to NBU-Catalog.

[Marianne]

I just read through your initial post again. The restore that you've done only restored images, policies, etc ( netbackup/db/ )  - not the relational database that contains media, pool, etc info.

You NEED a tape drive!

Beg, borrow, buy a stand-alone drive.

Or, if possible, attach one tape drive in the existing library to the old master. DOWN it on all media servers in existing environment.

Manually insert catalog tape in the drive to run bprecover.

[mph999 · Accepted solution]

or .. this might work ...

Here I run /usr/openv/netbackup/bin/tar xvf on my Catalog tape ...

root@womble tmp $ /usr/openv/netbackup/bin/tar xvf /dev/rmt/1cbn

Blocksize = 128 records

/

Removing leading / from absolute path names in the archive.

/PROXY_BACKUP

/

/netbackup/

/netbackup/nbu7102/

/netbackup/nbu7102/openv/

/netbackup/nbu7102/openv/db/

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/DARS_DATA.db

/usr/openv/db/staging/DARS_DATA.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/DARS_INDEX.db

/usr/openv/db/staging/DARS_INDEX.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/DBM_DATA.db

/usr/openv/db/staging/DBM_DATA.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/DBM_INDEX.db

/usr/openv/db/staging/DBM_INDEX.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/EMM_DATA.db

/usr/openv/db/staging/EMM_DATA.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/EMM_INDEX.db

/usr/openv/db/staging/EMM_INDEX.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/NBAZDB.db

/usr/openv/db/staging/NBAZDB.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/NBAZDB.log.1

/usr/openv/db/staging/NBAZDB.log.1

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/NBDB.db

/usr/openv/db/staging/NBDB.db

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/NBDB.log.1

/usr/openv/db/staging/NBDB.log.1

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/databases.conf

/usr/openv/db/staging/databases.conf

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/server.conf

/usr/openv/db/staging/server.conf

/netbackup/nbu7102/openv/db/staging/

/netbackup/nbu7102/openv/db/staging/vxdbms.conf

/usr/openv/db/staging/vxdbms.conf

 cd /netbackup/tmp/netbackup/nbu7102/openv/db/staging

 ls

 DARS_DATA.db    databases.conf  DBM_INDEX.db    EMM_INDEX.db    NBAZDB.log.1    NBDB.log.1      vxdbms.conf

 DARS_INDEX.db   DBM_DATA.db     EMM_DATA.db     NBAZDB.db       NBDB.db         server.conf

Here I will - 'break my EMM db

root@womble staging $ mv /usr/openv/db/data /usr/openv/db/data.orig

I then copied the DB files back cp /usr/openv/db/data 

Job done ...

[KG-Airbus]

Unfortunately a standalone tape drive will be difficult and connecting the server to the existing backup SAN will require us to go through a whole lot of change control which will delay things by weeks. If it becomes the last resort then I'll have to do that I guess but I'll exhaust all other options first!

[mph999]

As per my post above - you can untar the tape on an exisiting server ... USe /usr/openv/netbackup/bin/tar command and it will drop the leading / , so will become a relative path.

Copy the files in the xxx/openv/db/staging dir back to the old server.  STop NBU first and mv the files in /usr/openv/db/data , then copy the 'recovered' files into there - worked for me.

Regards,

Martin

[mph999]

How are you getting on with this ?

Martin