Catalog backup on 5.1 MP2 fails with status 124

[afacey]

In trying to get to 6.5.x on new hw, I have migrated production 5.1MP2 (solaris 8 SPARC) to new hw (solaris 10) as a stepping stone. All backups and restores run as expected. The catalog will not backup sucessfully though.

The paths I have listed are consistant with the guidelines and worked without issue on the old hardware. The only thing that has changed is our catalog filesystem is now NFS mounted.

polaris-T2000# pwd
/opt/openv/netbackup/logs/admin
13:27:54.360 [15340] <2> job_disconnect: Disconnected
13:27:54.374 [15340] <16> bpbackupdb: NB database backup to media id A00054 FAILED
13:27:54.374 [15340] <2> bpbackupdb: executing - /usr/openv/netbackup/bin/dbbackup_notify TAPE A00054 FAIL
13:27:54.441 [15340] <2> get_behavior_rec: ?
13:27:54.463 [15340] <2> LOCAL CLASS_ATT_DEFS: Product ID = 6
13:27:54.466 [15340] <2> get_behavior_rec: MD5 digest line


I have not had a sucessful backup since we migrated to this new HW. The catalog backup runs for about 3 hours and then fails. The size is ~200GB to LTO3 tape
How do I find the path that is failing?

[schmaustech]

Couple questions:

-On the old master, were their media servers as well?  Can the new system access the media severs?

-Do you see any nfs errors in /var/log/messages?

Regards,

Benjamin Schmaus

[John_Stockard]

Try enabling the bpbkar log on the master server.  Since a catalog backup in NBU 5.1 is always a cold backup, it uses the same NBU processes as a normal filesystem backup (bpbkar, bptm, bpbrm).  I have a gut feeling that bpbkar is throwing errors because it doesn't like trying to backup an NFS mount point in this context.

[afacey]

Master and media are same machine (both new and old HW). Not seeing any NFS errors in /var/adm/messages. Strangely though the backup runs for nearly 3 hours and then fails.
I'm going to run another attempt when my fulls complete with golobal logging set to 5 and see what turns up.

[Alex_Vasquez]

I'm not sure how much time I would spend troubleshooting something that isn't patched to where it needs to be.  You're five maintenance packs behind on 5.1.x.  My advice to you, would be to update to the latest maintenance pack, which is MP7.  Then see what you get: http://seer.entsupport.symantec.com/docs/303010.htm

That may not resolve the issue, but at least if you take the case to support you'll have already done the first thing that they would advise. ;)  John's suggestion is also solid and will help you further diagnose the issue.

[afacey]

True - but two things:
1. I migrated from old HW to new; it worked fine on old - no config changes on new HW
2. Before upgrading to 5.1MP7, wouldn't it be wise to have a successful catalog backup?

[Alex_Vasquez]

3. It is wise to have a good catalog backup before upgrading, do you have enough local disk to redirect your catalog backup, rather than using tape?

[John_Stockard]

If you can't get the catalog backup working at NBU 5.1 MP2, you could manually do an equivalent catalog backup, like this:

1. Wait for a time period when there is no activity in your NetBackup environment (no backups, duplications, or restores).

2. Stop all of the NetBackup daemons on your master/media server by running the script "/opt/openv/netbackup/bin/goodies/bp.kill_all".

3. Manually use the Solaris "tar" command (or an equivalent utility) to backup the following directories to a separate tape or a flat tarball file:

/opt/openv/netbackup/db
/opt/openv/var
/opt/openv/volmgr/database

These are the same three directories that get backed up by NetBackup during a catalog backup.  The first path contains your images and media databases, the second path contains your license keys (and a few other things), and the third path contains your volume database and the device configuration.  If you choose to put this manual backup on a tape, pick a tape that you won't be using with NetBackup.  Once a utility such as "tar" writes to the tape, it will destroy the NetBackup header on the tape and NetBackup will refuse to use it (unless you checked the "Overwrite tar" option in the media server properties, and then you'll run the risk of destroying your manual catalog backup with NetBackup backups).

At this point you should be in a good position to upgrade to NBU 5.1 MP7.

You should also be aware that NBU 5.1 reached it's "end-of-support" date back on March 31, 2008.  Symantec is not producing any more patches or maintenance packs for NBU 5.1, and you may get the cold shoulder from Symantec tech support.  I would bet that Symantec tech support will tell you to upgrade to NBU 6.0 or NBU 6.5.x.  You may have to push back at them to get them to help you troubleshoot an issue on NBU 5.1.

[John_Stockard]

Actually, you did make a couple of config changes to your environment when you moved your NBU master/media server from an old server to a new server:

1. The OS changed from Solaris 8 to Solaris 10.
2. The NetBackup catalogs are now hosted on a remote machine via an NFS export.

It's possible that the problem lies more with NBU interacting with the NFS export than with the change from Solaris 8 to Solaris 10 (since Solaris 10 was supported starting with NBU 5.1 MP2).

[afacey]

thanks for your input. The intended goal is to get to 6.5.4, however I must ensure the current system is working properly before attempting an upgrade. Since that is the goal, support is helping to some extent. If it is indeed an issue with NFS, I will most likely proceed as you suggested in making a copy on disk.

I'm running the catalog backup again now with logging set to 5.

Thanks to all that have replied thus far!

[afacey]

Update on issue:

Ended up being an issue with the .snapshots being visable in the /usr/openv/netbackup/db/images dir. NBU was failing trying to reset acces times or related. The fix is to turn off snapshots visabliity at the filer level. Catalogs completed without issue.

Thanks to all.