Catalog backup failing with 87 using DD

[PhaniKumar82]

Team

Need solution on the below issue 

Netbackup CATALOG_DRIVEN_BACKUP is failing with status 87 using datadomain but same time normal backups using same DD storage unit is completing successfully 

Master server : Netbackup 7.1.0.4 on Windows 2003 

Media server : Master server act as media server itself as this is remote master server 

Please let me know how to proceed with the above

Regards,

Kumar 

[mph999]

The writing part of the catalog backup is not really any different than a normal backup, the 'clever' bit of the catalog backup is performed before the write is made.

There is one difference however which you probably don't have sent on you regular backups, and that is the TIR option is set.

I would suggest enabling the TIR option on a regulat policy (set on the attributes screen) and run a test - this will narrow the issue down, if the regular backup than fails.

Additionally, we would need the logs from the catalog backup - bptm, bpdm, bpbrm, bpbkar and bpcd 

M

[PhaniKumar82]

Martin i have attached bpdm bpbrm bpcd and bpbkar logs Please let me know if you need any other information Regards, Kumar

[mph999]

Can you also look at the logs on the datadomain - I suspect it might log something (apologies, I am not too famliar with DD so cannot really advise on which logs).

For info, this is how to set logs/ change verbose levels - see part 2.

Setting up logs in NetBackup

For a given issue, it may be necessary to gather multiple logs.  This MUST cover the time the issue happens.

If an additional log is required, that has to be created, then ALL the logs must be supplied again.

There are two types of logs in NetBackup.  Legacy logs and VX logs.

1) Creating Legacy Logs

2) Setting Verbose Level for Legacy Logs

3) Collecting Legacy Logs

4) Creating VX Logs and setting the log level

5) Collecting VX logs

6) Volmgr Logs

1) Creating Legacy Logs

************************

These are created in either 

Unix

/usr/openv/netbackup/logs/

/usr/openv/volmgr/debug/

Windows

<install path>\veritas\netbackup\logs

<install path>\veritas\volmgr\debug

For example to create bptm log, simply create a directory called the <process> name.

mkdir /usr/openv/netbackup/logs/bptm

A newly created log will not log anything /detect a change of verbose level until the process is restarted.  For logs such as bptm, this will be when the next backup runs.  Other logs such as bprm and bpdm may require a restart of the NBU services.  I say 'may', if the process starts a child process, then this would write to a newly created log or pick up a verbose level change.

2) Setting Verbose Level for Legacy Logs

****************************************

There are two ways this can be done :

(Unix) 

To increase the verbose level of all logs (except vault)

Add the entry VERBOSE = <level> into /usr/openv/netbackup/bp.conf.  <level> is a value between 0 and 5, with 5 being the highest.

(Windows)

On the server you are gathering the logs from, run the BAR GUI

From the File menu, select Client Properties and in the pop-up window, goto the Troubleshooting tab.

Set General to 2 and Verbose to 5

3) Collecting Legacy Logs

*************************

The log file is simply found in the <process> name directory.  There is one log per day.

The name of the log file will be log.<date>

If you are sending multiple log files in, they will all have the same name.  Please therefore rename the log files to :

<process>.log.<date>

4) Creating VX Logs and setting the log level

**********************************************

These are more complex, and have to be set with specific commands.  NOTE: Some of these logs, for example, 'mds' do NOT create a log file.  Instead the lines are entered in to other log files.  In the case of mds (143), it logs into EMM (111).

The vxlogs cover various processes, for example, nbemm, nbrb, nbjm, nbrb, mds

To set these up on either Unix or Windows, use this command :

vxlogcfg -a -p 51216 -o <oid> -s DebugLevel=<1-6> -s DiagnosticLevel=<1-6>

For example, to set the EMM and MDS logs to levels 6 and 6 use

vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -a -p 51216 -o 143 -s DebugLevel=6 -s DiagnosticLevel=6

To confirm the log level has been set, simply look in the nblog.conf file, which is located in the netbackup diorectory.

(NOTE: DO NOT EDIT THIS FILE MANUALLY)

5) Collecting VX Logs

*********************

To collect the vx logs, use the nbcplogs command.  This copies the raw logs, which is the preference of Technical Support.

NOTE:  The destination directory MUST be empty.

nbcplogs --no-nbsu -d 2hrs --logs nbemm,nbjm /tmp/logs   (Ex.  Coleect the past 2 hrs of logs, RELATIVE, to when the command is run )

nbcplogs --no-nbsu -s 07/11/2012-10:17:58 -e 07/11/2012-12:17:58 --logs nbjm,nbpem /tmp/logs (Collect the logs between two times -s <start> -e <end>  )

In these examples, the nbpem and nbjm logs would be copied to /tmp/logs

For details of using vxlogview please see TN:  http://www.symantec.com/docs/TECH75805

6) Volmgr Debug Logs

********************

These are very similar to legacy logs, the difference being the location and the verbose setting.

There is no value for verbose level, simply it is turned up by adding the work VERBOSE to a line in the vm.conf file.

To turn on Media Manager logging:

UNIX:

Add VERBOSE to the /usr/openv/volmgr/vm.conf file.  If this file does not exist, just create it.

If necessary, create the directory /usr/openv/volmgr/debug

mkdir /usr/openv/volmgr/debug/acssi

mkdir /usr/openv/volmgr/debug/acsd

mkdir /usr/openv/volmgr/debug/robots

mkdir /usr/openv/volmgr/debug/daemon

mkdir /usr/openv/volmgr/debug/ltid

mkdir /usr/openv/volmgr/debug/oprd

mkdir /usr/openv/volmgr/debug/reqlib

mkdir /usr/openv/volmgr/debug/tpcommand

The following empty files increase the details logged further

touch /usr/openv/volmgr/DRIVE_DEBUG

touch /usr/openv/volmgr/ROBOT_DEBUG 

touch /usr/openv/volmgr/AVRD_DEBUG 

touch /usr/openv/volmgr/SSO_DEBUG 

Restart ltid :

/usr/openv/volmgr/bin/stopltid

/usr/openv/volmgr/bin/ltid -v

Windows:

Add VERBOSE to the <install path>\veritas\volmgr\vm.conf file.  If this file does not exist, just create it.

If necessary, create the directory /usr/openv/volmgr/debug

<install path>\veritas\volmgr\debug/acssi

<install path>\veritas\volmgr\debug/acsd

<install path>\veritas\volmgr\debug/robots

<install path>\veritas\volmgr\debug/daemon

<install path>\veritas\volmgr\debug/ltid

<install path>\veritas\volmgr\debug/oprd

<install path>\veritas\volmgr\reqlib

<install path>\veritas\volmgr\debug\tpcommand

Create the following empty files to increase the details logged further

Ensure that windows does not craete a 'suffix', for example .txt

<install path>\veritas\volmgr\DRIVE_DEBUG

<install path>\veritas\volmgr\ROBOT_DEBUG 

<install path>\veritas\volmgr\AVRD_DEBUG 

<install path>\veritas\volmgr\SSO_DEBUG 

Restart ltid :

<install path>\veritas\volmgr\bin/stopltid

<install path>\veritas\volmgr\bin/ltid -v

[mph999]

Missing the bptm log (and bpdm was empty, though there might not be anything logged in here).

Can you create the bptm dir, set the verbose level to 5 and general to 2, and repost the logs.

One really important question - did this ever work ?

Is this a big catalog - how long do you think the catalog backup should run for - are we taking a 1 or so or many hours ?  Just need an estimate.

Thanks,

Martin

[PhaniKumar82]

Martin We dont have any tape backups in our environment do you still need bptm logs ? Regards, Kumar

[mph999]

Hmm, seems that there can be many causes of 87 on disk - I've been looking through a few previous cases.

Network issues seem quite common

Time out issues (but this was on a netapp)

Disk failure

However, the one over riding factor, which is making this a bit confusing is you only report the issue on catalog backups, so it would appear something that is different about this type of backup.

The only thing I can think of (as mentioned) that is probably different is that TIR is always enabled for a catalog backup (hence my request to test a 'normal' backup with this enabled.

The other possibe difference of a catalog backup, is that it is backing up (part of the master) (other backups are probably of different clients) and the length of the catlog backup could be very long compared to others.

These are just a few ideas, and odd as it seems, sometimes these sort of things are a cause or contributing factor.

Is there anything on the datadomain that is unique for the catalog backup - apologies, not familar with these, but my thoughts are if the catalog backups use a different 'location' of the DD (if this is possible ?).

M

[mph999]

Yep, bptm also writes to disk ...

TBH, I'm not convinced that we going to get much more info than we have from the details you have posted, but we should look.

Most imporantly are the questions I asked.

Has this worked in the past

If it did work previously, what has been changed

What happens if you enable TIR on a regular policy

How big is the catalog backup 

Anything special on the DD

Thanks,

Martin

[Nicolai]

Can you post the detailed job text from the activitry monitor ?

Have you tried to delete the catalog backup policy and re-create it ?

[mph999]

Right I've got my brain in gear tonight re datadomains ...  The DD has luns mapped to diskpools in NBU, are you using a separate diskpool for the catalog backup ?

Thanks,

Martin

[William_Jansen_]

Hi As far as I know TIR is not yet supported on NBU and DD(need 2.6 ost which is GA and 5.3 DDOS that isn't GA yet). Sonif you have it configured, turn it off and try again. Like martin said, check the STU you selected and possibly create a new catalog backup policy and see if that works.

[mph999]

You can't turm TIR off on Catalog backups, it's always on.

M

[William_Jansen_]

Sorry my bad, I interprited TIR as True Image Replication. The stuff that goes with AIR. Guess I have some reading to do in the manuals :)

[Marianne]

As from NBU 6.5, ALL backups log to bptm - disk and tape.
bpdm only logs disk cleanup info.

So, yes - we need bptm log.