DB and redolog backups failing with status code 6

[Jecintha]

Hi All,

For more than a month now, I have a couple of servers for which the database and redolog backups are continuously failing.  The filesystem backup is running without any problem.  The bphdb log has the below error message:

Error: Option '--vardir' is not set. Set the option correctly in the flag file '/oracle/GED/112_64/dbs/initGED.utl'.

I have logged a case with Symantec and have been informed that this error is completely new.  The case has been escalated to the backline team.  In the meantime, I just want to know if anyone has ever faced similar issue and how was it resolved.

Regards,

Jecintha

[Michael_G_Ander]

As it is seems to be a SAP parameter, maybe there is something on the SAP community ?

[Nicolai]

Is this SAP backup via brtools or normal RMAN backup ?

What version of Netbackup are you using ?

Can we see the error text ?

[Jecintha]

This is an SAP Backup using brtool.  We are using Netbackup 7.5.  Please refer to the below detailed status of the job:

9/18/2014 7:10:32 PM - Info nbjm(pid=10640) starting backup job (jobid=188545) for client GDCDEVECC01, policy TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily, schedule Daily 
9/18/2014 7:10:32 PM - Info nbjm(pid=10640) requesting MEDIA_SERVER_WITH_ATTRIBUTES resources from RB for backup job (jobid=188545, request id:{DC7B8DA9-0509-4AC8-AA7E-25FE04AA7894}) 
9/18/2014 7:10:32 PM - requesting resource Any
9/18/2014 7:10:32 PM - requesting resource mumesm01.NBU_CLIENT.MAXJOBS.GDCDEVECC01
9/18/2014 7:10:32 PM - requesting resource mumesm01.NBU_POLICY.MAXJOBS.TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily
9/18/2014 7:10:32 PM - granted resource mumesm01.NBU_CLIENT.MAXJOBS.GDCDEVECC01
9/18/2014 7:10:32 PM - granted resource mumesm01.NBU_POLICY.MAXJOBS.TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily
9/18/2014 7:10:32 PM - granted resource MUMESM01-hcart-robot-tld-0
9/18/2014 7:10:32 PM - estimated 0 Kbytes needed
9/18/2014 7:10:33 PM - started process bpbrm (8300)
9/18/2014 7:10:33 PM - Info nbjm(pid=10640) started backup (backupid=GDCDEVECC01_1411047632) job for client GDCDEVECC01, policy TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily, schedule Daily on storage unit MUMESM01-hcart-robot-tld-0
9/18/2014 7:10:34 PM - Info bpbrm(pid=8300) GDCDEVECC01 is the host to backup data from    
9/18/2014 7:10:34 PM - Info bpbrm(pid=8300) reading file list from client       
9/18/2014 7:10:34 PM - connecting
9/18/2014 7:10:35 PM - Info bpbrm(pid=8300) starting bphdb on client        
9/18/2014 7:10:35 PM - connected; connect time: 00:00:01
9/18/2014 7:10:59 PM - Info bphdb(pid=25335) Backup started          
9/18/2014 7:10:59 PM - Info bphdb(pid=25335) Processing /usr/openv/netbackup/ext/db_ext/sap_redo_log_backup          
9/18/2014 7:10:59 PM - Info bphdb(pid=25335) Waiting for the child status       
9/18/2014 7:11:02 PM - Error bpbrm(pid=8300) from client GDCDEVECC01: ERR - Script exited with status = 5 <the restore failed to recover the requested files>
9/18/2014 7:11:02 PM - Error bpbrm(pid=8300) from client GDCDEVECC01: ERR - bphdb exit status = 6: the backup failed to back up the requested files
9/18/2014 7:11:04 PM - Info bphdb(pid=25335) done. status: 6: the backup failed to back up the requested files
9/18/2014 7:11:04 PM - end writing
the backup failed to back up the requested files(6)

[jim_dalton]

Have you checked the scripts, utl, etc files end-to-end ie everything thats involved in the process? Did it previously work and then stopped? What about verbose logging?

Not seen it myself.

Jim

[Jecintha]

Yes Jim... The utl and the sap files were verfied.  The backup policy name, master and client server details are all updated correctly in the utl file.  The verbosity level was set to 5 and this backup did run fine till last month.

[jim_dalton]

So thats the only error you see in all the logs? What happened last month: somethings changed...

You could also just copy script/utl/sap files from a known working source and create a new policy.

Or more...uninstall the client and then reinstall from afresh.

Odd it says "The restore..." when its a backup.

Jim 

[Nicolai]

Can we  see the output from brtools as well ?

[Jecintha]

Last month, there was some activity wherein there was an attempt to install Avamar for DB backup in this server - this was immediately cancelled as it was showing some issue with the Avamar tool.  The backup failure has been observed since then.  Just to be sure that this is not the cause of failure, I tried to check to which path is the backint refering to and it was found that it is pointing to the path created for Netbackup only.

[Jecintha]

Sure Nicolai.  Please let me know the log files required...

[Nicolai]

Go to /oracle/SID/sapbackup/

Attach the last log file - usually with extension *.anf file to a post as a text file.

Please be aware the extension of the log file may be different depending on the SAP backup you defined.

See:

http://help.sap.com/saphelp_nw70/helpdata/en/0d/d30ec54a0c11d182b80000e829fbfe/content.htm

http://help.sap.com/saphelp_nw70/helpdata/en/0d/d30ed24a0c11d182b80000e829fbfe/content.htm

[Jecintha]

Hello Nicolai,

The SAP file has been attached. 

[Nicolai]

I think I know what is going wrong. After SAP brbackup/brarchiev has run it uses bplist to verify that Netbackup has all the files. However - if the local bp.conf exist or the master server is wrong in the global bp.conf the bplist operation will fails.

Verify that no bp.conf exist in /oracle

Verify that the .util has the right master server name (if you have multiple NIC at the SAP host - double check)

Verify that /usr/openv/netbackup/bp.conf is correct

Ensure you got VERBOSE = 5 in bp.conf and re-run the operation. Ensure /usr/openv/netbackup/logs/backint exist and has 777 rigts. Attach the debug log file as a file to a post - I this will verify if my theory is correct.

[Jecintha]

The backint and dbclient log files are not getting generated.  The bp.conf files have been correctly updated and the utl file also has the details of the master server correctly updated.  

[Marianne]

Can you confirm that the log folders exist in the right place on the client, that the spelling is 100% correct and that folders have 777 permission?

If everything is correct, it means that something very basic is wrong... Like the symbolic link to backint...

[Jecintha]

The path to the log folder is - /usr/openv/netbackup/logs.  The folder has 777 permission

drwxrwxrwx 28 root bin   4096 Sep 23 15:40 logs

Log folders:

drwxrwxrwx 2 root     root      4096 Sep 16 10:28 backint

drwxrwxrwx 2 oraged   dba       4096 Sep 26 00:00 bpcd

drwxrwxrwx 2 iccl2bkp iccl2bkp  4096 Sep 18 15:33 dbclient

drwxrwxrwx 2 oraged   dba       4096 Sep 26 14:57 bphdb

[Marianne]

Please also verify that the symbolic link is still in place:

cd  /usr/sap/CER/SYS/exe/run/

ls -l backint

[Nicolai]

You need to check that the parent directories also has read and execution rights for DBA.

For example, if  /usr/openv is 750 owned by root:bin dba cant access /usr/openv/netbackup/logs.

[Jecintha]

I have received an update from the Symantec backline team stating that the problem is with the brtool and that a case needs to be logged with the SAP Support.  

There seems some problem with the backint file as the disk backup taken without invoking the backint file had got completed successfully.

[Marianne]

There seems some problem with the backint file....

Sure that is what they said?

backint is a Symantec NetBackup binary and must be linked in the folder that I have mentioned in my previous post.