09-19-2014 12:54 AM
Hi All,
For more than a month now, I have a couple of servers for which the database and redolog backups are continuously failing. The filesystem backup is running without any problem. The bphdb log has the below error message:
Error: Option '--vardir' is not set. Set the option correctly in the flag file '/oracle/GED/112_64/dbs/initGED.utl'.
I have logged a case with Symantec and have been informed that this error is completely new. The case has been escalated to the backline team. In the meantime, I just want to know if anyone has ever faced similar issue and how was it resolved.
Regards,
Jecintha
09-19-2014 01:24 AM
As it is seems to be a SAP parameter, maybe there is something on the SAP community ?
09-19-2014 01:53 AM
Is this SAP backup via brtools or normal RMAN backup ?
What version of Netbackup are you using ?
Can we see the error text ?
09-19-2014 01:57 AM
This is an SAP Backup using brtool. We are using Netbackup 7.5. Please refer to the below detailed status of the job:
9/18/2014 7:10:32 PM - Info nbjm(pid=10640) starting backup job (jobid=188545) for client GDCDEVECC01, policy TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily, schedule Daily
9/18/2014 7:10:32 PM - Info nbjm(pid=10640) requesting MEDIA_SERVER_WITH_ATTRIBUTES resources from RB for backup job (jobid=188545, request id:{DC7B8DA9-0509-4AC8-AA7E-25FE04AA7894})
9/18/2014 7:10:32 PM - requesting resource Any
9/18/2014 7:10:32 PM - requesting resource mumesm01.NBU_CLIENT.MAXJOBS.GDCDEVECC01
9/18/2014 7:10:32 PM - requesting resource mumesm01.NBU_POLICY.MAXJOBS.TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily
9/18/2014 7:10:32 PM - granted resource mumesm01.NBU_CLIENT.MAXJOBS.GDCDEVECC01
9/18/2014 7:10:32 PM - granted resource mumesm01.NBU_POLICY.MAXJOBS.TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily
9/18/2014 7:10:32 PM - granted resource MUMESM01-hcart-robot-tld-0
9/18/2014 7:10:32 PM - estimated 0 Kbytes needed
9/18/2014 7:10:33 PM - started process bpbrm (8300)
9/18/2014 7:10:33 PM - Info nbjm(pid=10640) started backup (backupid=GDCDEVECC01_1411047632) job for client GDCDEVECC01, policy TCS_VSNL_DADAR_GINGER_LOGS_58.91_DEV_Daily, schedule Daily on storage unit MUMESM01-hcart-robot-tld-0
9/18/2014 7:10:34 PM - Info bpbrm(pid=8300) GDCDEVECC01 is the host to backup data from
9/18/2014 7:10:34 PM - Info bpbrm(pid=8300) reading file list from client
9/18/2014 7:10:34 PM - connecting
9/18/2014 7:10:35 PM - Info bpbrm(pid=8300) starting bphdb on client
9/18/2014 7:10:35 PM - connected; connect time: 00:00:01
9/18/2014 7:10:59 PM - Info bphdb(pid=25335) Backup started
9/18/2014 7:10:59 PM - Info bphdb(pid=25335) Processing /usr/openv/netbackup/ext/db_ext/sap_redo_log_backup
9/18/2014 7:10:59 PM - Info bphdb(pid=25335) Waiting for the child status
9/18/2014 7:11:02 PM - Error bpbrm(pid=8300) from client GDCDEVECC01: ERR - Script exited with status = 5 <the restore failed to recover the requested files>
9/18/2014 7:11:02 PM - Error bpbrm(pid=8300) from client GDCDEVECC01: ERR - bphdb exit status = 6: the backup failed to back up the requested files
9/18/2014 7:11:04 PM - Info bphdb(pid=25335) done. status: 6: the backup failed to back up the requested files
9/18/2014 7:11:04 PM - end writing
the backup failed to back up the requested files(6)
09-19-2014 02:42 AM
Have you checked the scripts, utl, etc files end-to-end ie everything thats involved in the process? Did it previously work and then stopped? What about verbose logging?
Not seen it myself.
Jim
09-19-2014 02:52 AM
Yes Jim... The utl and the sap files were verfied. The backup policy name, master and client server details are all updated correctly in the utl file. The verbosity level was set to 5 and this backup did run fine till last month.
09-19-2014 04:39 AM
So thats the only error you see in all the logs? What happened last month: somethings changed...
You could also just copy script/utl/sap files from a known working source and create a new policy.
Or more...uninstall the client and then reinstall from afresh.
Odd it says "The restore..." when its a backup.
Jim
09-19-2014 06:07 AM
Can we see the output from brtools as well ?
09-22-2014 08:52 PM
Last month, there was some activity wherein there was an attempt to install Avamar for DB backup in this server - this was immediately cancelled as it was showing some issue with the Avamar tool. The backup failure has been observed since then. Just to be sure that this is not the cause of failure, I tried to check to which path is the backint refering to and it was found that it is pointing to the path created for Netbackup only.
09-22-2014 08:52 PM
Sure Nicolai. Please let me know the log files required...
09-23-2014 01:28 AM
Go to /oracle/SID/sapbackup/
Attach the last log file - usually with extension *.anf file to a post as a text file.
Please be aware the extension of the log file may be different depending on the SAP backup you defined.
See:
http://help.sap.com/saphelp_nw70/helpdata/en/0d/d30ec54a0c11d182b80000e829fbfe/content.htm
http://help.sap.com/saphelp_nw70/helpdata/en/0d/d30ed24a0c11d182b80000e829fbfe/content.htm
09-23-2014 10:38 PM
Hello Nicolai,
The SAP file has been attached.
09-26-2014 12:45 AM
I think I know what is going wrong. After SAP brbackup/brarchiev has run it uses bplist to verify that Netbackup has all the files. However - if the local bp.conf exist or the master server is wrong in the global bp.conf the bplist operation will fails.
Verify that no bp.conf exist in /oracle
Verify that the .util has the right master server name (if you have multiple NIC at the SAP host - double check)
Verify that /usr/openv/netbackup/bp.conf is correct
Ensure you got VERBOSE = 5 in bp.conf and re-run the operation. Ensure /usr/openv/netbackup/logs/backint exist and has 777 rigts. Attach the debug log file as a file to a post - I this will verify if my theory is correct.
09-26-2014 02:49 AM
The backint and dbclient log files are not getting generated. The bp.conf files have been correctly updated and the utl file also has the details of the master server correctly updated.
09-26-2014 03:02 AM
Can you confirm that the log folders exist in the right place on the client, that the spelling is 100% correct and that folders have 777 permission?
If everything is correct, it means that something very basic is wrong... Like the symbolic link to backint...
09-26-2014 03:23 AM
The path to the log folder is - /usr/openv/netbackup/logs. The folder has 777 permission
drwxrwxrwx 28 root bin 4096 Sep 23 15:40 logs
Log folders:
drwxrwxrwx 2 root root 4096 Sep 16 10:28 backint
drwxrwxrwx 2 oraged dba 4096 Sep 26 00:00 bpcd
drwxrwxrwx 2 iccl2bkp iccl2bkp 4096 Sep 18 15:33 dbclient
drwxrwxrwx 2 oraged dba 4096 Sep 26 14:57 bphdb
09-26-2014 03:37 AM
Please also verify that the symbolic link is still in place:
cd /usr/sap/CER/SYS/exe/run/
ls -l backint
09-26-2014 06:18 AM
You need to check that the parent directories also has read and execution rights for DBA.
For example, if /usr/openv is 750 owned by root:bin dba cant access /usr/openv/netbackup/logs.
09-30-2014 11:51 PM
I have received an update from the Symantec backline team stating that the problem is with the brtool and that a case needs to be logged with the SAP Support.
There seems some problem with the backint file as the disk backup taken without invoking the backint file had got completed successfully.
10-01-2014 01:22 AM
There seems some problem with the backint file....
Sure that is what they said?
backint is a Symantec NetBackup binary and must be linked in the folder that I have mentioned in my previous post.