Rman backup is working manually, but not working by netbackup. Cuz of zone??

[turguns]

Hi all,

Environment:

master,media server  (Redhat) Netbackup 7.1.

Client Solaris 10 sparc, Zone (virtual)

We have an issue with oracle backup.

When we start the policy getting the error : the backup failed to back up the requested files

But when we start the backup with rman ,backup is working properly.

What is the source of problem? Did u have similar problem?

P.S.Standart backup is working fine.

May be it is because of  client is on solaris zone??? We need smth change on config file?

With BR,

Turgunbek

[RiaanBadenhorst]

If a script is being used, the RMAN output file is, by default, in the same directory as the script, and is named script_name.out.  The output file can be redirected to other locations in the RMAN script.  If the output fine is NOT in one of these locations, review the script for the current location of the output file.

Post the output of the rman script as ran from NetBackup to see what the issue is.

[turguns]

output is :

==== started on Mon Jun 2 11:55:48 GET 2014 ====

RMAN: /home/ora10/geo/bin/rman
CMD: /home/ora10/geo/bin/sqlplus /nolog
ORACLE_SID: geo
ORACLE_USER: ora10
ORACLE_HOME: /home/ora10/geo
NB_ORA_FULL: 1
NB_ORA_INCR: 0
NB_ORA_CINC: 0
NB_ORA_SERV: nbumedia
NB_ORA_POLICY: test_cboss_offline
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
You have mail.
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
You have mail.

Full backup requested
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
You have mail.

Script /oracle/arch/rman_scripts/cboss_main_oracle_full_offline_nb_vtl.sh
==== ended in error on Mon Jun 2 11:55:48 GET 2014 ====

[RiaanBadenhorst]

Hi,

That does't look like an output I'm used to. Did you create the rman script yourself or use the sample scripts that came with Netbackup? The output of the run commands aren't captured.....

[Marianne]

Please show us the rman script and the NBU policy (bppllist <policy-name> -U)

Please bear in mind that rman from the client will probably be issued as 'oracle' user, right?

When the backup is kicked from NBU scheduler on the master, the script will be executed a 'root' user.
For this reason you need to have 'su - <oracle-user> .....' 
Have a look at the sample scripts in /usr/openv/netbackup/ext/db_ext/oracle/samples/rman and compare with your script.

Test your script from cmd on the Oracle zone while logged in as root.
Does that work?

[Michael_G_Ander · Accepted solution]

The bphdb & dbclient (needs 777 permission) logs on the client might be able to indicate what the error is

I would also try to run the script(s) by hand from a prompt to see if there was some output about the error and look in Oracle alert log

See that Marianne already answered while I was typing :)

[turguns]

1. The output of bpplist is :

[root@nbumedia ~]# bppllist test_cboss_offline -U
------------------------------------------------------------

Policy Name:       test_cboss_offline

  Policy Type:         Oracle
  Active:              yes
  Effective date:      06/02/2014 11:22:33
  Block Incremental:   no
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           nbumedia-hcart2-robot-tld-0
  Volume Pool:         Svalka
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      0 seconds

  Granular Restore Info:  no
  Ignore Client Direct:  no
  HW/OS/Client:  Solaris       Solaris10     oracle-zone

  Include:  /oracle/arch/rman_scripts/cboss_main_oracle_full_offline_nb_vtl.sh

  Schedule:          Default-Application-Backup
    Type:            Application Backup
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 3 (1 month)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Sunday     00:00:00  -->  Sunday     24:00:00
          Monday     00:00:00  -->  Monday     24:00:00
          Tuesday    00:00:00  -->  Tuesday    24:00:00
          Wednesday  00:00:00  -->  Wednesday  24:00:00
          Thursday   00:00:00  -->  Thursday   24:00:00
          Friday     00:00:00  -->  Friday     24:00:00
          Saturday   00:00:00  -->  Saturday   24:00:00

  Schedule:          default
    Type:            Automatic Full Backup
    Frequency:       every 7 days
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 0 (1 week)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:

  Schedule:          cboss_inc
    Type:            Differential Incremental Backup
    Frequency:       every 7 days
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 0 (1 week)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:

2. output of RMan

 more cboss_main_oracle_full_offline_nb_vtl.sh
#!/bin/sh
# $Header: cold_database_backup.sh,v 1.2 2002/11/20 00:47:37 $
#
#bcpyrght
#***************************************************************************
#* $VRTScprght: Copyright 1993 - 2007 Symantec Corporation, All Rights Reserved $ *
#***************************************************************************
#ecpyrght
#
# ---------------------------------------------------------------------------
#                       cold_database_backup.sh
# ---------------------------------------------------------------------------
#  This script uses Recovery Manager to take a cold (consistent) database
#  backup. A cold backup is one where the database is shutdown cleanly before
#  performing the backup. It is assumed that this script will be executed
#  by user root. In order for the Oracle utilities, sqlplus, svrmgrl and rman,
#  to work properly we switch user (su -) to the oracle dba account before
#  each execution. If this script runs under a user account that has Oracle
#  dba privilege, it will be executed using this user's account.
#
#  Note: As written, this script will bring down the target database and
#        open it when done.
# ---------------------------------------------------------------------------

# ---------------------------------------------------------------------------
# Determine the user which is executing this script.
# ---------------------------------------------------------------------------

CUSER=`id |cut -d"(" -f2 | cut -d ")" -f1`

# ---------------------------------------------------------------------------
# Put output in <this file name>.out. Change as desired.
# Note: output directory requires write permission.
# ---------------------------------------------------------------------------

RMAN_LOG_FILE=${0}.out

# ---------------------------------------------------------------------------
# You may want to delete the output file so that backup information does
# not accumulate.  If not, delete the following lines.
# ---------------------------------------------------------------------------

if [ -f "$RMAN_LOG_FILE" ]
then
        rm -f "$RMAN_LOG_FILE"
fi

# -----------------------------------------------------------------
# Initialize the log file.
# -----------------------------------------------------------------
 
echo >> $RMAN_LOG_FILE
chmod 777 $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Log the start of this script.
# ---------------------------------------------------------------------------
 
echo Script $0 >> $RMAN_LOG_FILE
echo ==== started on `date` ==== >> $RMAN_LOG_FILE
echo >> $RMAN_LOG_FILE
 
# ---------------------------------------------------------------------------
# Replace /db/oracle/product/ora81, below, with the Oracle home path.
# ---------------------------------------------------------------------------

ORACLE_HOME=/home/ora10/geo
export ORACLE_HOME

# ---------------------------------------------------------------------------
# Replace ora81, below, with the Oracle SID of the target database.
# ---------------------------------------------------------------------------

ORACLE_SID=geo
export ORACLE_SID

# ---------------------------------------------------------------------------
# Replace ora81, below, with the Oracle DBA user id (account).
# ---------------------------------------------------------------------------

ORACLE_USER=ora10

# ---------------------------------------------------------------------------
# Set the target connect string.
# Replace "sys/manager", below, with the target connect string.  
# This connect string is used by the Oracle utility and rman
# ---------------------------------------------------------------------------

TARGET_CONNECT_STR=system/nomoresystem

# ---------------------------------------------------------------------------
# Set the Oracle utility.
# Replace sqlplus with the appropriate Oracle utility call.
# For example, CMD=$ORACLE_HOME/bin/svrmgrl
# ---------------------------------------------------------------------------
# CMD="$ORACLE_HOME/bin/sqlplus /nolog"
# CMD="$ORACLE_HOME/bin/svrmgrl"

CMD="$ORACLE_HOME/bin/sqlplus /nolog"

# ---------------------------------------------------------------------------
# Relplace the following with the fully qualified path name of the init file
# for your target database.
# ---------------------------------------------------------------------------
TARGET_INIT_FILE=/db/oracle/product/admin/ora81/pfile/initorac81.ora

# ---------------------------------------------------------------------------
# Set the Oracle Recovery Manager name.
# ---------------------------------------------------------------------------

RMAN=$ORACLE_HOME/bin/rman

# ---------------------------------------------------------------------------
# Print out the value of the variables set by this script.
# ---------------------------------------------------------------------------

echo >> $RMAN_LOG_FILE
echo   "RMAN: $RMAN" >> $RMAN_LOG_FILE
echo   "CMD: $CMD" >> $RMAN_LOG_FILE
echo   "ORACLE_SID: $ORACLE_SID" >> $RMAN_LOG_FILE
echo   "ORACLE_USER: $ORACLE_USER" >> $RMAN_LOG_FILE
echo   "ORACLE_HOME: $ORACLE_HOME" >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Print out the value of the variables set by bphdb.
# ---------------------------------------------------------------------------

echo
echo   "NB_ORA_FULL: $NB_ORA_FULL" >> $RMAN_LOG_FILE
echo   "NB_ORA_INCR: $NB_ORA_INCR" >> $RMAN_LOG_FILE
echo   "NB_ORA_CINC: $NB_ORA_CINC" >> $RMAN_LOG_FILE
echo   "NB_ORA_SERV: $NB_ORA_SERV" >> $RMAN_LOG_FILE
echo   "NB_ORA_POLICY: $NB_ORA_POLICY" >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Shutdown the target database in immediate priority.
# This lets current calls to the database complete, but prevents further
# logons or calls.
#
# The shutdown-startup logic of this script can be commented out if you know
# that the database will always be mounted and in a consistent state before
# this script is executed.
#
# CMD_STR is set up with the proper sequence of commands to initiate the
# Oracle utility, connect to the database and perform the shutdown.
#
# The current user is checked and if it is root a su to the oracle user
# is performed and then the CMD_STR is executed.  This will be performed
# for all scheduled backups.  If the current user is not root, a user
# directed backup is assumed and the CMD_STR is just executed.
# ---------------------------------------------------------------------------

# Initiate the command string

if [ "$CUSER" = "root" ]
then
    su - $ORACLE_USER -c "$CMD_STR" >> $RMAN_LOG_FILE
else
    /usr/bin/sh -c "$CMD_STR" >> $RMAN_LOG_FILE
fi

# ---------------------------------------------------------------------------
# Now we know that the database is cleanly closed and is ready for a
# cold backup. RMAN requires that the database be started and mounted
# to perform a backup.
#
# CMD_STR is set up with the proper sequence of commands to initiate the
# Oracle utility, connect to the database, startup the database associated
# to the specified pfile in a mounted mode.
#
# The current user is checked and if it is root a su to the oracle user
# is performed and then the CMD_STR is executed.  This will be performed
# for all scheduled backups.  If the current user is not root, a user
# directed backup is assumed and the CMD_STR is just executed.
# ---------------------------------------------------------------------------

# Initiate the command string
 
if [ "$CUSER" = "root" ]
then
    su - $ORACLE_USER -c "$CMD_STR" >> $RMAN_LOG_FILE
else
    /usr/bin/sh -c "$CMD_STR" >> $RMAN_LOG_FILE
fi
 
echo >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# If this script is executed from a NetBackup schedule, NetBackup
# sets an NB_ORA environment variable based on the schedule type.
# The NB_ORA variable is then used to dynamically set BACKUP_TYPE
# For example, when:
#     schedule type is                BACKUP_TYPE is
#     ----------------                --------------
# Automatic Full                     INCREMENTAL LEVEL=0
# Automatic Differential Incremental INCREMENTAL LEVEL=1
# Automatic Cumulative Incremental   INCREMENTAL LEVEL=1 CUMULATIVE
#
# For user initiated backups, BACKUP_TYPE defaults to incremental
# level 0 (full).  To change the default for a user initiated
# backup to incremental or incremental cumulative, uncomment
# one of the following two lines.
# BACKUP_TYPE="INCREMENTAL LEVEL=1"
# BACKUP_TYPE="INCREMENTAL LEVEL=1 CUMULATIVE"
#
# Note that we use incremental level 0 to specify full backups.
# That is because, although they are identical in content, only
# the incremental level 0 backup can have incremental backups of
# level > 0 applied to it.
# ---------------------------------------------------------------------------

if [ "$NB_ORA_FULL" = "1" ]
then
        echo "Full backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=0"

elif [ "$NB_ORA_INCR" = "1" ]
then
        echo "Differential incremental backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=1"

elif [ "$NB_ORA_CINC" = "1" ]
then
        echo "Cumulative incremental backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=1 CUMULATIVE"

elif [ "$BACKUP_TYPE" = "" ]
then
        echo "Default - Full backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=0"
fi

# ---------------------------------------------------------------------------
# Call Recovery Manager to initiate the backup. This example does not use a
# Recovery Catalog. If you choose to use one, replace the option 'nocatalog'
# from the rman command line below with the
# 'rcvcat <userid>/<passwd>@<tns alias>' statement.
#
# Note: Any environment variables needed at run time by RMAN
#       must be set and exported within the switch user (su) command.
#
#  NOTE WHEN USING TNS ALIAS: When connecting to a database
#  using a TNS alias, you must use a send command or a parms operand to
#  specify environment variables.  In other words, when accessing a database
#  through a listener, the environment variables set at the system level are not
#  visible when RMAN is running.  For more information on the environment
#  variables, please refer to the NetBackup for Oracle Admin. Guide.
#
# ---------------------------------------------------------------------------

CMD_STR="
ORACLE_HOME=$ORACLE_HOME
export ORACLE_HOME
ORACLE_SID=$ORACLE_SID
export ORACLE_SID
$RMAN target /  msglog $RMAN_LOG_FILE append << EOF
RUN {
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE' parms='SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64.1 ENV=(NB_ORA_POLICY=bo_cboss_test_oracle_full_offline)';
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE' parms='SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64.1 ENV=(NB_ORA_POLICY=bo_cboss_test_oracle_full_offline)';
    BACKUP  
    tag full_cboss_test_bk_vtl
        database;
        DELETE NOPROMPT OBSOLETE;       
        delete noprompt expired backup;
        delete noprompt expired archivelog all;
}
EOF
"
# Initiate the command string
 
if [ "$CUSER" = "root" ]
then
    su - $ORACLE_USER -c "$CMD_STR" >> $RMAN_LOG_FILE
    RSTAT=$?
else
    /usr/bin/sh -c "$CMD_STR" >> $RMAN_LOG_FILE
    RSTAT=$?
fi
 
# ---------------------------------------------------------------------------
# Log the completion of this script.
# ---------------------------------------------------------------------------

if [ "$RSTAT" = "0" ]
then
    LOGMSG="ended successfully"
else
    LOGMSG="ended in error"
fi
 
echo >> $RMAN_LOG_FILE
echo Script $0 >> $RMAN_LOG_FILE
echo ==== $LOGMSG on `date` ==== >> $RMAN_LOG_FILE
echo >> $RMAN_LOG_FILE

exit $RSTAT

[turguns]

Rman was created by my coworker who is responsible from DB. Till today he created many Rman scripts.On the other systems they are working properly.

[turguns]

P.S. from CMD..it is working. But from netbackup GUI not working

[Marianne]

I would like to see rman output file when script is executed from the client and backup runs successfully.

When I look at the output file that you posted earlier, it seems as if CMD_STR is never executed.

Maybe get your dba to carefully compair working script on other clients with this one.

[turguns]

Dear All,

Thank you for your support.

I found the reason of problem.

from log I detected like this problem:

cd /usr/openv/netbackup/logs/bphdb/

more obk_stderr.060214
-sh: /tmp/sh276680: cannot create
-sh: /tmp/sh278350: cannot create
-sh: /tmp/sh281160: cannot create
 

There was /tmp directory permission was like this:

drwxr-xr-x   6 root     root         390 Jun  2 19:08 tmp

I changed it, now it seems like this. drwxrwxrwt   6 root     root         390 Jun  2 19:08 tmp

After that through Netbackup GUI I started the policy. It is working ))

Full backup requested
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
You have mail.
RMAN> 2> 3> 4> er: Release 10.2.0.5.0 - Production on Mon Jun 2 19:15:19 2014

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: CBOSS (DBID=2467411025)

RMAN> 2> 3> 4>
using target database control file instead of recovery catalog
allocated channel: cha01
channel cha01: sid=292 devtype=SBT_TAPE
channel cha01: Veritas NetBackup for Oracle - Release 7.1 (2011020313)

Starting backup at 02-JUN-14
channel cha01: starting full datafile backupset
channel cha01: specifying datafile(s) in backupset
input datafile fno=00010 name=/ora4/geo/cb_t2
channel cha01: starting piece 1 at 02-JUN-14

Thank you again))

With BR,

Turgun

[Marianne]

Please mark Michael's post as solution. He said that bphdb might indicate where the problem is:

The bphdb & dbclient (needs 777 permission) logs on the client might be able to indicate what the error is