NBU script not connecting to RMAN, no errors shown

We had scheduled backup using netbackup, for oracle database on Exadata server.

There are 3 databases running on 3 different homes on the same Exadata server.

All the 3 backups were running fine.

The backups for 1 database stopped running after the sys password for that database was changed.

The remaining 2 database backups runs fine.

Below is the RMAN script and the log file.

--------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------------------------------

#!/bin/sh
# $Header: hot_database_backup.sh,v 1.3 2010/08/04 17:56:02 $
#
#bcpyrght
#***************************************************************************
#* $VRTScprght: Copyright 1993 - 2011 Symantec Corporation, All Rights Reserved $ *
#***************************************************************************
#ecpyrght
#
# ---------------------------------------------------------------------------
#                       hot_database_backup.sh
# ---------------------------------------------------------------------------
#  This script uses Recovery Manager to take a hot (inconsistent) database
#  backup. A hot backup is inconsistent because portions of the database are
#  being modified and written to the disk while the backup is progressing.
#  You must run your database in ARCHIVELOG mode to make hot backups. It is
#  assumed that this script will be executed by user root. In order for RMAN
#  to work properly we switch user (su -) to the oracle dba account before
#  execution. If this script runs under a user account that has Oracle dba
#  privilege, it will be executed using this user's account.
# ---------------------------------------------------------------------------

# ---------------------------------------------------------------------------
# Determine the user which is executing this script.
# ---------------------------------------------------------------------------

RDATE=`date +%d%b%y%H%M%S`
CUSER=`id |cut -d"(" -f2 | cut -d ")" -f1`

# ---------------------------------------------------------------------------
# Put output in <this file name>.out. Change as desired.
# Note: output directory requires write permission.
# ---------------------------------------------------------------------------

#RMAN_LOG_FILE=/u01/app/oracle/admin/dwprd/scripts/rman/log
#export RMAN_LOG_FILE
RMAN_LOG_FILE=${0}_${RDATE}.out

# ---------------------------------------------------------------------------
# You may want to delete the output file so that backup information does
# not accumulate.  If not, delete the following lines.
# ---------------------------------------------------------------------------

if [ -f "$RMAN_LOG_FILE" ]
then
        rm -f "$RMAN_LOG_FILE"
fi

# -----------------------------------------------------------------
# Initialize the log file.
# -----------------------------------------------------------------

echo >> $RMAN_LOG_FILE
chmod 666 $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Log the start of this script.
# ---------------------------------------------------------------------------

echo Script $0 >> $RMAN_LOG_FILE
echo ==== started on `date` ==== >> $RMAN_LOG_FILE
echo >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Replace /db/oracle/product/ora102, below, with the Oracle home path.
# ---------------------------------------------------------------------------

ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
export ORACLE_HOME

# ---------------------------------------------------------------------------
# Replace ora102, below, with the Oracle SID of the target database.
# ---------------------------------------------------------------------------

ORACLE_SID=PRDDW2C12
export ORACLE_SID

# ---------------------------------------------------------------------------
# Replace ora102, below, with the Oracle DBA user id (account).
# ---------------------------------------------------------------------------

ORACLE_USER=oracle

# ---------------------------------------------------------------------------
# Set the target connect string.
# Replace "sys/manager", below, with the target connect string.
# ---------------------------------------------------------------------------

# TARGET_CONNECT_STR=/
TARGET_CONNECT_STR=sys/S$19t2m1@rmanbkup
#TARGET_CONNECT_STR=sys/S$19t2m1@njprddw2c1_dg

# ---------------------------------------------------------------------------
# Set the Oracle Recovery Manager name.
# ---------------------------------------------------------------------------

RMAN=$ORACLE_HOME/bin/rman

#----------------------------------------------------------------------------
# Custom environment variables added by Mediacom (Saidi Hassan, 2/12/2016)
#----------------------------------------------------------------------------

MGT=/u01/oracle/admin/tools/db_mgmt
LOG_DIR=/u01/app/oracle/admin/NJPRDDW2C1/scripts/rman/log

# ---------------------------------------------------------------------------
# Print out the value of the variables set by this script.
# ---------------------------------------------------------------------------

echo >> $RMAN_LOG_FILE
echo   "RMAN: $RMAN" >> $RMAN_LOG_FILE
echo   "ORACLE_SID: $ORACLE_SID" >> $RMAN_LOG_FILE
echo   "ORACLE_USER: $ORACLE_USER" >> $RMAN_LOG_FILE
echo   "ORACLE_HOME: $ORACLE_HOME" >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Print out the value of the variables set by bphdb.
# ---------------------------------------------------------------------------

echo  >> $RMAN_LOG_FILE
echo   "NB_ORA_FULL: $NB_ORA_FULL" >> $RMAN_LOG_FILE
echo   "NB_ORA_INCR: $NB_ORA_INCR" >> $RMAN_LOG_FILE
echo   "NB_ORA_CINC: $NB_ORA_CINC" >> $RMAN_LOG_FILE
echo   "NB_ORA_SERV: $NB_ORA_SERV" >> $RMAN_LOG_FILE
echo   "NB_ORA_POLICY: $NB_ORA_POLICY" >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# NOTE: This script assumes that the database is properly opened. If desired,
# this would be the place to verify that.
# ---------------------------------------------------------------------------

echo >> $RMAN_LOG_FILE
# ---------------------------------------------------------------------------
# If this script is executed from a NetBackup schedule, NetBackup
# sets an NB_ORA environment variable based on the schedule type.
# The NB_ORA variable is then used to dynamically set BACKUP_TYPE
# For example, when:
#     schedule type is                BACKUP_TYPE is
#     ----------------                --------------
# Automatic Full                     INCREMENTAL LEVEL=0
# Automatic Differential Incremental INCREMENTAL LEVEL=1
# Automatic Cumulative Incremental   INCREMENTAL LEVEL=1 CUMULATIVE
#
# For user initiated backups, BACKUP_TYPE defaults to incremental
# level 0 (full).  To change the default for a user initiated
# backup to incremental or incremental cumulative, uncomment
# one of the following two lines.
# BACKUP_TYPE="INCREMENTAL LEVEL=1"
# BACKUP_TYPE="INCREMENTAL LEVEL=1 CUMULATIVE"
#
# Note that we use incremental level 0 to specify full backups.
# That is because, although they are identical in content, only
# the incremental level 0 backup can have incremental backups of
# level > 0 applied to it.
# ---------------------------------------------------------------------------

if [ "$NB_ORA_FULL" = "1" ]
then
        echo "Full backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=0"

elif [ "$NB_ORA_INCR" = "1" ]
then
        echo "Differential incremental backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=1"

elif [ "$NB_ORA_CINC" = "1" ]
then
        echo "Cumulative incremental backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=1 CUMULATIVE"

elif [ "$BACKUP_TYPE" = "" ]
then
        echo "Default - Full backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=0"
fi

# ---------------------------------------------------------------------------
# Call Recovery Manager to initiate the backup. This example does not use a
# Recovery Catalog. If you choose to use one, replace the option 'nocatalog'
# from the rman command line below with the
# 'catalog <userid>/<passwd>@<net service name>' statement.
#
# Note: Any environment variables needed at run time by RMAN
#       must be set and exported within the switch user (su) command.
# ---------------------------------------------------------------------------
#  Backs up the whole database.  This backup is part of the incremental
#  strategy (this means it can have incremental backups of levels > 0
#  applied to it).
#
#  We do not need to explicitly request the control file to be included
#  in this backup, as it is automatically included each time file 1 of
#  the system tablespace is backed up (the inference: as it is a whole
#  database backup, file 1 of the system tablespace will be backed up,
#  hence the controlfile will also be included automatically).
#
#  Typically, a level 0 backup would be done at least once a week.
#
#  The scenario assumes:
#     o you are backing your database up to two tape drives
#     o you want each backup set to include a maximum of 5 files
#     o you wish to include offline datafiles, and read-only tablespaces,
#       in the backup
#     o you want the backup to continue if any files are inaccessible.
#     o you are not using a Recovery Catalog
#     o you are explicitly backing up the control file.  Since you are
#       specifying nocatalog, the controlfile backup that occurs
#       automatically as the result of backing up the system file is
#       not sufficient; it will not contain records for the backup that
#       is currently in progress.
#     o you want to archive the current log, back up all the
#       archive logs using two channels, putting a maximum of 20 logs
#       in a backup set, and deleting them once the backup is complete.
#
#  Note that the format string is constructed to guarantee uniqueness and
#  to enhance NetBackup for Oracle backup and restore performance.
#
#
#  NOTE WHEN USING NET SERVICE NAME: When connecting to a database
#  using a net service name, you must use a send command or a parms operand to
#  specify environment variables.  In other words, when accessing a database
#  through a listener, the environment variables set at the system level are not
#  visible when RMAN is running.  For more information on the environment
#  variables, please refer to the NetBackup for Oracle Admin. Guide.
#
# ---------------------------------------------------------------------------

CMD_STR="
ORACLE_HOME=$ORACLE_HOME
export ORACLE_HOME
ORACLE_SID=$ORACLE_SID
export ORACLE_SID
$RMAN target $TARGET_CONNECT_STR nocatalog msglog $RMAN_LOG_FILE append << EOF
RUN {
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch04 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch05 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch06 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch07 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch08 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch09 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch10 TYPE 'SBT_TAPE';
CONFIGURE CONTROLFILE AUTOBACKUP ON;
SEND 'NB_ORA_POLICY=Oracle12-njtoexadbadm02-PRDDW2C12,NB_ORA_SERV=njtonbus01,NB_ORA_CLIENT=njtoexadb02-vip';
BACKUP
    $BACKUP_TYPE
    SKIP INACCESSIBLE
    TAG hot_db_bk_level0
    FILESPERSET 5
    # recommended format
    FORMAT 'bk_%s_%p_%t'
    DATABASE;
#    sql 'alter system archive log current';
# backup all archive logs
BACKUP
   filesperset 20
   FORMAT 'al_%s_%p_%t'
   ARCHIVELOG ALL NOT BACKED UP
   DELETE INPUT;
#
# Note: During the process of backing up the database, RMAN also backs up the
# control file.  This version of the control file does not contain the
# information about the current backup because "nocatalog" has been specified.
# To include the information about the current backup, the control file should
# be backed up as the last step of the RMAN section.  This step would not be
# necessary if we were using a recovery catalog or auto control file backups.
#
BACKUP
    # recommended format
    FORMAT 'cntrl_%s_%p_%t'
    CURRENT CONTROLFILE;
sql 'alter database backup controlfile to trace';
RELEASE CHANNEL ch01;
RELEASE CHANNEL ch02;
RELEASE CHANNEL ch03;
RELEASE CHANNEL ch04;
RELEASE CHANNEL ch05;
RELEASE CHANNEL ch06;
RELEASE CHANNEL ch07;
RELEASE CHANNEL ch08;
RELEASE CHANNEL ch09;
RELEASE CHANNEL ch10;
}
EOF
"
# Initiate the command string

if [ "$CUSER" = "root" ]
then
    su - $ORACLE_USER -c "$CMD_STR" >> $RMAN_LOG_FILE
    RSTAT=$?
else
    /usr/bin/sh -c "$CMD_STR" >> $RMAN_LOG_FILE
    RSTAT=$?
fi

# ---------------------------------------------------------------------------
# Log the completion of this script.
# ---------------------------------------------------------------------------

if [ "$RSTAT" = "0" ]
then
    LOGMSG="ended successfully"
else
    LOGMSG="ended in error"
fi

echo >> $RMAN_LOG_FILE
echo Script $0 >> $RMAN_LOG_FILE
echo ==== $LOGMSG on `date` ==== >> $RMAN_LOG_FILE
echo >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Custom code to clean up log files - added by Mediacom (Saidi Hassan, 9/14/2015)
# ---------------------------------------------------------------------------

echo "Recording disk free info ..." >> $RMAN_LOG_FILE
df -h | sort >> $RMAN_LOG_FILE

echo "Deleting outdated (+70 days) backup log files ..." >> $RMAN_LOG_FILE
find $LOG_DIR -name "*.out" -mtime +70 -print -exec rm {} ";" >> $RMAN_LOG_FILE 2>&1

# ---------------------------------------------------------------------------
# Custom code to identify errors and send e-mail notifications - added by
#       Mediacom (Saidi Hassan, 2/12/2016)
# ---------------------------------------------------------------------------

cat "$RMAN_LOG_FILE" | grep "RMAN-" | grep -v "RMAN-06908" | grep -v "RMAN-06909" | grep -v "RMAN-08132" > rm                                   an.err

if [ -s "rman.err" ] || [ "$RSTAT" -ne "0" ]
then

for i in `cat $MGT/Distributionlist.txt`
do
cat "$RMAN_LOG_FILE" | mailx -s "*** ALERT-rman backup failed for $ORACLE_SID ***" ""$i""
done

else

for i in `cat $MGT/Distributionlist.txt`
do
cat "$RMAN_LOG_FILE" | mailx -s "** SUCCESS-rman backup successful for $ORACLE_SID ***" ""$i""
done
fi

exit $RSTAT

------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------------------------------

 

Log:

Script /u01/app/oracle/admin/NJPRDDW2C1/scripts/rman/hot_database_backup.sh
==== started on Fri Apr 22 17:59:06 EDT 2016 ====

RMAN: /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/rman
ORACLE_SID: PRDDW2C12
ORACLE_USER: oracle
ORACLE_HOME: /u01/app/oracle/product/12.1.0.2/dbhome_1

NB_ORA_FULL: 0
NB_ORA_INCR: 1
NB_ORA_CINC: 0
NB_ORA_SERV: njtonbus01
NB_ORA_POLICY: Oracle12-njtoexadbadm02-PRDDW2C12

Differential incremental backup requested

Warning:Your Environment is set but your DB Instance is not
Usage: -bash <your SID>

Script /u01/app/oracle/admin/NJPRDDW2C1/scripts/rman/hot_database_backup.sh
==== ended in error on Fri Apr 22 17:59:06 EDT 2016 ====

Recording disk free info ...
                       30G   16G   13G  57% /
                       99G   28G   67G  30% /u03
                       99G   46G   48G  50% /u02
                       99G   64G   30G  69% /u01
/dev/mapper/VGExaDb-LVDbOra1
/dev/mapper/VGExaDb-LVDbOra2
/dev/mapper/VGExaDb-LVDbOra3
/dev/mapper/VGExaDb-LVDbSys1
/dev/sda1             504M   44M  435M  10% /boot
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 252G   16K  252G   1% /dev/shm
Deleting outdated (+70 days) backup log files ...

 

-------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------------------