RMAN Backup issue

[Sachin507]

Hi,

We are taking backup AIX client on netbackup 6.5.6 and master server is 6.5.6 (red hat). On the AIX client, we have two oracle instance A and B.

Both A(deveai) and B(devedd) has weekly full backup with retention 4 weeks and Archive policy has retention 8 days.

Backup scripts for full backup are configured to take first archive backup (FORMAT al_...) then datafile backup (FORMAT bk_....) then archive backup (FORMAT al_...) and last control file backup.

Another script is configured two take only Archive backup every 4 hours (FORMAT al....).

But it has been observed that the Weekly full netbackup policy is only taking archive log backup (al_....) while the ARCHIVE policy is taking backup of both al_.... and bk_... file. And both backup  parent Job are completing Successfully (Status Code 0).

Due to this, the retention of bk_... file is taking as 8 days as no datafile is backed up in weekly full backup and hence expired. It mostly happens if Weekly Full backup policy  of instance A (deveai) and ARCHIVE backup policy of instance B(devedd) is happening simultaneously.

I need help and guidance to resolve this issue

Also please find below the Weekly Full backup script of instance A (deveai), in which only al file are backedup and bk files and control files are missed.

#!/bin/sh
# $Header: hot_database_backup.sh,v 1.2 2002/08/06 23:51:42 $
#
#bcpyrght
#***************************************************************************
#* $VRTScprght: Copyright 1993 - 2009 Symantec Corporation, All Rights Reserved $ *
#***************************************************************************
#ecpyrght
#
# ---------------------------------------------------------------------------
#                       hot_database_backup.sh
# ---------------------------------------------------------------------------
#  This script uses Recovery Manager to take a hot (inconsistent) database
#  backup. A hot backup is inconsistent because portions of the database are
#  being modified and written to the disk while the backup is progressing.
#  You must run your database in ARCHIVELOG mode to make hot backups. It is
#  assumed that this script will be executed by user root. In order for RMAN
#  to work properly we switch user (su -) to the oracle dba account before
#  execution. If this script runs under a user account that has Oracle dba
#  privilege, it will be executed using this user's account.
# ---------------------------------------------------------------------------

# ---------------------------------------------------------------------------
# Determine the user which is executing this script.
# ---------------------------------------------------------------------------

CUSER=`id |cut -d"(" -f2 | cut -d ")" -f1`

# ---------------------------------------------------------------------------
# Put output in <this file name>.out. Change as desired.
# Note: output directory requires write permission.
# ---------------------------------------------------------------------------

RMAN_LOG_FILE=${0}.out

# ---------------------------------------------------------------------------
# You may want to delete the output file so that backup information does
# not accumulate.  If not, delete the following lines.
# ---------------------------------------------------------------------------

if [ -f "$RMAN_LOG_FILE" ]
then
        rm -f "$RMAN_LOG_FILE"
fi

# -----------------------------------------------------------------
# Initialize the log file.
# -----------------------------------------------------------------

echo >> $RMAN_LOG_FILE
chmod 666 $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Log the start of this script.
# ---------------------------------------------------------------------------

echo Script $0 >> $RMAN_LOG_FILE
echo ==== started on `date` ==== >> $RMAN_LOG_FILE
echo >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Replace /db/oracle/product/ora81, below, with the Oracle home path.
# ---------------------------------------------------------------------------

ORACLE_HOME=/u01/app/oracle/product/10.2/db_1
export ORACLE_HOME

# ---------------------------------------------------------------------------
# Replace ora81, below, with the Oracle SID of the target database.
# ---------------------------------------------------------------------------

ORACLE_SID=deveai
export ORACLE_SID

# ---------------------------------------------------------------------------
# Replace ora81, below, with the Oracle DBA user id (account).
# ---------------------------------------------------------------------------

ORACLE_USER=oracle

# ---------------------------------------------------------------------------
# Set the target connect string.
# Replace "sys/manager", below, with the target connect string.
# ---------------------------------------------------------------------------

TARGET_CONNECT_STR=sys/asm1234

# ---------------------------------------------------------------------------
# Set the Oracle Recovery Manager name.
# ---------------------------------------------------------------------------

RMAN=$ORACLE_HOME/bin/rman

# ---------------------------------------------------------------------------
# Print out the value of the variables set by this script.
# ---------------------------------------------------------------------------

echo >> $RMAN_LOG_FILE
echo   "RMAN: $RMAN" >> $RMAN_LOG_FILE
echo   "ORACLE_SID: $ORACLE_SID" >> $RMAN_LOG_FILE
echo   "ORACLE_USER: $ORACLE_USER" >> $RMAN_LOG_FILE
echo   "ORACLE_HOME: $ORACLE_HOME" >> $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# Print out the value of the variables set by bphdb.
# ---------------------------------------------------------------------------

echo  >> $RMAN_LOG_FILE
echo   "NB_ORA_FULL: $NB_ORA_FULL" >> $RMAN_LOG_FILE
echo   "NB_ORA_INCR: $NB_ORA_INCR" >> $RMAN_LOG_FILE
echo   "NB_ORA_CINC: $NB_ORA_CINC" >> $RMAN_LOG_FILE
echo   "NB_ORA_SERV: $NB_ORA_SERV" >> $RMAN_LOG_FILE
echo   "NB_ORA_POLICY: $NB_ORA_POLICY" >> $RMAN_LOG_FILE
echo   "NB_ORA_SCHED: $NB_ORA_SCHED" >>  $RMAN_LOG_FILE

# ---------------------------------------------------------------------------
# NOTE: This script assumes that the database is properly opened. If desired,
# this would be the place to verify that.
# ---------------------------------------------------------------------------

echo >> $RMAN_LOG_FILE
# ---------------------------------------------------------------------------
# If this script is executed from a NetBackup schedule, NetBackup
# sets an NB_ORA environment variable based on the schedule type.
# The NB_ORA variable is then used to dynamically set BACKUP_TYPE
# For example, when:
#     schedule type is                BACKUP_TYPE is
#     ----------------                --------------
# Automatic Full                     INCREMENTAL LEVEL=0
# Automatic Differential Incremental INCREMENTAL LEVEL=1
# Automatic Cumulative Incremental   INCREMENTAL LEVEL=1 CUMULATIVE
#
# For user initiated backups, BACKUP_TYPE defaults to incremental
# level 0 (full).  To change the default for a user initiated
# backup to incremental or incremental cumulative, uncomment
# one of the following two lines.
# BACKUP_TYPE="INCREMENTAL LEVEL=1"
# BACKUP_TYPE="INCREMENTAL LEVEL=1 CUMULATIVE"
#
# Note that we use incremental level 0 to specify full backups.
# That is because, although they are identical in content, only
# the incremental level 0 backup can have incremental backups of
# level > 0 applied to it.
# ---------------------------------------------------------------------------

if [ "$NB_ORA_FULL" = "1" ]
then
        echo "Full backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=0"

elif [ "$NB_ORA_INCR" = "1" ]
then
        echo "Differential incremental backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=1"

elif [ "$NB_ORA_CINC" = "1" ]
then
        echo "Cumulative incremental backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=1 CUMULATIVE"

elif [ "$BACKUP_TYPE" = "" ]
then
        echo "Default - Full backup requested" >> $RMAN_LOG_FILE
        BACKUP_TYPE="INCREMENTAL LEVEL=0"
fi

# ---------------------------------------------------------------------------
# Call Recovery Manager to initiate the backup. This example does not use a
# Recovery Catalog. If you choose to use one, replace the option 'nocatalog'
# from the rman command line below with the
# 'rcvcat <userid>/<passwd>@<tns alias>' statement.
#
# Note: Any environment variables needed at run time by RMAN
#       must be set and exported within the switch user (su) command.
# ---------------------------------------------------------------------------
#  Backs up the whole database.  This backup is part of the incremental
#  strategy (this means it can have incremental backups of levels > 0
#  applied to it).
#
#  We do not need to explicitly request the control file to be included
#  in this backup, as it is automatically included each time file 1 of
#  the system tablespace is backed up (the inference: as it is a whole
#  database backup, file 1 of the system tablespace will be backed up,
#  hence the controlfile will also be included automatically).
#
#  Typically, a level 0 backup would be done at least once a week.
#
#  The scenario assumes:
#     o you are backing your database up to two tape drives
#     o you want each backup set to include a maximum of 5 files
#     o you wish to include offline datafiles, and read-only tablespaces,
#       in the backup
#     o you want the backup to continue if any files are inaccessible.
#     o you are not using a Recovery Catalog
#     o you are explicitly backing up the control file.  Since you are
#       specifying nocatalog, the controlfile backup that occurs
#       automatically as the result of backing up the system file is
#       not sufficient; it will not contain records for the backup that
#       is currently in progress.
#     o you want to archive the current log, back up all the
#       archive logs using two channels, putting a maximum of 20 logs
#       in a backup set, and deleting them once the backup is complete.
#
#  Note that the format string is constructed to guarantee uniqueness and
#  to enhance NetBackup for Oracle backup and restore performance.
#
#
#  NOTE WHEN USING TNS ALIAS: When connecting to a database
#  using a TNS alias, you must use a send command or a parms operand to
#  specify environment variables.  In other words, when accessing a database
#  through a listener, the environment variables set at the system level are not
#  visible when RMAN is running.  For more information on the environment
#  variables, please refer to the NetBackup for Oracle Admin. Guide.
#
# ---------------------------------------------------------------------------

CMD_STR="
ORACLE_HOME=$ORACLE_HOME
export ORACLE_HOME
ORACLE_SID=$ORACLE_SID
export ORACLE_SID
$RMAN target $TARGET_CONNECT_STR nocatalog msglog $RMAN_LOG_FILE append << EOF
RUN {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
SEND 'NB_ORA_SERV=BCKMASTERPROD, NB_ORA_CLIENT=cfrsv50151028db,NB_ORA_POLICY=BP3_28DB_Weekly_deveai,NB_ORA_SCHED=Weekly_User_Application';
BACKUP
   filesperset 20
   FORMAT 'al_%s_%p_%t'
   ARCHIVELOG ALL DELETE INPUT;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
BACKUP
    $BACKUP_TYPE
    SKIP INACCESSIBLE
    TAG hot_db_bk_level0
 FILESPERSET 5
    # recommended format
    FORMAT 'bk_%s_%p_%t'
    DATABASE;
    sql 'alter system archive log current';
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
# backup all archive logs
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
BACKUP
   filesperset 20
   FORMAT 'al_%s_%p_%t'
   ARCHIVELOG ALL DELETE INPUT;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
#
# Note: During the process of backing up the database, RMAN also backs up the
# control file.  This version of the control file does not contain the
# information about the current backup because "nocatalog" has been specified.
# To include the information about the current backup, the control file should
# be backed up as the last step of the RMAN section.  This step would not be
# necessary if we were using a recovery catalog.
#
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
BACKUP
    # recommended format
    FORMAT 'cntrl_%s_%p_%t'
    CURRENT CONTROLFILE;
RELEASE CHANNEL ch00;
}
EOF
"
# Initiate the command string

if [ "$CUSER" = "root" ]
then
    su - $ORACLE_USER -c "$CMD_STR" >> $RMAN_LOG_FILE
    RSTAT=$?
else
    /usr/bin/sh -c "$CMD_STR" >> $RMAN_LOG_FILE
    RSTAT=$?
fi

# ---------------------------------------------------------------------------
# Log the completion of this script.
# ---------------------------------------------------------------------------

if [ "$RSTAT" = "0" ]
then
    LOGMSG="ended successfully"
else
    LOGMSG="ended in error"
fi

echo >> $RMAN_LOG_FILE
echo Script $0 >> $RMAN_LOG_FILE
echo ==== $LOGMSG on `date` ==== >> $RMAN_LOG_FILE
echo >> $RMAN_LOG_FILE
echo   "NB_ORA_SCHED: $NB_ORA_SCHED" >>  $RMAN_LOG_FILE

[RamNagalla]

hi 

it looks like scripts has been interchaged

could you post the script that you are using for archive logs for every 4 hours?

[Yasuhisa_Ishika]

Please check log file of backup script if any error occured during backup operation(or attach log files here).
This backup script erase log file in every session, so you should modify script not to erase $RMAN_LOG_FILE.

BTW, why did you delete last line("exit $RSTAT") that exist in sample script? Without this line, backup script always returns 0(successful) even if error occured in RMAN.

[Marianne]

Have you double-checked that correct script name is specified in the policy?

Please provide the $RMAN_LOG_FILE as well as Client's dbclient log file.
Please post logs as File attachments.
Also let us know what time the full backup ran and what time the Archive log backup.

[Genericus]

Try adding a line that searches for running rman processes, and exits if found - so you don't run both at once.

[Sachin507]

I have attached a .rar file , which has dbclient logs, archive scripts for both instances and and daily incremental scripts and weekly full backup script.

The Incremental backup starts at 8 PM CET and archive backup for devedd run 24*7 and deveai archive runs from 6:00am CET to 8 PM CET

This issue mostly happens when both archive backup and daily or weekly backup are running simultaneously.One policy backup pieces are being backed by another policy.

I have double checked the retentions and backup selection in various policy of 28DB cleint and everything seems to be fine in policy. 

[Marianne]

It will take some time to go through all of the files/logs.

Can you confirm that the issue was seen in the period for which the logs have been collected?

[Sachin507]

Yes issue occured in this period of time. This issue occurs on regular basis.

[Will_Restore]

dbclient log does not match date range and policies given in the attachment. 

I do see this error

ORA-19588: archived log recid 81386 stamp 804381823 is no longer valid

Script /u02/rman_script/RMAN_arch_deveai.sh

==== ended in error on Fri Jan 11 11:26:01 GMT+01:00 2013 ====

which indicates Oracle crosscheck needs to be run; not a NetBackup problem, per se