Restoring Oracle database using the Duplicate command

[Richard_Senda3]

Hello,

I have a backup image of an Oracle database that I would like to restore to an alternative client, the restore is unsuccessful with the error:

ORA-19507: failed to retrieve sequential file, handle="c-1454518342-20110713-01", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Backup file <c-1454518342-20110713-01> not found in NetBackup catalog

The source database
The source database was backed up from a 2 node cluster through the virtual host name, not the physical name.  The RMAN command string is this:
CMD_STR="
ORACLE_HOME=$ORACLE_HOME
export ORACLE_HOME
ORACLE_SID=$ORACLE_SID
export ORACLE_SID
$RMAN target $TARGET_CONNECT_STR rcvcat $RMANCAT_CONNECT msglog $RMAN_LOG_FILE append << EOF
RUN {
configure retention policy to recovery window  of 180 days;
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
SEND 'NB_ORA_CLIENT=Lonclupsdb01vip';
SEND 'NB_ORA_POLICY=Oracle_finlnprd_full';
BACKUP
    $BACKUP_TYPE
    SKIP INACCESSIBLE
    TAG hot_db_bk_level0
    FILESPERSET 5
    # recommended format
    FORMAT 'bk_%s_%p_%t'
    DATABASE;
    sql 'alter system archive log current';
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
# backup all archive logs
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
SEND 'NB_ORA_CLIENT=Lonclupsdb01vip';
SEND 'NB_ORA_POLICY=Oracle_finlnprd_full';
BACKUP
   filesperset 20
   FORMAT 'al_%s_%p_%t'
   ARCHIVELOG ALL DELETE INPUT;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
}
EOF

The physical node name is lonclupsdb01, but the VIP name is Lonclupsdb01vip.  In bp.conf on the physical host, we have set the CLIENT_NAME = Lonclupsdb01 which is the physical node name and not the cluster name.  We have used the SEND command to send the vip name to NetBackup and in the backup policy in NetBackup, the Client Name has been set to the VIP – Lonclupsdb01vip.

The backups work perfectly and finish in status code 0.

The destination
We want to restore the backups of the source to a different server into a different database.
On the NetBackup Master Server, we have created the touch file /usr/openv/netbackup/db/altnames/No.Restrictions
On the destination client, we have set in bp.conf CLIENT_NAME = Lonpclupsdbdev01 which is the physical host name on the node where we are running the RMAN restore

The RMAN command string we are using is this one:
run
{
   set until time '11/08/2011 08:00:00';
   ALLOCATE AUXILIARY CHANNEL FINLNQA device type sbt_tape;
   SEND "NB_ORA_CLIENT=Lonclupsdb01vip";
   duplicate target database to FINLNQA;
}

When we run this command, we see the following output:

channel FINLNQA: starting datafile backup set restore
channel FINLNQA: restoring control file
channel FINLNQA: reading from backup piece c-1454518342-20110713-02
channel FINLNQA: ORA-19870: error while restoring backup piece c-1454518342-20110713-02
ORA-19507: failed to retrieve sequential file, handle="c-1454518342-20110713-02", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Backup file <c-1454518342-20110713-02> not found in NetBackup catalog

I don't quite understand why I am seeing this error as we have used the SEND command to send the correct client name.  There is a similar issue described in technote TECH43525.

Many thanks.

Richard.

[Nicolai]

Using bplist see if the altname permissions work. As root on the destination host issue:

/usr/openv/netbackup/bin/bplist -C SourceHost  -R -l -t 4 -s 07/25/2011 /

Where -C is the name of the source. You should see a listing of all the backup/archives made on the source. Just make sure to get the start date right.

[Yasuhisa_Ishika]

Have you already configured altnames to allow alternative cilent to read original client's data?

[Richard_Senda3]

Hello,

Many thanks, but on the Master Server, I have created the touch file

/usr/openv/netbackup/db/altnames/No.Restrictions
 

I can confirm that running the bplist command is successful, it successfully lists all of the available backup images.  I can also confirm that running the RMAN string with the RESTORE command also works correctly, it is only the DUPLICATE command that doesn't work.

I have had a detailed look through the dbclient log from the client and also the bprd log on the Master Server, it looks to me that NetBackup is searching the catalog using the wrong client name - it is using the local node name and not the one specified in the SEND parameter, hence it returns status code 227 - no entity found.

I'll have another detailed look at the logs today.

[Richard_Senda3]

Having spent a very boring morning looking through the dbclient log, found something interesting.  We are using the RMAN duplicate command to restore the production database onto a development system for testing.  When using the duplicate command, we can see it opening and closing the connection to NetBackup 3 times:

First time:
13:36:53.404 [29751] <4> VxBSAInit: Veritas NetBackup XBSA Interface - 7.1  2011020316

Second time:
13:37:07.882 [29971] <4> VxBSAInit: Veritas NetBackup XBSA Interface - 7.1  2011020316

Third time:
13:37:26.306 [30458] <4> VxBSAInit: Veritas NetBackup XBSA Interface - 7.1  2011020316

On the first time it opens the connection to NetBackup, we can see these commands issued:
13:36:54.030 [29751] <2> sbtcommand: INF - command string=<NB_ORA_CLIENT=Lonclupsdb01vip>
13:36:54.030 [29751] <2> int_ProcessCommandString: INF - entering
13:36:54.030 [29751] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_CLIENT> cmd_val=<Lonclupsdb01vip>
13:36:54.030 [29751] <2> int_ProcessCommand: INF - entering
13:36:54.030 [29751] <2> xbsa_SetEnv: INF - entering
13:36:54.030 [29751] <4> VxBSASetEnv: INF - entering SetEnv - NBBSA_CLIENT_HOST
13:36:54.030 [29751] <2> xbsa_SetEnv: INF - leaving (0)
13:36:54.030 [29751] <2> int_ProcessCommand: INF - Client <Lonclupsdb01vip> will be used for this API session.

You can see that it has correctly picked up the SEND command and set the correct client name - Lonclupsdb01vip.

However, on the second and third times that the connection to NetBackup is opened, we do not see these commands at all, the SEND command is not being sent.  As a result, when the command is formed to query the NetBackup catalog, we see this:
13:37:27.265 [30458] <4> dbc_GetMediaListByName: Request String = <7.1 Lonpclupsdbdev01 Lonpclupsdbdev01 *NULL* 4 1312974000 1313146800 /c-1454518342-20110811-01>

This is incorrect, the correct command should be:
13:37:27.265 [30458] <4> dbc_GetMediaListByName: Request String = <7.1 Lonclupsdb01vip Lonpclupsdbdev01 *NULL* 4 1312974000 1313146800 /c-1454518342-20110811-01>

As a result, we get this error:
13:37:27.702 [30458] <4> VxBSAQueryObject: INF - Object </c-1454518342-20110811-01> was not found in the NetBackup catalog.

This looks like a problem with RMAN that it is not correctly sending the SEND command to NetBackup when used with the duplicate command.  I have other servers with different versions of RMAN so I will enable the logging on them to see if there are any differences.  The RMAN version we are using is 11.2.0.2.0, which I believe is the latest.

We have a workaround for this problem that is to create a bp.conf file in /home/oracle.  In that bp.conf file, enter in:

CLIENT_NAME = Lonclupsdb01vip

Then the restore using the duplicate command works correctly.

[Richard_Senda3 · Accepted solution]

I can confirm that this problem is due to Oracle bug ID 11783792

The workaround is either to use that in the bug ID, or to create a bp.conf file in /home/oracle and put in that bp.conf file

CLIENT_NAME = Lonclupsdb01vip