Oracle RAC backup with failover VIP exists and backup is load balanced

[esofronov]

Task: Testing Oracle RAC backup in loadbalanced mode, so as test restore from the backup to different client system outside the RAC environment.

In Netbackup for Oracle Guide our scheme is named as "Failover VIP exists and backup is load balanced"

To realize Oracle RAC backup we have made the following settings:

1. Specific sh-script was written

cat /acfsshare/backup/dbm_rman_backup_11g.sh

that includes necessary lines

RUN {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' CONNECT $TARGET_CONNECT_STR;
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE' CONNECT $TARGET_CONNECT_STR;
SEND 'NB_ORA_POLICY=EXT_ORA';
BACKUP
    archivelog all
BACKUP SPFILE FORMAT 'spfile_${DATE}_%U.ora';
BACKUP CURRENT CONTROLFILE FORMAT 'control_${DATE}_%U.ctl';
delete noprompt obsolete;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
}

2. Backup policies in Netbackup were created

EXT_ora_dbm1 - starts the backup script which starts the next policy

EXT_ORA - starts the loadbalanced backup

3. Alternate names files were created in the following directory 

C:\Program Files\Veritas\NetBackup\db\altnames

file dm2db01

dm2db01.domain.ru
dm2db02.domain.ru
dm2-scan.domain.ru
oratest01
oratest01.localdomain

file dm2db02

dm2db01.domain.ru
dm2db02.domain.ru
dm2-scan.domain.ru
oratest01
oratest01.localdomain

file oratest01

oratest01
oratest01.localdomain
dm2db01.domain.ru
dm2db02.domain.ru
dm2-scan.domain.ru
dm2db01
dm2db02

As you can see files were fulfilled with surplus data for eliminating any name mistakes.

4. In order to store all backup image data  in one directory 2 directory junctions were created in the image-directory

C:\Program Files\Veritas\NetBackup\db\images\dm2db01.domain.ru <<===>> C:\Program Files\Veritas\NetBackup\db\images\dm2-scan.domain.ru

C:\Program Files\Veritas\NetBackup\db\images\dm2db02.domain.ru <<===>> C:\Program Files\Veritas\NetBackup\db\images\dm2-scan.domain.ru

It allows netbackup to store all the backup data into one directory from both RAC-nodes C:\Program Files\Veritas\NetBackup\db\images\dm2-scan.domain.ru

As a result we have successfully backup archivelogs from both nodes in loadbalanced mode. As we can see below within one session backuppieces copied from both nodes in parallel

But when we trying to restore from that backup we ran into the following problem.

Firstly, when we check which images owned by which client

 .\bplist.exe -B -C <client name> -t 4 -l -R /

we can see that storing all backups into one directory does not give us the required result (look at the last copied controlfiles - it is different)

PS C:\Program Files\Veritas\NetBackup\bin>  .\bplist.exe -B -C dm2db02.domain.ru -s 04/29/2015 -e 04/29/2015 -t 4 -l -R
/
-rw-rw---- oracle    dba         189267968 ряЁ 29 13:08 c-1282940970-20150429-19:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 13:08 control_20150429_1306_duq5jm1u_1_1.ctl:\
-rw-rw---- oracle    dba            262144 ряЁ 29 13:07 spfile_20150429_1306_dtq5jm12_1_1.ora:\
-rw-rw---- oracle    dba           4456448 ряЁ 29 13:07 arch_DBM_20150429_1306_drq5jlvp_1_1.rman:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 11:08 c-1282940970-20150429-17:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 11:08 control_20150429_1106_dkq5jf1c_1_1.ctl:\
-rw-rw---- oracle    dba            262144 ряЁ 29 11:08 spfile_20150429_1106_djq5jf0g_1_1.ora:\
-rw-rw---- oracle    dba           2097152 ряЁ 29 11:07 arch_DBM_20150429_1106_diq5jevh_1_1.rman:\
-rw-rw---- oracle    dba            524288 ряЁ 29 11:06 arch_DBM_20150429_1106_dgq5jeue_1_1.rman:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 10:37 c-1282940970-20150429-16:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 10:37 control_20150429_1035_deq5jd6i_1_1.ctl:\
-rw-rw---- oracle    dba            262144 ряЁ 29 10:36 spfile_20150429_1035_ddq5jd5m_1_1.ora:\
-rw-rw---- oracle    dba            262144 ряЁ 29 10:36 arch_DBM_20150429_1035_dcq5jd4o_1_1.rman:\
-rw-rw---- oracle    dba           2097152 ряЁ 29 10:35 arch_DBM_20150429_1035_dbq5jd3k_1_1.rman:\
-rw-rw---- oracle    dba           2621440 ряЁ 29 10:35 arch_DBM_20150429_1035_daq5jd3k_1_1.rman:\
PS C:\Program Files\Veritas\NetBackup\bin>  .\bplist.exe -B -C dm2db01.domain.ru -s 04/29/2015 -e 04/29/2015 -t 4 -l -R
/
-rw-rw---- oracle    dba         189267968 ряЁ 29 14:07 c-1282940970-20150429-1a:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 14:06 control_20150429_1406_e3q5jpf8_1_1.ctl:\
-rw-rw---- oracle    dba            262144 ряЁ 29 14:06 spfile_20150429_1406_e2q5jped_1_1.ora:\
-rw-rw---- oracle    dba           4456448 ряЁ 29 14:05 arch_DBM_20150429_1406_e1q5jpd4_1_1.rman:\
-rw-rw---- oracle    dba           4718592 ряЁ 29 14:05 arch_DBM_20150429_1406_e0q5jpd4_1_1.rman:\
-rw-rw---- oracle    dba           4718592 ряЁ 29 13:05 arch_DBM_20150429_1306_dsq5jlt0_1_1.rman:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 12:07 c-1282940970-20150429-18:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 12:06 control_20150429_1206_dpq5jiep_1_1.ctl:\
-rw-rw---- oracle    dba            262144 ряЁ 29 12:06 spfile_20150429_1206_doq5jidu_1_1.ora:\
-rw-rw---- oracle    dba           4456448 ряЁ 29 12:05 arch_DBM_20150429_1206_dnq5jicl_1_1.rman:\
-rw-rw---- oracle    dba           4718592 ряЁ 29 12:05 arch_DBM_20150429_1206_dmq5jicl_1_1.rman:\
-rw-rw---- oracle    dba           2621440 ряЁ 29 11:05 arch_DBM_20150429_1106_dhq5jerk_1_1.rman:\
-rw-rw---- oracle    dba          61865984 ряЁ 29 10:08 c-3799112179-20150429-15:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 10:08 c-1282940970-20150429-15:\
-rw-rw---- oracle    dba          61865984 ряЁ 29 10:07 cntrl_FSSDBE_18910_1_878292478:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 10:07 cntrl_DBM_16808_1_878292467:\
-rw-rw---- oracle    dba          61865984 ряЁ 29 10:07 c-3799112179-20150429-14:\
-rw-rw---- oracle    dba         189267968 ряЁ 29 10:07 c-1282940970-20150429-14:\
-rw-rw---- oracle    dba          65536000 ряЁ 29 10:06 al_FSSDBE_18908_1_878292409:\
-rw-rw---- oracle    dba           9175040 ряЁ 29 10:06 al_DBM_16806_1_878292408:\
 

So Netbackup does not accept backups from first node to be the same backups from second node and otherwise.

At the time of restore to the test client oratest01, when we allocate only one chennal (for node dm2db01 for example), RMAN does not recieve the last backed up spfile.

RMAN> RUN
{
  set dbid=1282940970;
  ALLOCATE CHANNEL c1 DEVICE TYPE sbt ;
  SEND 'NB_ORA_CLIENT=dm2db02.domain.ru';
  RESTORE SPFILE TO pfile '/tmp/pfile.ora';
}

2> 3> 4> 5> 6> 7>
executing command: SET DBID
database name is "DBM" and DBID is 1282940970

allocated channel: c1
channel c1: SID=171 device type=SBT_TAPE
channel c1: Veritas NetBackup for Oracle - Release 7.5 (2012020801)

sent command to channel: c1

Starting restore at 29-APR-15

channel c1: starting datafile backup set restore
channel c1: restoring SPFILE to PFILE
output file name=/tmp/pfile.ora
channel c1: reading from backup piece c-1282940970-20150429-1a
channel c1: ORA-19870: error while restoring backup piece c-1282940970-20150429-1a
ORA-19507: failed to retrieve sequential file, handle="c-1282940970-20150429-1a", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Backup file <c-1282940970-20150429-1a> not found in NetBackup catalog

failover to previous backup

channel c1: starting datafile backup set restore
channel c1: restoring SPFILE to PFILE
output file name=/tmp/pfile.ora
channel c1: reading from backup piece spfile_20150429_1406_e2q5jped_1_1.ora
channel c1: ORA-19870: error while restoring backup piece spfile_20150429_1406_e2q5jped_1_1.ora
ORA-19507: failed to retrieve sequential file, handle="spfile_20150429_1406_e2q5jped_1_1.ora", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Backup file <spfile_20150429_1406_e2q5jped_1_1.ora> not found in NetBackup catalog

failover to previous backup

channel c1: starting datafile backup set restore
channel c1: restoring SPFILE to PFILE
output file name=/tmp/pfile.ora
channel c1: reading from backup piece c-1282940970-20150429-19
channel c1: piece handle=c-1282940970-20150429-19 tag=TAG20150429T130843
channel c1: restored backup piece 1
channel c1: restore complete, elapsed time: 00:00:25
Finished restore at 29-APR-15
released channel: c1

If we want to restore archivelogs we should exactly know from whitch node every file were backed up.

But we have found one undocumented solution, and we dont know how it will work in multinode RAC environment. You should allocete both channels to restore from both nodes:

RMAN> RUN
{
  set dbid=1282940970;
  ALLOCATE CHANNEL c1 DEVICE TYPE sbt PARMS='ENV=(NB_ORA_CLIENT=dm2db01.domain.ru)';
  ALLOCATE CHANNEL c2 DEVICE TYPE sbt PARMS='ENV=(NB_ORA_CLIENT=dm2db02.domain.ru)';
  RESTORE SPFILE TO pfile '/tmp/pfile.ora';
}
2> 3> 4> 5> 6> 7>
executing command: SET DBID
database name is "DBM" and DBID is 1282940970

allocated channel: c1
channel c1: SID=171 device type=SBT_TAPE
channel c1: Veritas NetBackup for Oracle - Release 7.5 (2012020801)

allocated channel: c2
channel c2: SID=11 device type=SBT_TAPE
channel c2: Veritas NetBackup for Oracle - Release 7.5 (2012020801)

Starting restore at 29-APR-15

channel c1: starting datafile backup set restore
channel c1: restoring SPFILE to PFILE
output file name=/tmp/pfile.ora
channel c1: reading from backup piece c-1282940970-20150429-1a
channel c1: piece handle=c-1282940970-20150429-1a tag=TAG20150429T140702
channel c1: restored backup piece 1
channel c1: restore complete, elapsed time: 00:00:35
Finished restore at 29-APR-15
released channel: c1
released channel: c2
 

Now we can see that the last control file restores at first try.

But in multinode cluster (as example with 20 nodes), we should allocate 20 channels and we do not know how it will work then.

The main question is the following.

All that we did is correct i think, so why the command responds incorrectly?

 .\bplist.exe -B -C <node name> -t 4 -l -R /

It does not show the same results for both nodes, and how to get rid of allocating more channels at time of restore?? Why in the official guide Netbackup for Oracle  vendor offers to store backups from both nodes in one directory if it doesn't give us the required result?

Sorry for my English...

[Michael_G_Ander]

As I read this set up, I would think the backups was stored under dm2-scan.domain.ru. Which I guess is your VIP name.

Guess that NB_ORA_CLIENT in backup/restore scripts also need to the VIP name dm2-scan.domain.ru to get all images stored under one name.

[esofronov]

It doesn't matter

bplist does not work with this name either.

And i don't use client names in backup scripts because of node load balancing.

[Michael_G_Ander]

As long as you don't use NB_ORA_CLIENT, I can't see how you can expect to be able to list all the backup images under one client name as the backup images are saved as clientname_epochtime.

Have not done RAC with VIP, but mutiple other clusters and there NB_ORA_CLIENT = logical name was/is the way to go

[Mouse]

Well, if you check this technote http://www.symantec.com/docs/HOWTO89687 then it is clear that you have to send SEND 'NB_ORA_CLIENT=$NB_ORA_CLIENT' in each RMAN clause and the client name should be your VIP load balanced name, by doing so you will record all your load-balanced backups under common name in the NBU catalog.

When you do recovery, you need to use that name to look for images with bplist -l -R -C <VIP_NAME> -t 4 /

[esofronov]

We have one RMAN script and, for example, 4 RAC nodes.

When the script starts on any node it has SEND 'NB_ORA_CLIENT=vip-name_of_other_node. Or we should allocate channels for every node in the script?

I cant understand how i need to write one rman script as it will work on any node?

[Marianne]

The same NB_ORA_CLIENT name used in the backup script must also be used for the restore.
If NB_ORA_CLIENT was not specified in the backup script, then this will be the same as local nodename.

If restore is done from any other node, NB_ORA_CLIENT must be specified as the name of the source client.

[esofronov]

I want to organize the backup strategy written in this document

https://support.symantec.com/en_US/article.TECH43187.html

Can you explain, why do they write

After the backups complete, restore requests can originate from either node using the common name to access all of the backup images.  E.g.
 

SEND 'NB_ORA_CLIENT=myRAC';

as it is not working. 

As far as i understand, it does not matter, what name do you use VIP or node_name, it is only need when you want to keep file and oracle backups separated.

[esofronov]

For example

I have 10 nodes.

I want to backup oracle through 2 rman channels.

I dont' know to which node i should connect at the time of backup (the database can be on any node).

What should I write in SEND?

[esofronov]

For example

I have 10 nodes.

I want to backup oracle through 2 rman channels.

I dont' know to which node i should connect at the time of backup (the database can be on any node).

What should I insert in SEND?

[Marianne]

"Failover VIP" should be associated with a hostname.

This is the name that should be used in NB_ORA_CLIENT parameter.

[esofronov]

Sorry, but it's not an answer.

Failover VIP - is just additional name of the node. I can use any name even fictional, backups will work, but for the proper environment we need it to be dynamically changable (taking from DNS, for example).

When the client's RMAN starts the session it connects to the master server through his node name (that written in bp.conf file), and all backups of this node will be placed in netbackup database under this node-name not vip-name.

Even if you are right and we use vipname (we should use 2 vipnames).

At the time of restore we should allocate two channels sending both names. So why we need to place all images into one folder? It is the general question.

[Michael_G_Ander]

If it is not working, there probably something you have missed. Have you tried with the No.Restrictions option before specific altname files ?