Virtual names and RAC catalog maintenance

[Brad_Crowder]

Greetings,

We have a 7.5.0.5 master on HP-UX IA64, and a two-node RAC cluster on linux with 7.5.0.5 agents.  Backups have been configured based on "Failover VIP exists and backup is load balanced" in oracle admin guide.  Each node has a bp.conf in the oracle owner home directory that specifies the VIP-name for that node for CLIENT_NAME.  The RMAN script is customized to only start if NB_ORA_CLIENT is set to the SCAN name, and exit with 0 status otherwise.  SCAN name and VIP names are listed in the policy used to start the backup.  Script is stored on ACFS (cluster) filesystem to be available to both nodes.  NB_ORA_PARENT_JOBID, NB_ORA_SID, and NB_ORA_SERV are passed via RMAN SEND.  NB_ORA_CLIENT is picked up from ~oracle/bp.conf as expected.

Backups work as expected.  Channels are balanced between nodes if both are available, or if only node one is available all channels are allocated to it.

Restore is somewhat confusing.  I stopped the instance on both nodes, deleted some files from an empty tablespace using asmcmd, brought up one instance, and attempted a restore/recover, using "set autolocate" per the adminguide, with NO parameters passed via SEND or PARMS.  altnames/No.Restrictions is in place.   RMAN failed to restore pieces that were backed up on the second instance.  Ok fine.  Allocate two channels, one with NB_ORA_CLIENT=node1-vip, other with NB_ORA_CLIENT=node2-vip.  Restore works great, rman picks the right channel (dunno how but OK).

I thought maybe I could get around the NB_ORA_CLIENT issue by following the steps on page 196 of the Netbackup7.5 for Oracle Admin Guide for "Virtual names and RAC catalog maintenance", but if I follow the steps there backups fail with error code 227.  Verbose logging doesn't shed any light for my untrained eye.

Question:  Is the procedure there obsolete with the changes in the catalog for 7.5?  If so, is there another way (maybe involving bpclient -add_alias?). 

Thanks,

Brad

[Marianne]

NB_ORA_CLIENT should match the client name that was used for backups.
Status 227 means that no backups can be found for particular client name.

Before attempting a restore, see if you can use bplist with correct Client name to browse for backups.
Command usage can be found in NBU for Oracle admin guide.
 

[Brad_Crowder]

Marianne,

Understood.  bplist -C node-vip -t 4 -R / works as expected for each VIP name.

I get code 227 on a Backup if I follow the steps on page 196 under "Virtual names and RAC catalog maintenance".  So I can't even backup the instance in order to test the restore .

Brad

[Marianne]

The steps on p.196 are for restores, not for backups.

As per your opening post: Backups work as expected. 

You seemed to have a restore issue.

Are you saying that you followed these steps and that it subsequently broke the backups?

If this is the case, you need to show us what exactly was done in images folder. 
Incorrect symbolic links will probably result in failed backups.
The symbolic links should ideally be created before first backup is done to ensure that only one actual folder exist for all images on RAC nodes instead of separate folders.

Please run 'ls -l' in images folder and post all lines that represent your RAC nodes.
 

[Brad_Crowder]

testrac-scan1 is SCAN name.  ractest1 and ractest2 are physical node names.

bpexpdate all images for all names involved.  Stopped all netbackup processes.  Verified that directories were empty.  Removed ractest1, ractest1-vip,ractest2,ractest2-vip directories.  restarted netbackup services.

#pwd
/usr/openv/netbackup/db/images
#ls -lad *rac*
lrwxr-xr-x   1 root       sys             13 May 16 22:33 ractest1 -> testrac-scan1
lrwxr-xr-x   1 root       sys             13 May 16 22:17 ractest1-vip -> testrac-scan1
lrwxr-xr-x   1 root       sys             13 May 16 22:33 ractest2 -> testrac-scan1
lrwxr-xr-x   1 root       sys             13 May 16 22:24 ractest2-vip -> testrac-scan1
drwxr-xr-x   3 root       sys             96 May 16 06:34 testrac-scan1
#

I see two errors in the detailed status:  "bpbrm db_IMAGE() failed: no entity was found" and "bptm db_getIMAGE() failed: no entity was found (227)"

If I remove the softlinks, netbackup creates the regular directories when a backup is started and backups then work successfully.

Brad

[Marianne]

"If I remove the softlinks, netbackup creates the regular directories when a backup is started and backups then work successfully." This means that there is something wrong with your softlinks. I am personally using softlinks on our master and it works well. Please share Is -l output.

[Brad_Crowder]

Hello,

So the part about creating the softlinks before the first backup is MANDATORY.  I tested with a sandbox VM by creating a softlink before the first backup.  Worked fine.  After creating the first backup, I renamed the target directory and recreated the softlink, and blammo the backup failed.

Using softlinks within the db/images directory caused netbackup to associate the real directory name to the client, with the softlink name as an alias (verified with bpclient -client vmname -list_all_aliases).

Since this is a test environment, I expired all backups, verified that the directories were empty for the vip names and scanname and nuked them.  Used nbdb_unload to save my database, then dbisqlc to remove all traces of vip from DBM_MAIN.DBM_CLient and DBM_MAIN.DBM_ClientAlias.  Recreated the main directory and softlinks, and backups AND restores work correctly.

FWIW.

Brad