NBU with Oracle RAC and FRA

Some notes from experience:

About the FRA:

• When you use NetBackup for Oracle, there shouldn't really be a need for the DBA to manually create RMAN backups and put them in the FRA, although they still can.
• If NetBackup backs up the FRA, and there is data to backup, you are in most cases just backing up someone else's backups. This has its purposes, but still a bit redundant.
• What actually happens when you backup the FRA is that you are creating additional copies of the existing Backup Sets inside the FRA, you are not creating brand new Backup Sets. You can see this by running LIST BACKUPSETS. You will see for each Backup Set, the Copy#1 is located in the FRA, and Copy#2 is located in NetBackup.
• Even if you decided that you don't need the FRA in a NetBackup managed Oracle environment, the FRA is still needed for Oracle features like Flashback. Note: Flashback logs inside the FRA are skipped by RMAN and not backed up.

Using NetBackup OIP:

• Con: Cannot do Load Balanced backups from multiple RAC nodes at the same time.
• Pro: You can keep everything in one single policy.
• OIP works even on RAC. In the OIP Edit Instance "Host" field, use one of the RAC nodes' VIP name to get some form of High Availability. Don't point to node's actual IP name.
• If the VIP name fails over from one node to another, backups would still work. Unless...
• Unless the tnsnames.ora file in each node in the RAC cluster have different Net Service Name blocks.
• In the OIP "Edit Instance" menu, what you fill inside the "Instance Name" field actually points to the a Net Service Name block inside the tnsnames.ora file, and not to the actual Instance name of that specific node. (Remember: RAC nodes each have different Instance names).
• I tested this by creating new arbitrary Net Service Name blocks in the file and type these Net Service Names in the OIP "Instance Name" field and everything still works (E.e., test.test.com). That OIP field should have been named "Net Service Name" instead, IMO.
• No matter which node the VIP points to, NetBackup should still find a tnsname.ora file with the same Net Service Name block. The "INSTANECE_NAME" entry inside the block should be different on different nodes because they should point to their local Instance name. This explains why backups after a VIP failover would still work. There is no problem with restore too because NetBackup just catalogs everything under the same VIP name.
• Sum up: tnsnames.ora in all RAC nodes should have the same Net Service Name blocks, but the INSTANCE_NAME field should be different. The SERVICE_NAME field should be the same though, because no matter which Instance you are backing up from, you are still backing up the same Database.
• It is best for Oracle to use the Recovery Catalog. If not, then there will be additional steps when trying to restore things the ControlFile has already forgetten (By default, the ControlFile flags backup records after 7 days eligible to be overwritten).
  
  • If a Backup Set is forgotten by the ControlFile, you cannot restore it. Fix this by updating the ControlFile: RMAN>catalog device type 'sbt_tape' backuppiece '<piece handle>';
• Besides, the Recovery Catalog is mandatory when you use RMAN in a Data Guard environment.
• Check with the DBA that the Recovery Catalog does not run from the RAC nodes. I've seen DBAs do this and this can create major headaches and confusion during NetBackup restores. The Recovery Catalog should really run elsewhere from a different database.
• You can and should also backup the Recovery Catalog database.

Using RMAN scripts:

• I don't really explain the reasonings much on the points below because they are mostly covered in the NetBackup Oracle admin guide. I'd be just repeating.
• Pro: You can do Load Balanced backups from multiple RAC nodes at the same time.
• Con: You will need at least 2 policies.
• Con: This path makes it very difficult to use the "Override policy storage selection" method to force certain days to use different SLPs. Possible, but you will need way more than just 2 policies to get this to work.
• The first policy should have:
  • No "Application Backup" Schedules.
  • One Full, and maybe one or more additional Differential and/or Cumulative Schedules (Sometimes called "Automatic Schedules"). Note: This is assuming you are using the sample RMAN scripts provided by NetBackup, which knows how to use the "NB_ORA" variable to make RMAN do Full, Differential or Cumulative backups depending on the Schedule that was triggered.
  • Client list should just include ONE VIP name from any RAC node. Don't include more than one.
  • The Backup Selection must include the RMAN script. This same script must be the same on all RAC nodes, in case the VIP fails over.
• The second policy should have:
  • One "Application Backup" Schedule, with the actual retention or SLP you want.
  • Client list must include both RAC nodes (assuming in this example you have two). VIP or Actual name doesn't really matter. It all depends on what name(s) you put in for the "NB_ORA_CLIENT" variable in your RMAN script. Don't need to worry about the bp.conf client name.
  • The Backup Selection should be empty, because this policy exists only to have a place to put the backups sending its way.
• A possible third policy:
  • Policy configs mostly similar to the first policy, but for when you need frequent Archived Redo Log backup and deletion. E.g., every 15min.
  • Use the Full schedule. What it does actually just depends on what you wrote in your script. Set the frequency to every 15min or something.
  • In the Backup Selection, include a different RMAN script that only does Archived Redo Log backup and deletion. Again, this same script must be the same on all RAC nodes, in case the VIP fails over.
• About the Load Balanced RMAN script.
  • No, the "su - oracle" trick to skip the oracle sysdba user plain text password does not work if you do the Load Balanced RMAN syntax. I tried. If you know of a way, please share.
  • To Load Balance the backup, the key is just these lines (add more lines if you have more nodes or want more channels):
    • ALLOCATE CHANNEL CH1 TYPE 'SBT_TAPE' PARMS 'ENV=(NB_ORA_CLIENT=vip1, NB_ORA_POLICY=OracleReceivePolicy, NB_ORA_SCHED=OracleReceive)' CONNECT='sys/password@node1';
      ALLOCATE CHANNEL CH2 TYPE 'SBT_TAPE' PARMS 'ENV=(NB_ORA_CLIENT=vip2, NB_ORA_POLICY=OracleReceivePolicy, NB_ORA_SCHED=OracleReceive)' CONNECT='sys/password@node2';
  • Where:
    • vipX = your VIP names for each node.
    • NB_ORA_POLICY = The policy with the Application Backup schedule.
    • NB_ORA_SCHED = Name of that schedule.
    • sys = any user with the SYSDBA privilege
    • password = the terrible plain text password in this script
    • nodeX = the RAC node actual IP names.
• About the NetBackup Catalog when using the Load Balaced method.
  • That whole mess the NetBackup Oracle admin guide wrote about how to centralize the backup image catalog entries by using all these "add_alias" and "altnames" steps is optional. I actually do not recommend doing all that just to get the same "name" in the catalog that represents the whole RAC cluster. It is perfectly fine to have the RAC Load Balanced backup scattered across different names in the NetBackup Catalog. Reminder: This happends because the "Application Backup" policy accepts the same RAC backup from two (or more) clients.
  • To restore from a Load Balanced backup where the catalog name is scattered, just include all nodes in the restore script. E.g.:
    • ALLOCATE CHANNEL t1 TYPE 'SBT_TAPE' PARMS='ENV=(NB_ORA_CLIENT=vip1)';
      ALLOCATE CHANNEL t2 TYPE 'SBT_TAPE' PARMS='ENV=(NB_ORA_CLIENT=vip2)';

Hi Marianne,

Keeping this at a high level we run several large multi node RAC clusters and we haven't had any issues. I'm not in a position to share documentation but I would say you need to liase with your DBA's to get them to think about how they are going to approach protecting data with RMAN and get them to configure the RMAN scripts.

We also backup the FRA as the backup images are only kept for a short time in this area. My advice would be to stay well away from OIP as even after getting it to backup all ok (if at all possible) you can get real problems restoring. After spending all the money, time and effort on RAC I would say to try and bandaid OIP on the side is not the right approach. My personal advice would be:

DO

• Use RMAN scripts with a Oracle type policy- your DBA's should be able to write these
• Backup your archive / redo logs seperately
• Spend time to think about how the Oracle level 1 & 0 backups allign with any full or incremental schedules you will create
• Try and use FILES PER SET = 1 for DB images as this will give better performance and dedupe ratio in most cases
• Put the onus on the DBA's to manage the protection of the databases and knowledge share information around the Netbackup back end

It really is more of a Oracle backup strategy rather than Netbackup. In this case NetBackup will just be the bucket you pour data in and out of. RMAN and it's recovery catalogue is at the heart of the solution and not NetBackup.

Hello Marianne,

I dont have project experience with FRA backups, but I have performed quick test and it seems that adding a command "BACKUP [FORCE] RECOVERY AREA" into a legacy based script should be enough - OIP with FRA in Backup Selection generates this command into parent Job detail.

FORCE means that a certain file from FRA is backed up again even if it has already been backup up before.

IMHO when a third-party backup solution with like NetBackup will be used for direct backups to sbt then additional backups of FRA does not make much sense (it will be just backup of backups).

Regards

<deleted double post>