Oracle backup failed

[Ajith_Sankar]

Hi Folks

I have oracle server which is in cluster with two nodes, active and passive. I configured policy with cluster name. When i checked the client property of the clustername, it shows the client name as node2.But when i start the backup it failed with sc6. I tried the db backup with only node1 in policy,it was success.But i am changing the clientname in the script and initiating the backup for node2, it failed again with sc6. And the script output file shows below.Also attached bphdb log.

I believe the oracle instance is not here in node2 and it is active in the node1. Even i checked with oracle dba he confirmed node1 is active. But the client propery of the clustername displays the clientname as node2 that means activenode is node2. Confused. 

Script "D:\DB_script\dcaabsdb_Daily_bkp1.cmd" 
==== started on Mon 12/08/2014  08:07 ==== 

Recovery Manager: Release 11.2.0.1.0 - Production on Mon Dec 8 20:07:25 2014

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21> 22> 23> 24> 25> 26> 27> 28> 29> 30> 
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of allocate command at 12/08/2014 20:07:26
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist

RMAN> 

Recovery Manager complete.
Script "D:\DB_script\dcaabsdb_Daily_bkp1.cmd" 
==== ended in error on Mon 12/08/2014  08:07 ==== 

[RiaanBadenhorst]

Hi Ajith,

You can only initiate the backup from where Oracle is active, its on node 1, you can backup from node1. To make you're life easier and not having to worry about which node its on, you use the virtual name as it would move between the nodes.

In the script you can add the parameter called NB_ORA_CLIENT which would tell nbu which client to associate the backup with.

[Ajith_Sankar]

Hi Riaan, 

What is happening is, when i initiate the backup with cluster name(absdbclus), it is automatically picking the node2 and failing with sc 6. In script i mentioned the clustername only. i dont understand why the netbackupis not abl to identify and pick active node. Normally when we check in the client property of clustername, the clientname would be active node. But in this case it is node2 which is not active.

12/7/2014 9:10:11 PM - Info nbjm(pid=11088) starting backup job (jobid=381762) for client ABSDBCLUS, policy DCAABSDB_DAILY_FULL, schedule DCAABSDB_Daily_Full  
12/7/2014 9:10:11 PM - Info nbjm(pid=11088) requesting MEDIA_SERVER_WITH_ATTRIBUTES resources from RB for backup job (jobid=381762, request id:{886A391F-A056-4CF0-8587-8432BCE99A3B})  
12/7/2014 9:10:11 PM - requesting resource srt3symapl02-msdp-1
12/7/2014 9:10:11 PM - requesting resource dcabak02.dca.com.NBU_CLIENT.MAXJOBS.ABSDBCLUS
12/7/2014 9:10:11 PM - requesting resource dcabak02.dca.com.NBU_POLICY.MAXJOBS.DCAABSDB_DAILY_FULL
12/7/2014 9:10:11 PM - granted resource dcabak02.dca.com.NBU_CLIENT.MAXJOBS.ABSDBCLUS
12/7/2014 9:10:11 PM - granted resource dcabak02.dca.com.NBU_POLICY.MAXJOBS.DCAABSDB_DAILY_FULL
12/7/2014 9:10:11 PM - granted resource srt3symapl02-msdp-1
12/7/2014 9:10:11 PM - estimated 0 Kbytes needed
12/7/2014 9:10:11 PM - Info nbjm(pid=11088) started backup (backupid=ABSDBCLUS_1417972211) job for client ABSDBCLUS, policy DCAABSDB_DAILY_FULL, schedule DCAABSDB_Daily_Full on storage unit srt3symapl02-msdp-1
12/7/2014 9:10:12 PM - started process bpbrm (52397)
12/7/2014 9:10:13 PM - connecting
12/7/2014 9:10:14 PM - Info bpbrm(pid=52397) ABSDBCLUS is the host to backup data from     
12/7/2014 9:10:14 PM - Info bpbrm(pid=52397) reading file list for client        
12/7/2014 9:10:14 PM - Info bpbrm(pid=52397) starting bphdb on client         
12/7/2014 9:10:14 PM - connected; connect time: 0:00:01
12/7/2014 9:10:15 PM - Info bphdb(pid=7648) Backup started           
12/7/2014 9:10:16 PM - Error bpbrm(pid=52397) from client ABSDBCLUS: ERR - Script <"D:\DB_script\dcaabsdb_Daily_bkp.cmd"> failed with status: <1>  
12/7/2014 9:10:16 PM - Error bpbrm(pid=52397) from client ABSDBCLUS: ERR - bphdb exit status = 6: the backup failed to back up the requested files
12/7/2014 9:10:16 PM - end writing
12/7/2014 9:10:17 PM - Info bphdb(pid=7648) done. status: 6: the backup failed to back up the requested files 
the backup failed to back up the requested files(6)

[Will_Restore]

Double-check name resolution.  ABSDBCLUS is somehow hardcoded with IP for node2 ?

[Yasuhisa_Ishika]

How "bpclntcmd -hn ABSDBCLUS" shows on both master and media server? Can you get cluster IP?

[RiaanBadenhorst]

As the other members stated, if you open the host properties of virtual name, you should actually end up opening the active node. If its not, then you've got some type of resolution issue.

Test

nslookup virtual_name

bpclntcmd -hn virtual_name <<<<these two results should show the same IP.

mstsc to the virtual_name and see where you end up.

run ipconfig on both nodes and see what is listed

If this is an MSCS cluste don't get confused between the "cluster name" and the "oracle virtual name" its two separate names and IPs.

[Ajith_Sankar]

Riaan,

I got one clue from db admin that the quorum is active and running in node2 and database sevices is active node1. This may be the reason ? 

Checked the name resolution and all looks ok. 

The RDP for Virtual name logins to node2.

[RiaanBadenhorst]

Yes, that sounds like it, there should be two virtual IP's (and names), one for cluster and one for Oracle. What you're connecting to now is the incorrect one. Take a look at the group that is made for Oracle in MSCS, there should be an IP. Use that one.

[Ajith_Sankar]

This has been fixed after rebooting the node2 as the quorum moved to node1 post reboot. Now both quorum and database on node1 and db backups are successfull. In client propery, the client name is showing node1.

[RiaanBadenhorst]

That is not a solution. The quorum and Oracle are not related and will not always be on the same node. You need to add an IP address to your Oracle Service group.

[Marianne]

I agree with Riaan - extremely important to separate the vrtual IP and hostname for the MS cluster from the Oracle Instance / Cluster Group. 

The NBU policy Client name and NB_ORA_CLIENT parameter in rman script MUST point to the Oracle Cluster Group hostname/Vitual IP.

[Ajith_Sankar]

Riaan, You mean appart from cluster IP and name, need to assign new ip and name for Oracle service group? and configure that new name in the policy ? We also have 6-7 db servers in clusters and all are configured with single cluster name and ip and backup configured with cluster name. 

[RiaanBadenhorst · Accepted solution]

When you set up a MSCS cluster (or FOC) it would aks you for a clustername, and IP. But this is as the name suggests, for the cluster, not for the application. It is for you to connect to for management purposes. Suppose you have a cluster with 2 Oracle Databases in 2 groups. If you'd like to run these one of these groups on each server (which I believe MSCS can do same as VCS) then how would people connect to the oracle group that is not co-located with the Quorum / Cluster name?

That is why you have an IP / Name in the group for the application you're clustering and NOT in some other unrelated group. Clustering is supposed virtualise the application so that it can move freely between nodes without the users having to worry about where it ir residing. In your config that is not achieved.

So, Cluster names/IPs are for the cluster, Database (oracle) names/IPs are for Oracle.

Hope that helps :)

[RiaanBadenhorst]

Since the IP/Name is already used by Oracle clients I suggest moving that IP/Name to the Oracle Group and Adding new ones for the Cluster. Then clients don't need to be reconfigured.

[Ajith_Sankar]

Hi Riaan and Marianne, Thank You for your valuable suggestions, i will try to communicate the same to Oracle DBA