10-16-2014 07:07 PM
NBU: 7.5
OS: AIX 6.1
DB: oracle 10.2.0.5 RAC
This error message appears when I am trying level 0 backup.
I do not know how to fix it,please help me,thanks
released channel: ch01
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch00 channel at 10/16/2014 23:17:40
ORA-19506: failed to create sequential file, name="al_60179_1_861144290", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
Solved! Go to Solution.
10-22-2014 02:33 AM
Hi,
Is there any reason you're using "SEND 'NB_ORA_CLIENT= XXXXX';" only for the archive logs? Are they not shared between the hosts?
I think that is the issue. You're only doing a "SEND 'NB_ORA_CLIENT" for 1 host, but you're allocating channels from both instances. Can you try and use a config such as below, or alternatively only allocate channels from the host you've put in the "SEND 'NB_ORA_CLIENT"
ALLOCATE CHANNEL 1 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 2 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';
ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';
10-16-2014 09:43 PM
Show us the bphdb and dbclient logs
also run on the client
bpclntcmd -pn
bpclntcmd -self
from master/media
bptestbpcd -client "client name"
10-16-2014 09:51 PM
Your Oracle client cannot communicate with master server.
Check forward and reverse lookup between master and client as per Riaan's post above and ensure port 1556 is open in both directions.
10-17-2014 12:10 AM
10-17-2014 12:34 AM
Might be a timeout issue then. Does it always fail on the archive logs?
10-17-2014 12:55 AM
Okay - this is a RAC backup.
Which Client name is used in the backup policy? Local client name ydyx02 or virtual hostname?
What is CLIENT_NAME in bp.conf on client?
Is there a NB_ORA_CLIENT name defined in the rman script?
Other than dbclient log on the client, we also need bprd log on the master server to see incoming request from client IP address and to see if master can resolve this IP to hostname that corresponds with Client name in the policy.
bprd log folder does not exist by default on the master server - create the folder under /usr/openv/netbackup/logs and restart NBU to enable the log.
dbclient log folder on RAC nodes needs 777 permissions.
10-19-2014 07:45 PM
10-19-2014 10:46 PM
The dbclient log only shows successful backup of one archive log and control file.
We need to see logs from a failed backup.
After next failure, post the text in Details tab of failed job along with new dbclient and bprd logs.
You may want to have a look at the RAC section of for Oracle Administrator's Guide: http://www.symantec.com/docs/DOC5175
and find out from Oracle DBAs how archive logs are configured among RAC nodes and see if NBU policies and backup scripts are configured as per recommendations in the manual.
Something else to look at is timeouts on backup server(s).
CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT for large Oracle db's should be set to 1800.
10-20-2014 01:21 AM
The CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT both be set as 9000 before.
Now, there is another 2 nodes RAC system appears the same error.And the backup were all failed from 2014-10-18 to now.
The ports were opened in both direction.
[root@SOCdb2 ~]# hostname
SOCdb2
[root@SOCdb2 ~]# cat /nbudata/openv/netbackup/bp.conf
SERVER = idc-nbu
SERVER = SOCdb2
CLIENT_NAME = SOCdb2
SOCdb2 is used in the backup policy,It did not defined the NB_ORA_CLIENT in the rman script.
The logs from a failed backup were as the attchments.
(the node2 logs contains more than one failure backup logs.)
10-20-2014 01:50 AM
We need bprd log on the master server.
10-20-2014 02:00 AM
This si the bprd log on master server.
10-20-2014 05:55 AM
Hi,
HAve you checked the Best practices section for Oracle RAC in the admin guide? Which scenarios matches yours? Are you using a VIP? Are you using load balancing?
Show us your backup scripts please.
10-21-2014 08:49 PM
The backups are not using VIP
These are the backup scripts.
10-22-2014 02:33 AM
Hi,
Is there any reason you're using "SEND 'NB_ORA_CLIENT= XXXXX';" only for the archive logs? Are they not shared between the hosts?
I think that is the issue. You're only doing a "SEND 'NB_ORA_CLIENT" for 1 host, but you're allocating channels from both instances. Can you try and use a config such as below, or alternatively only allocate channels from the host you've put in the "SEND 'NB_ORA_CLIENT"
ALLOCATE CHANNEL 1 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 2 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';
ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';
10-30-2014 06:16 PM
10-30-2014 10:52 PM
Try with one server/node at first
10-31-2014 09:28 AM
Me I would move to OIP , it just makes life much more simple and if thats not avail to you at your current level, i'd just simplify as per Riaan above. Whilst you might have reasons for it, personally I'd split out the clients into discrete scripts/policies and leave them there forever!
If this is how rac is supposed to be handled, then do please ignore my comments.
Jim