ORA-19506 ORA-27028 ORA-19511

NBU: 7.5 

OS: AIX 6.1

DB: oracle 10.2.0.5 RAC

This error message appears when I am trying level 0 backup.

I do not know how to fix it,please help me,thanks

released channel: ch01
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch00 channel at 10/16/2014 23:17:40
ORA-19506: failed to create sequential file, name="al_60179_1_861144290", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve

1 Solution

Accepted Solutions
Accepted Solution!

Hi,   Is there any reason

Hi,

 

Is there any reason you're using "SEND 'NB_ORA_CLIENT= XXXXX';" only for the archive logs? Are they not shared between the hosts?

 

I think that is the issue. You're only doing a "SEND 'NB_ORA_CLIENT" for 1 host, but you're allocating channels from both instances. Can you try and use a config such as below, or alternatively only allocate channels from the host you've put in the "SEND 'NB_ORA_CLIENT"

 

ALLOCATE CHANNEL 1 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 2 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';
ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';
16 Replies

Show us the bphdb and

Show us the bphdb and dbclient logs

also run on the client

bpclntcmd -pn

bpclntcmd -self

from master/media

bptestbpcd -client "client name"

Your Oracle client cannot

Your Oracle client cannot communicate with master server.

Check forward and reverse lookup between master and client as per Riaan's post above and ensure port 1556 is open in both directions.

# ./bpclntcmd -pn expecting

# ./bpclntcmd -pn expecting response from server idc-nbu ydyx02 ydyx02 10.150.8.13 42089 # ./bpclntcmd -self yp_get_default_domain failed: (6) internal yp server or client error NIS does not seem to be running: (1) args to yp function are bad gethostname() returned: ydyx02 host ydyx02: ydyx02 at 10.150.8.13 aliases: ydyx02 10.150.8.13 getfqdn: Error 0 D:\Program Files\Veritas\NetBackup\bin>bptestbpcd -client ydyx02 1 1 1 10.150.31.131:65359 -> 10.150.8.13:1556 10.150.31.131:65360 -> 10.150.8.13:1556 10.150.31.131:65362 -> 10.150.8.13:1556 The 1556 port is open in both directions.(TELNET 1556 port on MASTER and CLIENT is ok) This failure appeared when the script has backuped some datafiles,not started to fail. And the most time,this backup completed successfully,this error just appears sometimes.

Might be a timeout issue

Might be a timeout issue then. Does it always fail on the archive logs?

Okay - this is a RAC

Okay - this is a RAC backup.

Which Client name is used in the backup policy? Local client name ydyx02 or virtual hostname?
What is CLIENT_NAME in bp.conf on client?
Is there a NB_ORA_CLIENT name defined in the rman script?

Other than dbclient log on the client, we also need bprd log on the master server to see incoming request from client IP address and to see if master can resolve this IP to hostname that corresponds with Client name in the policy.

bprd log folder does not exist by default on the master server - create the folder under /usr/openv/netbackup/logs and restart NBU to enable the log.
dbclient log folder on RAC nodes needs 777 permissions.

This error mostly appears in

This error mostly appears in archivelog backup. 1.ydyx02 is used in the backup policy. 2.# hostname ydyx02 # cat bp.conf SERVER = idc-nbu SERVER = ydyx02 CLIENT_NAME = ydyx02 CONNECT_OPTIONS = localhost 1 0 2 USE_VXSS = PROHIBITED VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY EMMSERVER = idc-nbu HOST_CACHE_TTL = 3600 TELEMETRY_UPLOAD = YES 3.It did not defined the NB_ORA_CLIENT in the rman script. 4.I just enabled the logs,and started a archivelog backup,but the backup ended in success. the attachments are the dbclient log and bprd log

The dbclient log only shows

The dbclient log only shows successful backup of one archive log and control file.

We need to see logs from a failed backup.

After next failure, post the text in Details tab of failed job along with new dbclient and bprd logs.

You may want to have a look at the RAC section of for Oracle Administrator's Guide:  http://www.symantec.com/docs/DOC5175 
and find out from Oracle DBAs how archive logs are configured among RAC nodes and see if NBU policies and backup scripts are configured as per recommendations in the manual.

Something else to look at is timeouts on backup server(s).
CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT for large Oracle db's should be set to 1800.

The CLIENT_READ_TIMEOUT and

The CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT  both be set as 9000 before.

Now, there is another 2 nodes RAC system appears the same error.And the backup were all failed from 2014-10-18 to now.

The ports were opened in both direction.

[root@SOCdb2 ~]# hostname
SOCdb2
[root@SOCdb2 ~]# cat /nbudata/openv/netbackup/bp.conf
SERVER = idc-nbu
SERVER = SOCdb2
CLIENT_NAME = SOCdb2

SOCdb2 is used in the backup policy,It did not defined the NB_ORA_CLIENT in the rman script.

The logs from a failed backup were as the attchments.

(the node2 logs contains more than one failure backup logs.)

 

We need bprd log on the

We need bprd log on the master server.

This si the bprd log on

This si the bprd log on master server.
 

Hi,   HAve you checked the

Hi,

 

HAve you checked the Best practices section for Oracle RAC in the admin guide? Which scenarios matches yours? Are you using a VIP? Are you using load balancing?

 

Show us your backup scripts please.

The backups are not using

The backups are not using VIP

These are the backup scripts.

Accepted Solution!

Hi,   Is there any reason

Hi,

 

Is there any reason you're using "SEND 'NB_ORA_CLIENT= XXXXX';" only for the archive logs? Are they not shared between the hosts?

 

I think that is the issue. You're only doing a "SEND 'NB_ORA_CLIENT" for 1 host, but you're allocating channels from both instances. Can you try and use a config such as below, or alternatively only allocate channels from the host you've put in the "SEND 'NB_ORA_CLIENT"

 

ALLOCATE CHANNEL 1 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 2 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';
ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=vipname1)' CONNECT='sys/passwd@vipname1';
ALLOCATE CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=vipname2)' CONNECT='sys/passwd@vipname2';

hello,I have changed the

hello,I have changed the backup script like this,but it did not work either. $RMAN target / catalog rman/password@idccata msglog $RMAN_LOG_FILE append << EOF RUN { ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS='ENV=(NB_ORA_CLIENT=SOCdb1)' connect "sys/password@bhm161"; SEND 'NB_ORA_CLIENT=SOCdb1'; ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE' PARMS='ENV=(NB_ORA_CLIENT=SOCdb2)' connect "sys/password@bhm162"; SEND 'NB_ORA_CLIENT=SOCdb2'; BACKUP filesperset 100 FORMAT 'al_%s_%p_%t' ARCHIVELOG ALL DELETE INPUT; RELEASE CHANNEL ch00; RELEASE CHANNEL ch01; ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE'; BACKUP # recommended format FORMAT 'cntrl_%s_%p_%t' CURRENT CONTROLFILE; RELEASE CHANNEL ch00; }

Try with one server/node at

Try with one server/node at first

Me I would move to OIP , it

Me I would move to OIP , it just makes life much more simple and if thats not avail to you at your current level, i'd just simplify as per Riaan above. Whilst you might have reasons for it, personally I'd split out the clients into discrete scripts/policies and leave them there forever!

If this is how rac is supposed to be handled, then do please ignore my comments.

Jim