Rman backup failed with status 6, child job with error 21 (Repost)

[SSOESVR]

Hi All, 

Reposting my thread thatt have gone cold, we need a solution for this problem we are having:

I have a case of Netbackup having problem backing up oracle database on Solaris 10 client.

The parent job will fail with eror status 6, with 1 or 2 child job having error code status 21 on creating data socket portion.

Majority of the child jobs passed.  

From the RMAN output logs, we are also seeing some errors as below

The DB itself seem to have completed backup, but is having error on backing up the controlfile portion

Have attached the following logs:

- On media server: bpbrm
- On client: bpcd and dbclient

Pls let me know if you have any insight into what is happening

==============

Error on child Job

==============

11/7/2012 5:49:21 PM - started process bpbrm (2160)

11/7/2012 5:49:23 PM - connecting

11/7/2012 5:55:25 PM - Error bpbrm(pid=2160) cannot create data socket, The operation completed successfully.  (0)   

11/7/2012 5:55:25 PM - end writing

socket open failed(21)

========================

RMAN OUTPUT

========================

Script /softwares/nbu/rman_bkup/hot_database_backup.sh

==== started on Fri Nov 2 12:36:56 SGT 2012 ====

....

.....

....

....

.....

....

channel ch01: starting piece 1 at 02-NOV-12

channel ch00: finished piece 1 at 02-NOV-12

piece handle=bk_4332_1_798295019 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:03:49

channel ch01: finished piece 1 at 02-NOV-12

piece handle=bk_4334_1_798295245 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:35

Finished backup at 02-NOV-12

Starting Control File and SPFILE Autobackup at 02-NOV-12

piece handle=c-3995589581-20121102-09 comment=API Version 2.0,MMS Version 5.0.0.0

Finished Control File and SPFILE Autobackup at 02-NOV-12

sql statement: alter system archive log current

released channel: ch00

released channel: ch01

allocated channel: ch00

channel ch00: SID=23 device type=SBT_TAPE

channel ch00: Veritas NetBackup for Oracle - Release 7.0 (2010070723)

allocated channel: ch01

channel ch01: SID=767 device type=SBT_TAPE

channel ch01: Veritas NetBackup for Oracle - Release 7.0 (2010070723)

Starting backup at 02-NOV-12

current log archived

channel ch00: starting archived log backup set

channel ch00: specifying archived log(s) in backup set

input archived log thread=1 sequence=2029 RECID=1867 STAMP=798295408

channel ch00: starting piece 1 at 02-NOV-12

channel ch01: starting archived log backup set

channel ch01: specifying archived log(s) in backup set

input archived log thread=1 sequence=2030 RECID=1868 STAMP=798295408

channel ch01: starting piece 1 at 02-NOV-12

channel ch00: finished piece 1 at 02-NOV-12

piece handle=al_4336_1_798295409 tag=TAG20121102T124329 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:01:05

channel ch00: deleting archived log(s)

archived log file name=/orasw/app/oracle/fast_recovery_area/IAMSUAT/archivelog/2012_11_02/o1_mf_1_2029_896mvj03_.arc RECID=1867 STAMP=798295408

channel ch01: finished piece 1 at 02-NOV-12

piece handle=al_4337_1_798295409 tag=TAG20121102T124329 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch01: backup set complete, elapsed time: 00:01:05

channel ch01: deleting archived log(s)

archived log file name=/orasw/app/oracle/fast_recovery_area/IAMSUAT/archivelog/2012_11_02/o1_mf_1_2030_896mvjwh_.arc RECID=1868 STAMP=798295408

Finished backup at 02-NOV-12

Starting Control File and SPFILE Autobackup at 02-NOV-12

piece handle=c-3995589581-20121102-0a comment=API Version 2.0,MMS Version 5.0.0.0

Finished Control File and SPFILE Autobackup at 02-NOV-12

released channel: ch00

released channel: ch01

allocated channel: ch00

channel ch00: SID=23 device type=SBT_TAPE

channel ch00: Veritas NetBackup for Oracle - Release 7.0 (2010070723)

Starting backup at 02-NOV-12

channel ch00: starting full datafile backup set

channel ch00: specifying datafile(s) in backup set

including current control file in backup set

channel ch00: starting piece 1 at 02-NOV-12

channel ch00: finished piece 1 at 02-NOV-12

piece handle=cntrl_4339_1_798295541 tag=TAG20121102T124541 comment=API Version 2.0,MMS Version 5.0.0.0

channel ch00: backup set complete, elapsed time: 00:00:55

Finished backup at 02-NOV-12

Starting Control File and SPFILE Autobackup at 02-NOV-12

released channel: ch00

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of Control File and SPFILE Autobackup command on ch00 channel at 11/02/2012 12:54:03

ORA-19509: failed to delete sequential file, handle="c-3995589581-20121102-0c", parms=""

ORA-27027: sbtremove2 returned error

ORA-19511: Error received from media manager layer, error text:

   Failed to remove, c-3995589581-20121102-0c, from image catalog.

ORA-19513: failed to identify sequential file

ORA-27206: requested file not found in media management catalog

RMAN> 

Recovery Manager complete.

Script /softwares/nbu/rman_bkup/hot_database_backup.sh

==== ended in error on Fri Nov 2 12:54:04 SGT 2012 ====

[Yasuhisa_Ishika]

It seems that Error occured in Control File and SPFILE Autobackup. Can you try to disable Control File and SPFILE Autobackup? In general, SPFILE will be backed up in database backup. And you configured control backup in your script. I think you don't have to enable autobackup.