Oracle fullbackups status 6

[Fredrik_Dahlber]

HI, please help a Tech in distress. My full backup of some Oracle db ends with 6. so again please advice.. the dbclient log (verbose 2) says nothing to help.

channel ch00: starting piece 1 at 11-OCT-12
channel ch02: finished piece 1 at 11-OCT-12
piece handle=bk_34653_1_796353122 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch02: backup set complete, elapsed time: 01:19:53
channel ch02: starting incremental level 0 datafile backup set
channel ch02: specifying datafile(s) in backup set
input datafile file number=00029 name=/db/durtvaORA/w/index/prod_idxlogg_w_1_1.dbf
input datafile file number=00054 name=/db/durtvaORA/x/index/prod_idxlogg_x_1_1.dbf
channel ch02: starting piece 1 at 11-OCT-12
RMAN-03009: failure of backup command on ch02 channel at 10/11/2012 02:49:12
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: Error received from media manager layer, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
continuing other job steps, job failed will not be re-run
channel ch03: finished piece 1 at 11-OCT-12
piece handle=bk_34654_1_796354438 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch03: backup set complete, elapsed time: 01:22:18
channel ch01: finished piece 1 at 11-OCT-12
piece handle=bk_34655_1_796354584 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch01: backup set complete, elapsed time: 01:24:03
channel ch00: finished piece 1 at 11-OCT-12
piece handle=bk_34656_1_796357240 tag=HOT_DB_BK_LEVEL0 comment=API Version 2.0,MMS Version 5.0.0.0
channel ch00: backup set complete, elapsed time: 00:40:26
released channel: ch00
released channel: ch01
released channel: ch02
released channel: ch03
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on ch02 channel at 10/11/2012 02:49:12
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: Error received from media manager layer, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
RMAN-03009: failure of backup command on ch03 channel at 10/10/2012 19:12:30
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: Error received from media manager layer, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
RMAN-03009: failure of backup command on ch02 channel at 10/10/2012 19:17:11
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: Error received from media manager layer, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
RMAN-03009: failure of backup command on ch01 channel at 10/10/2012 19:16:16
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: Error received from media manager layer, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.

RMAN> RMAN>

Recovery Manager complete.

Script /db/durtvaORA/sid1/etc/netbackup/hot_db_backup_durp01.sh
==== ended in error on Thu Oct 11 03:01:13 CEST 2012 ====

Hot_Database_Backup.sh.out lines 309-378/378 (END)

[Fredrik_Dahlber]

The master & media are  Linux and we back it up to a basic DISK SU

[Will_Restore]

review this technote:

Article URL http://www.symantec.com/docs/TECH189853

Cause

The oracle.exe process serves multiple purposes.  Each RMAN channel is a thread of oracle.exe, as are any extjob.exe tasks that are run from the SYS schema. 

Consequently, any extjob.exe task that starts while dbclient is connected to the media server processes will inherit copies of the file descriptors for the sockets.  If the extjob.exe task has not yet exited when dbclient closes it's copies of the file descriptors, the client host TCP stack will not send the TCP FINs as expected and the media server will be unaware that the backup is complete because the sockets have not yet closed.

If the extjob.exe completes within 15 minutes, a disordly shutdown of the sockets will occur (because extjob.exe does not know the sockets exist and hence can't perform an orderly close) resulting in the TCP RESETs.

If the extjob.exe does not complete for at least 15 minutes, then dbclient will timeout as noted above.

Solution

The behaviors documented above are not explicitly addressed within the Oracle SBT API and neither Oracle nor Symantec have a code level change to fix this situation at the current time.  However, several workarounds are available.

1) Use a scheduler other than extjob.exe for the conflicting tasks.

2) Schedule the external tasks to initiate at a time when RMAN backups are not running.

3) Configure the external tasks to run from an Oracle schema other than SYS.  When run from a non-SYS schema, the extjob.exe process will not be spawned directly from oracle.exe and will not conflict with the backup.  When run from the SYS schema, the thread is spawned using the parent oracle.exe process which is required to also be used by RMAN for the backups. 

Symantec Corporation has acknowledged that the above-mentioned issue (ETrack 2717058) is present in the current version(s) of the product(s) mentioned in this article.  Symantec Corporation is committed to product quality and satisfied customers.  This issue was scheduled to be addressed in a future maintenance release or update of the product.

[HEMANPR]

Hello

Are you using Windows 2008?

If yes, you need to add your netbackup user to the ora_dba group.

For more info check the oracle logs on drive:Program Files\Veritas\NetBackup\logs\user_ops\dbext\oracle

[Yasuhisa_Ishika]

Please provide more detail about your configuration like OS name and version, Oracle version, NetBackup version or so.

[Yasuhisa_Ishika]

Please tell us minor version of Oracle like 11.2.0.x. Tell us whether this is newly configured backup, or existing working backup as well.

BTW, your configuration seems irregal, version of media server is higher than master server.
Master server must have most highest(or equal) version.

[Fredrik_Dahlber]

Master&media&client =  SLES10sp4

db = Oracle 11g

NBU 7.1.0.4 media server

NBU 7.0.1 Client

NBU 7.1.0.3 Master server

[Fredrik_Dahlber]

The Oracle is 11.2.0.1 and Yes we know that it is not standard to have master with a lower version.

but our local symantec office told us that in 7.1 it not a must to have master with a greater version than media servers and clients.

Yes this is a working policy. it just has a great tendency to fail full backups, diff works just fine

[Yasuhisa_Ishika · Accepted solution]

Except the NetBackup client version I mentioned before, there are no configuration issue here.

Have you already enable debug logging and log a case with Symantec?
To identify what's wrong, debug logs of various processes are needed. For example, similar issues are reported as below. These case require various debug logs. 

http://www.symantec.com/docs/TECH189853
http://www.symantec.com/docs/TECH73130

[Fredrik_Dahlber]

Thanks I will look into it.

[Nelson_Thomas]

Mine was Windows :

Edited the SQLNET.AUTHENTICATION_SERVICES line and set it to NTS in the file on oracle server as below :- C:\oracle\product\10.2.0\db_1\network\ADMIN\sqlnet.ora

SQLNET.AUTHENTICATION_SERVICES= (NTS)
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

This solved my isse. This may help you.

[jim_dalton]

The media server is in advance of the master server re: NB version, thats a nono, as Yasuhisa says. Pick a different media server, or upgrade it. Give us some history...new , modified recently,...

Jim

[jmorehea]

I am haveing the same issue. NBU master/media version 7.1 on Solaris 10 and NBU client version 6.5.6 on Solaris 8.The issue is confirmed in the tech notes http://www.symantec.com/docs/TECH189853. We have all ready implemented the change from Solution 3 by creating a new username in the DB and adding it to the RMAN backup script but no resolve. How do you implement the solution 1? My DBA's are unclear on exactly what needs to be changed on Oracle 9i.

Solution

The behaviors documented above are not explicitly addressed within the Oracle SBT API and neither Oracle nor Symantec have a code level change to fix this situation at the current time.  However, several workarounds are available.

1) Use a scheduler other than extjob.exe for the conflicting tasks.

2) Schedule the external tasks to initiate at a time when RMAN backups are not running.

3) Configure the external tasks to run from an Oracle schema other than SYS.  When run from a non-SYS schema, the extjob.exe process will not be spawned directly from oracle.exe and will not conflict with the backup.  When run from the SYS schema, the thread is spawned using the parent oracle.exe process which is required to also be used by RMAN for the backups. 

[Will_Restore]

@jmorehea  per http://www.softpanorama.org/Admin/Job_schedulers/oracle_scheduler.shtml

There are two interfaces for the Oracle Scheduler:

• Oracle API (DBMS_SCHEDULER package).
• UI which is part of Oracle Enterprise Manager

so the suggestion is to not use these features, or use them only when RMAN job is not active 

[jmorehea]

Just noticed that this tech document is for Windows http://www.symantec.com/business/support/index?page=content&id=TECH189853. The Unix document is

http://www.symantec.com/business/support/index?page=content&id=TECH73130 and the work around does not not apply to us since it refers to ASM. Has anyone else had this issue on the Solaris environment.