6) the backup failed to back up the requested file. But RMAN error ORA-27192:

[turguns]

Hi All,

Netbackup 7.1 Linux 2.6 (Master,Media,Client)

We have like this issue:

1.

The oracle backup with 3 stream. All jobs writing successfully. But Parent jobs fails with error (6) the backup failed to back up the requested file)

From bpbrm logs also positive feedback regarding backup job

Exit: client backup EXIT STATUS 0: the requested operation was successfully completed

2.

From RMAN like the following error

RMAN-03009: failure of backup command on ch03 channel at 03/25/2016 08:41:35
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
continuing other job steps, job failed will not be re-run

The message is non RMAN, but media manager or vendor specific failure ))

From this site also saying: This is not an Oracle error per-se, it has to do with the disk or tape. (http://www.dba-oracle.com/t_ora_27192.htm)

But  to this tape  backups from many different hosts are working without problem. even on that same volume pool.

Any idea ?

Thanks in advance,

Turgun

[Marianne]

Have a look at this TN: http://www.veritas.com/docs/000039562 You will need logs (see TN) to see if you are experiencing the same issue.

[turguns]

Hi Marianne,

First of all I want to point, the standart backup is working normal.

According to the dbclient logs, your last post is a  bit different .

I attached full logs of dbclient,bphdb,rman_scripts and its output.

I've set client read timeout to 1800

connections between client,media and master server are okay.

From Master and Media server

bpclntcmd -hn cdcdbn-vip
host cdcdbn-vip: cdcdbn-vip at 10.10.72.90
aliases:     cdcdbn-vip     10.10.72.90

From client
To Media-Server: bpclntcmd -hn fujitsu
host fujitsu: fujitsu at 10.10.72.101
aliases:     fujitsu     10.10.72.101

To Master-Server: bpclntcmd -hn nbumedia
host nbumedia: nbumedia at 10.10.8.92
aliases:     nbumedia     10.10.8.92

As I mentioned before, child jobs are finished successfully, But parent job is with error :(6) the backup failed to back up the requested files

From dbclient ,it is starting like this :

00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block

00:17:20.565 [10084] <2> int_LogSystemInfo: INF -
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
        System name:    Linux
        Node name:      cdcdb1.intra
        Release:        3.8.13-68.3.4.el6uek.x86_64
        Version:        #2 SMP Tue Jul 14 15:03:36 PDT 2015
        Machine:        x86_64
        User name:      oracle
        Client Host:    cdcdbn-vip

00:17:20.565 [10084] <2> int_GetMMInfo: INF - Initialized Signal
00:17:20.565 [10084] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
00:17:21.122 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:21.122 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf

Here Why is it  trying to fetch from /home oracle/bp.conf .

:

Ending like this

00:59:18.336 [10084] <2> logconnections: BPRD CONNECT FROM 10.10.72.81.32526 TO 10.10.8.92.1556 fd = 21
00:59:18.491 [10084] <4> VxBSAQueryObject: INF - No match was found for query
00:59:18.491 [10084] <2> int_FindBackupImage: INF - 24r1dgud_1_1 not found
00:59:18.491 [10084] <4> int_RemoveImage: INF - Backup file, 24r1dgud_1_1, not found in image catalog.
00:59:28.317 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002

00:59:28.332 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:59:28.332 [10085] <4> sbtend: INF - --- END of SESSION ---

bphdb ending like this

00:59:29.101 [10002] <16> bphdb: ERR - Script exited with status = 1 <the requested operation was partially successful>
00:59:29.101 [10002] <8> bphdb: WRN - Directory /usr/openv/netbackup/logs/bpubsora does not exist
00:59:29.101 [10002] <16> bphdb: ERR - bphdb exit status = 6: the backup failed to back up the requested files

RMAN script out ending like this :

RMAN-03009: failure of backup command on ch02 channel at 03/26/2016 00:39:42
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
continuing other job steps, job failed will not be re-run
released channel: ch01
released channel: ch02
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch01 channel at 03/26/2016 00:59:28
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.

BTW bp.conf is :

 more /usr/openv/netbackup/bp.conf
SERVER = nbumedia
SERVER = fujitsu
CLIENT_NAME = cdcdbn-vip
CONNECT_OPTIONS = localhost 0 0 2
CLIENT_READ_TIMEOUT = 1800

I Reinstalled the netbackup client and tried to change the hostname. No difference.

I've compared Rman script with others, no difference.

Any idea?

Thanks,

Turgun

[Marianne]

You most definitely have this error in dbclient log: <16> readCommFile: ERR - timed out after 900 seconds while reading from There are other similarities as well. There may be more similarities with level 5 logs. The TN also refers to (level 5) bptm and bpbrm logs. I would say that your issue is the same. Increasing client read timeout is not going to help. You will see in the TN that the issue with Oracle and that NBU 7.5.0.5 and higher works different and may fix the issue.

[turguns]

Dear Marianne, thank you for taking  time to analyse my post.

What is the TN?  Is it dbclient ?

[Marianne]

Have you even opened the URL? The logs used for troubleshooting are clearly mentioned in bold letters.

[turguns]

yeah, I've opened it even yesterday. Just I've asked the definitaion of TN J.

You are right. The issue is same. (client read timeout ). Because I see  from dbext/logs/  that it also pointing  the (Client read timeout = 300).

On your link  about the resolution:
Four sites have contacted Oracle support and have reported the problem resolved after upgrading Oracle components. 

The NetBackup 7.5.0.5 client contains a design change that minimizes the opportunity for this issue to occur.   Previously, the Oracle child process would inherit the socket if forked/created between sbtbackup and sbtclose processing.  That window has been narrowed to a few milliseconds during sbtbackup processing.
 

As per my understand ,there is nothing to change/modify on netbackup side. Just play with Oracle components .

Thanks,

I will get back later on that.

[Marianne · Accepted solution]

I see in the TN that NBU works different as from version 7.5.0.5 that minimizes chances for issue to occur. So, my suggestion is to ask Oracle team to escalate the issue to Oracle Support while you start planning to upgrade NBU in your environment. Support for all versions up to 7.6.x will reach EOSL beginning of next year.

[Michael_G_Ander]

There certainly is a problem if the rman update of control file or rman catalog takes 20 minutes.

Other than that, this looks to me as a lack of resource problem.

You need to figure out what is taking 20 minutes after the last stream has finished.

If it is lack of netbackup resources, CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT on both client and servers can migate some the problem.

[turguns]

Thanks a lot.

As Marianne suggested, after version 7.5.0.5 , issue was solved.

BR,

Turgun