6) the backup failed to back up the requested file. But RMAN error ORA-27192:

Have a look at this TN: http://www.veritas.com/docs/000039562 You will need logs (see TN) to see if you are experiencing the same issue.

Hi Marianne,

First of all I want to point, the standart backup is working normal.

According to the dbclient logs, your last post is a  bit different .

I attached full logs of dbclient,bphdb,rman_scripts and its output.

I've set client read timeout to 1800

connections between client,media and master server are okay.

From Master and Media server

bpclntcmd -hn cdcdbn-vip
host cdcdbn-vip: cdcdbn-vip at 10.10.72.90
aliases:     cdcdbn-vip     10.10.72.90

From client
To Media-Server: bpclntcmd -hn fujitsu
host fujitsu: fujitsu at 10.10.72.101
aliases:     fujitsu     10.10.72.101

To Master-Server: bpclntcmd -hn nbumedia
host nbumedia: nbumedia at 10.10.8.92
aliases:     nbumedia     10.10.8.92

As I mentioned before, child jobs are finished successfully, But parent job is with error :(6) the backup failed to back up the requested files

From dbclient ,it is starting like this :

00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block

00:17:20.565 [10084] <2> int_LogSystemInfo: INF -
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
        System name:    Linux
        Node name:      cdcdb1.intra
        Release:        3.8.13-68.3.4.el6uek.x86_64
        Version:        #2 SMP Tue Jul 14 15:03:36 PDT 2015
        Machine:        x86_64
        User name:      oracle
        Client Host:    cdcdbn-vip

00:17:20.565 [10084] <2> int_GetMMInfo: INF - Initialized Signal
00:17:20.565 [10084] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
00:17:21.122 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:21.122 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf

Here Why is it  trying to fetch from /home oracle/bp.conf .

:

Ending like this

00:59:18.336 [10084] <2> logconnections: BPRD CONNECT FROM 10.10.72.81.32526 TO 10.10.8.92.1556 fd = 21
00:59:18.491 [10084] <4> VxBSAQueryObject: INF - No match was found for query
00:59:18.491 [10084] <2> int_FindBackupImage: INF - 24r1dgud_1_1 not found
00:59:18.491 [10084] <4> int_RemoveImage: INF - Backup file, 24r1dgud_1_1, not found in image catalog.
00:59:28.317 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002

00:59:28.332 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:59:28.332 [10085] <4> sbtend: INF - --- END of SESSION ---

 

bphdb ending like this

00:59:29.101 [10002] <16> bphdb: ERR - Script exited with status = 1 <the requested operation was partially successful>
00:59:29.101 [10002] <8> bphdb: WRN - Directory /usr/openv/netbackup/logs/bpubsora does not exist
00:59:29.101 [10002] <16> bphdb: ERR - bphdb exit status = 6: the backup failed to back up the requested files

RMAN script out ending like this :

RMAN-03009: failure of backup command on ch02 channel at 03/26/2016 00:39:42
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
continuing other job steps, job failed will not be re-run
released channel: ch01
released channel: ch02
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch01 channel at 03/26/2016 00:59:28
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.

 

BTW bp.conf is :

 more /usr/openv/netbackup/bp.conf
SERVER = nbumedia
SERVER = fujitsu
CLIENT_NAME = cdcdbn-vip
CONNECT_OPTIONS = localhost 0 0 2
CLIENT_READ_TIMEOUT = 1800

I Reinstalled the netbackup client and tried to change the hostname. No difference.

I've compared Rman script with others, no difference.

Any idea?

Thanks,

Turgun

 

 

You most definitely have this error in dbclient log: <16> readCommFile: ERR - timed out after 900 seconds while reading from There are other similarities as well. There may be more similarities with level 5 logs. The TN also refers to (level 5) bptm and bpbrm logs. I would say that your issue is the same. Increasing client read timeout is not going to help. You will see in the TN that the issue with Oracle and that NBU 7.5.0.5 and higher works different and may fix the issue.

Dear Marianne, thank you for taking  time to analyse my post.

What is the TN?  Is it dbclient ?

 

Have you even opened the URL? The logs used for troubleshooting are clearly mentioned in bold letters.

yeah, I've opened it even yesterday. Just I've asked the definitaion of TN J.

You are right. The issue is same. (client read timeout ). Because I see  from dbext/logs/  that it also pointing  the (Client read timeout = 300).

On your link  about the resolution:
Four sites have contacted Oracle support and have reported the problem resolved after upgrading Oracle components. 

The NetBackup 7.5.0.5 client contains a design change that minimizes the opportunity for this issue to occur.   Previously, the Oracle child process would inherit the socket if forked/created between sbtbackup and sbtclose processing.  That window has been narrowed to a few milliseconds during sbtbackup processing.
 

As per my understand ,there is nothing to change/modify on netbackup side. Just play with Oracle components .

Thanks,

I will get back later on that.

 

There certainly is a problem if the rman update of control file or rman catalog takes 20 minutes.

Other than that, this looks to me as a lack of resource problem.

You need to figure out what is taking 20 minutes after the last stream has finished.

If it is lack of netbackup resources, CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT on both client and servers can migate some the problem.

Thanks a lot.

As Marianne suggested, after version 7.5.0.5 , issue was solved.

BR,

Turgun