03-25-2016 03:02 AM
Hi All,
Netbackup 7.1 Linux 2.6 (Master,Media,Client)
We have like this issue:
1.
The oracle backup with 3 stream. All jobs writing successfully. But Parent jobs fails with error (6) the backup failed to back up the requested file)
From bpbrm logs also positive feedback regarding backup job
Exit: client backup EXIT STATUS 0: the requested operation was successfully completed
2.
From RMAN like the following error
RMAN-03009: failure of backup command on ch03 channel at 03/25/2016 08:41:35
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
VxBSAEndTxn: Failed with error:
The transaction was aborted.
continuing other job steps, job failed will not be re-run
The message is non RMAN, but media manager or vendor specific failure ))
From this site also saying: This is not an Oracle error per-se, it has to do with the disk or tape. (http://www.dba-oracle.com/t_ora_27192.htm)
But to this tape backups from many different hosts are working without problem. even on that same volume pool.
Any idea ?
Thanks in advance,
Turgun
Solved! Go to Solution.
03-26-2016 06:06 AM
03-25-2016 03:19 AM
03-25-2016 02:29 PM
Hi Marianne,
First of all I want to point, the standart backup is working normal.
According to the dbclient logs, your last post is a bit different .
I attached full logs of dbclient,bphdb,rman_scripts and its output.
I've set client read timeout to 1800
connections between client,media and master server are okay.
From Master and Media server
bpclntcmd -hn cdcdbn-vip
host cdcdbn-vip: cdcdbn-vip at 10.10.72.90
aliases: cdcdbn-vip 10.10.72.90
From client
To Media-Server: bpclntcmd -hn fujitsu
host fujitsu: fujitsu at 10.10.72.101
aliases: fujitsu 10.10.72.101
To Master-Server: bpclntcmd -hn nbumedia
host nbumedia: nbumedia at 10.10.8.92
aliases: nbumedia 10.10.8.92
As I mentioned before, child jobs are finished successfully, But parent job is with error :(6) the backup failed to back up the requested files
From dbclient ,it is starting like this :
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:20.565 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:17:20.565 [10084] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
00:17:20.565 [10084] <2> int_LogSystemInfo: INF -
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: cdcdb1.intra
Release: 3.8.13-68.3.4.el6uek.x86_64
Version: #2 SMP Tue Jul 14 15:03:36 PDT 2015
Machine: x86_64
User name: oracle
Client Host: cdcdbn-vip
00:17:20.565 [10084] <2> int_GetMMInfo: INF - Initialized Signal
00:17:20.565 [10084] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
00:17:21.122 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:17:21.122 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
Here Why is it trying to fetch from /home oracle/bp.conf .
:
Ending like this
00:59:18.336 [10084] <2> logconnections: BPRD CONNECT FROM 10.10.72.81.32526 TO 10.10.8.92.1556 fd = 21
00:59:18.491 [10084] <4> VxBSAQueryObject: INF - No match was found for query
00:59:18.491 [10084] <2> int_FindBackupImage: INF - 24r1dgud_1_1 not found
00:59:18.491 [10084] <4> int_RemoveImage: INF - Backup file, 24r1dgud_1_1, not found in image catalog.
00:59:28.317 [10084] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
00:59:28.332 [10085] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
00:59:28.332 [10085] <4> sbtend: INF - --- END of SESSION ---
bphdb ending like this
00:59:29.101 [10002] <16> bphdb: ERR - Script exited with status = 1 <the requested operation was partially successful>
00:59:29.101 [10002] <8> bphdb: WRN - Directory /usr/openv/netbackup/logs/bpubsora does not exist
00:59:29.101 [10002] <16> bphdb: ERR - bphdb exit status = 6: the backup failed to back up the requested files
RMAN script out ending like this :
RMAN-03009: failure of backup command on ch02 channel at 03/26/2016 00:39:42
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
VxBSAEndTxn: Failed with error:
The transaction was aborted.
continuing other job steps, job failed will not be re-run
released channel: ch01
released channel: ch02
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch01 channel at 03/26/2016 00:59:28
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
VxBSAEndTxn: Failed with error:
The transaction was aborted.
BTW bp.conf is :
more /usr/openv/netbackup/bp.conf
SERVER = nbumedia
SERVER = fujitsu
CLIENT_NAME = cdcdbn-vip
CONNECT_OPTIONS = localhost 0 0 2
CLIENT_READ_TIMEOUT = 1800
I Reinstalled the netbackup client and tried to change the hostname. No difference.
I've compared Rman script with others, no difference.
Any idea?
Thanks,
Turgun
03-26-2016 02:24 AM
03-26-2016 04:30 AM
Dear Marianne, thank you for taking time to analyse my post.
What is the TN? Is it dbclient ?
03-26-2016 05:07 AM
03-26-2016 05:31 AM
yeah, I've opened it even yesterday. Just I've asked the definitaion of TN J.
You are right. The issue is same. (client read timeout ). Because I see from dbext/logs/ that it also pointing the (Client read timeout = 300).
On your link about the resolution:
Four sites have contacted Oracle support and have reported the problem resolved after upgrading Oracle components.
The NetBackup 7.5.0.5 client contains a design change that minimizes the opportunity for this issue to occur. Previously, the Oracle child process would inherit the socket if forked/created between sbtbackup and sbtclose processing. That window has been narrowed to a few milliseconds during sbtbackup processing.
As per my understand ,there is nothing to change/modify on netbackup side. Just play with Oracle components .
Thanks,
I will get back later on that.
03-26-2016 06:06 AM
03-29-2016 03:47 AM
There certainly is a problem if the rman update of control file or rman catalog takes 20 minutes.
Other than that, this looks to me as a lack of resource problem.
You need to figure out what is taking 20 minutes after the last stream has finished.
If it is lack of netbackup resources, CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT on both client and servers can migate some the problem.
03-29-2016 04:53 AM
Thanks a lot.
As Marianne suggested, after version 7.5.0.5 , issue was solved.
BR,
Turgun