RMAN failures

[stucci]

Hi all,

please I need help, from some days we have RMAN backup failures, I find some solution without success.

For troubleshooting I create these folder logs (I follow this article https://support.symantec.com/en_US/article.HOWTO89647.html):

Client: bpdbsbora, bporaexp (or boraexp64), bporaimp (or boraimp64), bpubsora, dbclient, bphdb, bpfis, and bpcd.

Master: bpbrm and bptm, bprd

In client logs I don' t finded nothing of interesting, but in Master bprd, I found this significant (I think) text:

02:08:31.504 [4904] <8> append_to_client_log: couldn't open log file SysTest_SDBSTS51_TXMETA_Archive on client sdbsts51-bk491, error = 13
02:08:31.505 [4904] <8> bkarfiles: failure writing progress log on client sdbsts51-bk491 in log SysTest_SDBSTS51_TXMETA_Archive:  file open failed (12)
02:08:31.672 [4904] <16> bkarfiles: Unable to write progress log <SysTest_SDBSTS51_TXMETA_Archive> on client sdbsts51-bk491. Policy=NONE Sched=6
02:08:31.682 [4904] <2> bkarfiles: User backup failed (client = sdbsts51-bk491 user = ora1023 group = sdbsts51-bk491): file open failed

I follow this post:

https://www-secure.symantec.com/connect/forums/sql-backup-bes

here the issue is referred to an SQL db, but the problem to "failure writing progress log" it is the same. Unfortunately I don' t find any issue related to a DNS

CLIENT:

hostname: sdbsts51-bk491; NBU version NetBackup-AIX53 7.1.0.1;

[root@SDBSTS51 logs]# /usr/openv/netbackup/bin/bpclntcmd -pn
expecting response from server nbumaster01
sdbsts51.sede.corp.sanpaoloimi.com sdbsts51 10.31.141.171 33144
[root@SDBSTS51 logs]# 
[root@SDBSTS51 logs]# nslookup nbumaster01-bk491
Server:         10.19.4.26
Address:        10.19.4.26#53

nbumaster01-bk491.sede.corp.sanpaoloimi.com     canonical name = papcap03-bk491.sede.corp.sanpaoloimi.com.
Name:   papcap03-bk491.sede.corp.sanpaoloimi.com
Address: 10.254.168.38

[root@SDBSTS51 logs]# 
[root@SDBSTS51 logs]# nslookup 10.254.168.38
Server:         10.19.4.26
Address:        10.19.4.26#53

Non-authoritative answer:
38.168.254.10.in-addr.arpa      name = papcap03-bk491.sede.corp.sanpaoloimi.com.

Authoritative answers can be found from:

[root@SDBSTS51 logs]# 

MASTER:

hostname: nbumaster01-bk491; NBU NetBackup-Solaris10 7.1.0.1;

root@sgemop54 # nslookup sdbsts51-bk491
Server:         10.248.158.8
Address:        10.248.158.8#53

Name:   sdbsts51-bk491.sede.corp.sanpaoloimi.com
Address: 10.254.170.41

root@sgemop54 # 
root@sgemop54 # nslookup 10.254.170.41
Server:         10.248.158.8
Address:        10.248.158.8#53

Non-authoritative answer:
41.170.254.10.in-addr.arpa      name = sdbsts51-bk491.sede.corp.sanpaoloimi.com.

Authoritative answers can be found from:

root@sgemop54 # 
root@sgemop54 # bptestbpcd -client sdbsts51-bk491 -verbose -debug
11:48:06.813 [28929] <2> bptestbpcd: VERBOSE = 0
11:48:06.830 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
11:48:06.830 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/086/6579c686+veritas_pbx,1,4,2,2,0+sdbsts51-bk491.txt
11:48:06.877 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
11:48:06.877 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/086/6579c686+vnetd,1,4,2,2,0+sdbsts51-bk491.txt
11:48:06.907 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
11:48:06.907 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/086/6579c686+bpcd,1,4,2,2,0+sdbsts51-bk491.txt
11:48:06.940 [28929] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:48:06.942 [28929] <2> logconnections: BPCD CONNECT FROM 10.254.168.37.55357 TO 10.254.170.41.1556 fd = 3
11:48:06.944 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
11:48:06.945 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/155/8eadf55+veritas_pbx,1,4,2,2,0+10.254.170.41.txt
11:48:06.954 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
11:48:06.954 [28929] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/155/8eadf55+vnetd,1,4,2,2,0+10.254.170.41.txt
11:48:06.963 [28929] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:48:06.965 [28929] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1776: 0: via PBX: VNETD CONNECT FROM 10.254.168.37.55366 TO 10.254.170.41.1556 fd = 4
11:48:06.972 [28929] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.445: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
11:48:07.128 [28929] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.462: 0: ipc_string: /tmp/vnet-34814439459287102602000000061-NOlMaZ
1 1 1
10.254.168.37:55357 -> 10.254.170.41:1556
10.254.168.37:55366 -> 10.254.170.41:1556
11:48:07.301 [28929] <2> bpcr_get_peername_rqst: Server peername length = 40
11:48:07.367 [28929] <2> bpcr_get_hostname_rqst: Server hostname length = 14
11:48:07.421 [28929] <2> bpcr_get_clientname_rqst: Server client name length = 14
11:48:07.480 [28929] <2> bpcr_get_version_rqst: bpcd version: 07100000
11:48:07.542 [28929] <2> bpcr_get_platform_rqst: Server client platform length = 9
11:48:07.601 [28929] <2> bpcr_get_version_rqst: bpcd version: 07100000
11:48:07.671 [28929] <2> bpcr_patch_version_rqst: theRest == > <
11:48:07.672 [28929] <2> bpcr_get_version_rqst: bpcd version: 07100000
11:48:07.741 [28929] <2> bpcr_patch_version_rqst: theRest == > <
11:48:07.742 [28929] <2> bpcr_get_version_rqst: bpcd version: 07100000
PEER_NAME = sgemop54-bk491.sede.corp.sanpaoloimi.com
HOST_NAME = sdbsts51-bk491
CLIENT_NAME = sdbsts51-bk491
VERSION = 0x07100000
PLATFORM = rs6000_53
PATCH_VERSION = 7.1.0.1 
SERVER_PATCH_VERSION = -1.-1.-1.-1 
MASTER_SERVER = nbumaster01
EMM_SERVER = nbumaster01
11:48:07.807 [28929] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:48:07.808 [28929] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1776: 0: via PBX: VNETD CONNECT FROM 10.254.168.37.55433 TO 10.254.170.41.1556 fd = 5
11:48:07.809 [28929] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.445: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
11:48:07.907 [28929] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.462: 0: ipc_string: /tmp/vnet-34818439459287902560000000061-SOlMaZ
10.254.168.37:55433 -> 10.254.170.41:1556
<2>bptestbpcd: EXIT status = 0
11:48:07.976 [28929] <2> bptestbpcd: EXIT status = 0

In attachment the job log detail and some logs.

Tank you everybody, regards

[GulzarShaikhAUS]

Please attach all the logs as you mentioned in the post. 

Client: bpdbsbora, bporaexp (or boraexp64), bporaimp (or boraimp64), bpubsora, dbclient, bphdb, bpfis, and bpcd.

Master: bpbrm and bptm, bprd

Make sure they are verbosity 5.

I would also need the RMAN script. The admin logs from client would also be useful

[Michael_G_Ander]

Looking at the Job Details your script is failing with a  status 1, get your DBA to check the rman output log and the database alert.log.

BTW. Please attach logs as txt so that they can be read in a browser.

[stucci]

Hi all,

I attach all the logs present on Master and Client. Have you idea because the "bprd" now it is not written ?

There are some problems with attachments upload, I must zip all the logs in one file sorry!

tks guys

[stucci]

dbclient,

09:52:20.630 [36962418] <4> sendRequest: Progress log not for monitoring=</usr/openv/netbackup/logs/user_ops/dbext/logs/36962418.0.1439884340>

09:52:20.631 [36962418] <4> sendRequest:    request = ora1023  sdbsts51-bk491 sdbsts51-bk491 sdbsts51-bk491 /usr/openv/netbackup/logs/user_ops/dbext/logs/36962418.0.1439884340 SysTest_SDBSTS51_TXMETA_Archive NONE 6 TXMETA 0 4 0 C C C C C 0 0 0 0 5

09:52:20.635 [36962418] <2> logconnections: BPRD CONNECT FROM 10.254.170.41.48457 TO 10.254.168.38.1556 fd = 20

09:52:20.635 [36962418] <4> serverResponse: initial client_read_timeout = <900>

09:52:20.635 [36962418] <4> readCommMessages: Entering readCommMessages

10:07:21.726 [36962418] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/36962418.0.1439884340

10:07:21.726 [36962418] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/36962418.0.1439884340>

10:07:21.726 [36962418] <16> CreateNewImage: ERR - serverResponse() failed

10:07:21.726 [36962418] <4> closeApi: entering closeApi.

10:07:21.726 [36962418] <4> closeApi: INF - EXIT STATUS 6: the backup failed to back up the requested files

bpcd,

09:52:30.074 [14942342] <2> process_requests: BPCD_BECOME_USER_GROUP_RQST

09:52:30.080 [14942342] <16> process_requests: can't become user ora1023 and group sdbsts51-bk491

bprd,

09:52:20.644 [11749] <2> logconnections: BPRD ACCEPT FROM 10.254.170.41.48457 TO 10.254.168.38.1556 fd = 9

09:52:20.679 [11749] <2> connected_peer: Connection from host sdbsts51-bk491.sede.corp.sanpaoloimi.com, 10.254.170.41, on non-reserved port 48457

09:52:29.787 [11749] <2> job_connect: Connected to the host nbumaster01 contype 10 jobid <6076385> socket <5>

09:52:29.787 [11749] <2> job_connect: Connected on port 65021

09:52:29.787 [11749] <2> set_job_details: Tfile (6076385): LOG 1439884349 16 bprd 11749 CLIENT sdbsts51-bk491  POLICY NONE  SCHED 6  EXIT STATUS 12 (file open failed)

09:52:29.793 [11749] <2> job_monitoring_exex: ACK disconnect

09:52:29.794 [11749] <2> job_disconnect: Disconnected

09:52:29.800 [11749] <2> bkarfiles: User backup failed (client = sdbsts51-bk491 user = ora1023 group = sdbsts51-bk491): file open failed

09:52:30.137 [11749] <2> add_msg_to_progress_file: Can't become user ora1023 and group sdbsts51-bk491 on sdbsts51-bk491

09:52:30.140 [11749] <2> bkarfiles: EXIT STATUS = 12

09:52:30.140 [11749] <2> process_request: EXIT STATUS 12

09:52:30.141 [11749] <2> bprd: /usr/openv/netbackup/bin/userreq_notify backup /usr/openv/netbackup/logs/user_ops/dbext/logs/36962418.0.1439884340 ora1023

[stucci]

we have solved !

the support OS added some parameter to the Access Control (software that manage users access) for the local users and the backup now work fine.