Error code 6 and 11

[Hendrik_Bezuide]

We have moved from old UNIX server with netbackup 6.5 to new Unix Server with netbackup 7.6.0.02

It work ,but from yesterday I am getting error 6 and 11,thne it work, but this morning it gives this error on all myh prod clients,  I am not sure where to look for this logs .

I have got one log file. and one log file bpbrm form master server.

No bphdb logs in master or media server

bphdb on db client is only for yesterday

Error on JOB

09/14/2016 07:50:40 - Info nbjm (pid=22718) starting backup job (jobid=376168) for client sapp1zdb02-ibvip.prasa.com, policy sapp1-GP1-DB-RMAN, schedule Archived-Redo-Log
09/14/2016 07:50:40 - Info nbjm (pid=22718) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=376168, request id:{2AE5FD82-7A3F-11E6-813C-0010E05C118E})
09/14/2016 07:50:40 - requesting resource ZFSHC_bck_source
09/14/2016 07:50:40 - requesting resource prasaerpbck01.prasa.com.NBU_POLICY.MAXJOBS.sapp1-GP1-DB-RMAN
09/14/2016 07:50:40 - granted resource prasaerpbck01.prasa.com.NBU_POLICY.MAXJOBS.sapp1-GP1-DB-RMAN
09/14/2016 07:50:42 - estimated 0 kbytes needed
09/14/2016 07:50:42 - Info nbjm (pid=22718) started backup (backupid=sapp1zdb02-ibvip.prasa.com_1473832242) job for client sapp1zdb02-ibvip.prasa.com, policy sapp1-GP1-DB-RMAN, schedule Archived-Redo-Log on storage unit
09/14/2016 07:50:42 - started process bpbrm (pid=7922)
09/14/2016 07:50:43 - Info bpbrm (pid=7922) sapp1zdb02-ibvip.prasa.com is the host to backup data from
09/14/2016 07:50:43 - Info bpbrm (pid=7922) reading file list for client
09/14/2016 07:50:43 - connecting
09/14/2016 07:50:44 - Info bpbrm (pid=7922) starting bphdb on client
09/14/2016 07:50:44 - Info bphdb (pid=29490) Backup started
09/14/2016 07:50:44 - Info bphdb (pid=29490) Waiting for the child status
09/14/2016 07:50:44 - connected; connect time: 0:00:00
09/14/2016 07:50:45 - Error bpbrm (pid=7922) from client sapp1zdb02-ibvip.prasa.com: ERR - Script exited with status = 11 <system call failed>
09/14/2016 07:50:45 - Error bpbrm (pid=7922) from client sapp1zdb02-ibvip.prasa.com: ERR - bphdb exit status = 6: the backup failed to back up the requested files
09/14/2016 07:50:45 - Info bphdb (pid=29490) done. status: 6: the backup failed to back up the requested files
09/14/2016 07:50:45 - end writing
the backup failed to back up the requested files (6)

 more.....

[Hendrik_Bezuide]

bpbrm

Part 1

07:17:26.369 [6276] <4> bpbrm main: logging will continue into new log at midnight
07:17:26.371 [6276] <2> bpbrm main: INITIATING (VERBOSE = 5): version NetBackup 7.6 2013111313
07:17:26.371 [6276] <2> logparams: -backup -S prasaerpbck01.prasa.com -c sapp1zdb02-ibvip.prasa.com -ct 4 -ru root -cl sapp1-RP1-DB-RMAN -sched Archived-Redo-Log -bt 1473830246 -dt 0 -st 5 -b sapp1zdb02-ibvip.prasa.com_1473830246 -mediasvr prasaerpbck01 -jobid 376164 -jobgrpid 376163 -masterversion 760000 -maxfrag 0 -bpstart_time 1473830546 -p 0 -use_ofb -use_otm -intelligent_db_opts -csschedule_key 108 -jm -secure 1 -kl 28 -rg root -fso -stream_count 34 -stream_number 34 -hodb -auto -connect_options 16974338
07:17:26.371 [6276] <2> init_resilient_cache: [vnet_nbrntd.c:867] Initialize resilient cache. 0 0x0
07:17:26.371 [6276] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:806] the socket is 0 0x0
07:17:26.371 [6276] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:806] the socket is 2 0x2
07:17:26.372 [6276] <2> bpbrm main: max_entries_per_add = 5000
07:17:26.372 [6276] <2> verify_client: ../bpbrm.c.41603: *connect_opts: 16974338 16974338 0x01030202
07:17:26.372 [6276] <2> read_client: ?
07:17:26.372 [6276] <2> read_client: dname=., offline=0, online_at=0 offline_at=0
07:17:26.372 [6276] <2> read_client: dname=.., offline=0, online_at=0 offline_at=0
07:17:26.372 [6276] <2> read_client: dname=host_info, offline=0, online_at=0 offline_at=0
07:17:26.372 [6276] <2> read_client: dname=CO_0, offline=0, online_at=0 offline_at=0
07:17:26.372 [6276] <2> read_client: dname=OA_0, offline=0, online_at=0 offline_at=0
07:17:26.372 [6276] <2> read_host_info: ?
07:17:26.372 [6276] <2> read_host_info: CONNECT_NON_RES_PORT 2
07:17:26.372 [6276] <2> read_host_info: DYNAMIC_ADDRESS 0
07:17:26.372 [6276] <2> read_host_info: FREE_BROWSE 0
07:17:26.372 [6276] <2> read_host_info: LIST_RESTORE 0
07:17:26.372 [6276] <2> read_host_info: MAX_JOBS_THIS_CLIENT 0
07:17:26.372 [6276] <2> read_host_info: NO_CALLBACK 2
07:17:26.372 [6276] <2> read_host_info: WOFB_ENABLED 1
07:17:26.372 [6276] <2> read_host_info: WOFB_FIM 1
07:17:26.372 [6276] <2> read_host_info: WOFB_USAGE 0
07:17:26.372 [6276] <2> read_host_info: WOFB_ERROR 0
07:17:26.372 [6276] <2> read_host_info: DAEMON_PORT_TYPE 3
07:17:26.372 [6276] <2> read_host_info: OST_CLIENT_DIRECT 0
07:17:26.372 [6276] <2> read_host_info: OST_PROXY 0
07:17:26.372 [6276] <2> read_host_info: OST_PROXY_SERVER *NULL*
07:17:26.372 [6276] <2> read_host_info: OST_CLIENT_DIRECT_RESTORE 0
07:17:26.372 [6276] <2> read_sec_info: ?
07:17:26.372 [6276] <2> read_sec_info: no sec_info: /usr/openv/netbackup/db/client/sapp1zdb02-ibvip.prasa.com/sec_info does not exist
07:17:26.372 [6276] <2> get_exdb_info1: ?
07:17:26.372 [6276] <2> read_exdb_info: ?
07:17:26.372 [6276] <2> read_exdb_info: no exdb_info files processed
07:17:26.372 [6276] <2> db_freeSEC_INFO: pinfo is NULL
07:17:26.373 [6276] <2> db_freeEXDB_INFO: ?
07:17:26.373 [6276] <2> db_freeEXDB_INFO: plist is NULL
07:17:26.373 [6276] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23144: bpcd_client: sapp1zdb02-ibvip.prasa.com
07:17:26.373 [6276] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23147: bpcd_client_hostname: sapp1zdb02-ibvip.prasa.com
07:17:26.373 [6276] <2> local_bpcr_connect: bpcr.c.333: connect_opts = 0x01000100 connect_opts2 = 0x01000100
07:17:26.373 [6276] <2> local_bpcr_connect: bpcr.c.340: connect_opts = 0x01000100
07:17:26.373 [6276] <2> local_bpcr_connect: bpcr.c.379: daemon_port_type = 0
07:17:26.373 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.373 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.373 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.374 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.374 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.374 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.374 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.374 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.374 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.374 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.374 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.374 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.374 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:26.375 [6276] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:17:26.375 [6276] <2> do_pbx_service: [vnet_connect.c:2160] via PBX bpcd CONNECT FROM 192.168.50.22.62652 TO 192.168.50.9.1556 fd = 5
07:17:26.375 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 192.168.50.22.62652 TO 192.168.50.9.1556 fd = 5
07:17:26.375 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO sapp1zdb02-ibvip.prasa.com 192.168.50.9 bpcd VIA pbx
07:17:26.375 [6276] <2> logconnections: BPCD CONNECT FROM 192.168.50.22.62652 TO 192.168.50.9.1556 fd = 5
07:17:26.375 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.375 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.375 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.375 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.375 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.375 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.375 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:26.376 [6276] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:17:26.380 [6276] <8> do_pbx_service: [vnet_connect.c:2152] via PBX VNETD CONNECT FROM 192.168.50.22.54019 TO 192.168.50.9.1556 fd = 7
07:17:26.380 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 192.168.50.22.54019 TO 192.168.50.9.1556 fd = 7
07:17:26.380 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 192.168.50.9 192.168.50.9 vnetd VIA pbx
07:17:26.380 [6276] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
07:17:26.444 [6276] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /tmp/vnet-17123473829831671557000001603-5pamUq
07:17:26.510 [6276] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:471] hash_str1 c016848f6c17c5f378a049544e451efc
07:17:26.650 [6276] <2> bpcr_get_version_rqst: bpcd version: 07600000
07:17:26.720 [6276] <2> bpcr_get_hostname_rqst: Server hostname length = 26
07:17:26.720 [6276] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 5
07:17:26.720 [6276] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 7
07:17:26.720 [6276] <2> verify_client: ../bpbrm.c.41930: hostname: prasaerpbck01.prasa.com
07:17:26.720 [6276] <2> verify_client: ../bpbrm.c.41931: is_on: 0 0 0x00000000
07:17:26.790 [6276] <2> bpcr_get_version_rqst: bpcd version: 07600000
07:17:26.860 [6276] <2> verify_client: ../bpbrm.c.41934: bpcr_update_resiliency : 0 0 0x00000000
07:17:26.864 [6276] <2> verify_client: ../bpbrm.c.41957: *connect_opts: 16843008 16843008 0x01010100
07:17:26.867 [6276] <2> read_legacy_touch_file: Found /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK; requested from (miscsvr.c.515).
07:17:26.867 [6276] <2> read_legacy_touch_file: 1048576 read ; requested from (miscsvr.c.515).
07:17:26.867 [6276] <2> vnet_same_host_and_update: [vnet_addrinfo.c:2905] matched as locals NAME1=prasaerpbck01.prasa.com NAME2=localhost
07:17:26.867 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.867 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.867 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4721] Local [strong] check, using interface ANY
07:17:26.867 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:26.868 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 172.22.60.213.65481 TO 172.22.60.213.13723 fd = 5
07:17:26.868 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO prasaerpbck01.prasa.com 172.22.60.213 bpjobd
07:17:26.868 [6276] <2> job_connect: SO_KEEPALIVE set on socket 5 for client prasaerpbck01.prasa.com
07:17:26.868 [6276] <2> logconnections: BPJOBD CONNECT FROM 172.22.60.213.65481 TO 172.22.60.213.13723 fd = 5
07:17:26.868 [6276] <2> job_authenticate_connection: ignoring VxSS authentication check for now...
07:17:26.868 [6276] <2> job_connect: Connected to the host prasaerpbck01.prasa.com contype 53 jobid <376164> socket <5>
07:17:26.868 [6276] <2> job_connect: Connected on port 65481
07:17:26.868 [6276] <2> set_job_details: Tfile (376164): LOG 1473830246 4 bpbrm 6276 sapp1zdb02-ibvip.prasa.com is the host to backup data from

[Hendrik_Bezuide]

part 2

07:17:26.868 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 87, msg = LOG 1473830246 4 bpbrm 6276 sapp1zdb02-ibvip.prasa.com is the host to backup data from

07:17:26.869 [6276] <2> set_job_details: Tfile (376164): LOG 1473830246 4 bpbrm 6276 reading file list for client

07:17:26.869 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 57, msg = LOG 1473830246 4 bpbrm 6276 reading file list for client

07:17:26.869 [6276] <2> bpbrm main: File read = "job_type="8""
07:17:26.869 [6276] <2> bpbrm main: File read = "version="1""
07:17:26.869 [6276] <2> bpbrm main: File read = "dedupe="1""
07:17:26.869 [6276] <2> bpbrm main: File read = "oracle_sid="RP11""
07:17:26.869 [6276] <2> bpbrm main: File read = "oracle_home="/oracle/RP1/112""
07:17:26.869 [6276] <2> bpbrm main: File read = "client="sapp1zdb02-ibvip.prasa.com""
07:17:26.869 [6276] <2> bpbrm main: File read = "OS_check_password="1""
07:17:26.869 [6276] <2> bpbrm main: File read = "OS_authenticated="1""
07:17:26.869 [6276] <2> bpbrm main: File read = "num_channels="4""
07:17:26.869 [6276] <2> bpbrm main: File read = "skip_readonly="0""
07:17:26.869 [6276] <2> bpbrm main: File read = "force_readonly="0""
07:17:26.869 [6276] <2> bpbrm main: File read = "skip_offline="0""
07:17:26.869 [6276] <2> bpbrm main: File read = "rman_defaults="1""
07:17:26.869 [6276] <2> bpbrm main: File read = "obj_type="Archived Logs""
07:17:26.869 [6276] <2> bpbrm main: File read = "server="prasaerpbck01.prasa.com""
07:17:26.869 [6276] <2> bpbrm main: File read = "classname="sapp1-RP1-DB-RMAN""
07:17:26.869 [6276] <2> bpbrm main: File read = "schedule="Archived-Redo-Log""
07:17:26.869 [6276] <2> bpbrm main: File read = "backup_archlogs="101""
07:17:26.869 [6276] <2> bpbrm main: File read = "auto_archdelete="2""
07:17:26.870 [6276] <2> bpbrm main: File read = "arch_num_channels="4""
07:17:26.870 [6276] <2> bpbrm main: File read = "CONTINUE"
07:17:26.870 [6276] <2> local_bpcr_connect: bpcr.c.333: connect_opts = 0x01000100 connect_opts2 = 0x01000100
07:17:26.870 [6276] <2> local_bpcr_connect: bpcr.c.340: connect_opts = 0x01000100
07:17:26.870 [6276] <2> local_bpcr_connect: bpcr.c.379: daemon_port_type = 0
07:17:26.870 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.870 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.870 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.870 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.870 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.870 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.870 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.870 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.870 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.870 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.870 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.870 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.870 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:26.871 [6276] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:17:26.871 [6276] <2> do_pbx_service: [vnet_connect.c:2160] via PBX bpcd CONNECT FROM 192.168.50.22.46578 TO 192.168.50.9.1556 fd = 7
07:17:26.871 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 192.168.50.22.46578 TO 192.168.50.9.1556 fd = 7
07:17:26.871 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO sapp1zdb02-ibvip.prasa.com 192.168.50.9 bpcd VIA pbx
07:17:26.872 [6276] <2> logconnections: BPCD CONNECT FROM 192.168.50.22.46578 TO 192.168.50.9.1556 fd = 7
07:17:26.872 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.872 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.872 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.872 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:26.872 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:26.872 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:26.872 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:26.873 [6276] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:17:26.876 [6276] <8> do_pbx_service: [vnet_connect.c:2152] via PBX VNETD CONNECT FROM 192.168.50.22.40390 TO 192.168.50.9.1556 fd = 8
07:17:26.876 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 192.168.50.22.40390 TO 192.168.50.9.1556 fd = 8
07:17:26.876 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 192.168.50.9 192.168.50.9 vnetd VIA pbx
07:17:26.876 [6276] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
07:17:26.944 [6276] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /tmp/vnet-17127473829832171443000001603-TxaCUq
07:17:27.010 [6276] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:471] hash_str1 da32a95250c4eed1b0b2f8115285d4eb
07:17:27.150 [6276] <2> bpcr_get_version_rqst: bpcd version: 07600000
07:17:27.220 [6276] <2> bpcr_get_version_rqst: bpcd version: 07600000
07:17:27.290 [6276] <2> bpcr_get_charset_rqst: bpcd charset: 0000000c
07:17:27.294 [6276] <2> ConnectionCache::connectAndCache: Acquiring new connection for host prasaerpbck01.prasa.com, query type 89
07:17:27.294 [6276] <2> vnet_same_host_and_update: [vnet_addrinfo.c:2905] matched as locals NAME1=prasaerpbck01.prasa.com NAME2=localhost
07:17:27.294 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.294 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:27.294 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4721] Local [strong] check, using interface ANY
07:17:27.294 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:27.294 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 172.22.60.213.61886 TO 172.22.60.213.13721 fd = 7
07:17:27.294 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO prasaerpbck01.prasa.com 172.22.60.213 bpdbm
07:17:27.295 [6276] <2> logconnections: BPDBM CONNECT FROM 172.22.60.213.61886 TO 172.22.60.213.13721 fd = 7
07:17:27.295 [6276] <2> vnet_check_vxss_client_magic_with_info_ex: [vnet_vxss_helper.c:943] Ignoring VxSS authentication 2 0x2
07:17:27.306 [6276] <2> db_end: Need to collect reply
07:17:27.382 [6276] <2> isDONE: reply DONE 0 <?xml version="1.0" encoding="utf-8"?><StatusMsgDoc><StatusMsg NBUStatus="0" Severity="0" VXULoid="271" VXULtid="0"><MsgString>the&#32;requested&#32;operation&#32;was&#32;successfully&#32;completed</MsgString></StatusMsg></StatusMsgDoc>
07:17:27.412 [6276] <2> isDONE: reply DONE 0
07:17:27.412 [6276] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23144: bpcd_client: sapp1zdb02-ibvip.prasa.com
07:17:27.412 [6276] <2> bpbrm start_bpcd_stat: ../bpbrm.c.23147: bpcd_client_hostname: sapp1zdb02-ibvip.prasa.com
07:17:27.412 [6276] <2> local_bpcr_connect: bpcr.c.333: connect_opts = 0x01010100 connect_opts2 = 0x01000100
07:17:27.412 [6276] <2> local_bpcr_connect: bpcr.c.340: connect_opts = 0x01010100
07:17:27.412 [6276] <2> local_bpcr_connect: bpcr.c.379: daemon_port_type = 1
07:17:27.412 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.412 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:27.412 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.412 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:27.412 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.412 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:27.412 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.412 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:27.412 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.412 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:27.412 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.412 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:27.412 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:27.413 [6276] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:17:27.413 [6276] <2> do_pbx_service: [vnet_connect.c:2160] via PBX bpcd CONNECT FROM 192.168.50.22.62069 TO 192.168.50.9.1556 fd = 7
07:17:27.413 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 192.168.50.22.62069 TO 192.168.50.9.1556 fd = 7
07:17:27.413 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO sapp1zdb02-ibvip.prasa.com 192.168.50.9 bpcd VIA pbx
07:17:27.413 [6276] <2> logconnections: BPCD CONNECT FROM 192.168.50.22.62069 TO 192.168.50.9.1556 fd = 7
07:17:27.413 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:27.413 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.413 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:27.413 [6276] <2> vnet_sortaddrs: [vnet_addrinfo.c:3856] sorted addrs: 1 0x1
07:17:27.413 [6276] <2> vnet_configured_stacks: [vnet_addrinfo.c:8473] ignoring IN6_IS_ADDR_UNSPECIFIED() 0 0x0
07:17:27.413 [6276] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4754] using interface ANY
07:17:27.413 [6276] <2> async_connect: [vnet_connect.c:1499] connect in progress 1 0x1
07:17:27.414 [6276] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:17:27.417 [6276] <8> do_pbx_service: [vnet_connect.c:2152] via PBX VNETD CONNECT FROM 192.168.50.22.33008 TO 192.168.50.9.1556 fd = 8
07:17:27.417 [6276] <2> async_connect: [vnet_connect.c:1691] connect async CONNECT FROM 192.168.50.22.33008 TO 192.168.50.9.1556 fd = 8
07:17:27.417 [6276] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 192.168.50.9 192.168.50.9 vnetd VIA pbx
07:17:27.417 [6276] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
07:17:27.484 [6276] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /tmp/vnet-17129473829832711585000001603-jGaKUq
07:17:27.550 [6276] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:471] hash_str1 20ac56b03c09208861bd2e5c2afd431c
07:17:27.690 [6276] <2> bpcr_get_version_rqst: bpcd version: 07600000
07:17:27.760 [6276] <2> bpcr_get_hostname_rqst: Server hostname length = 26
07:17:27.760 [6276] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 7
07:17:27.760 [6276] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 8
07:17:27.760 [6276] <2> set_job_details: Tfile (376164): LOG 1473830247 4 bpbrm 6276 starting bphdb on client

07:17:27.760 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 53, msg = LOG 1473830247 4 bpbrm 6276 starting bphdb on client

[Hendrik_Bezuide]

part 3

07:17:27.760 [6276] <2> bpbrm main: received bpcd success message
07:17:27.830 [6276] <2> bpbrm main: client_pid=17130
07:17:27.830 [6276] <2> set_job_details: Tfile (376164): LOG 1473830247 4 bphdb 17130 Backup started

07:17:27.830 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 44, msg = LOG 1473830247 4 bphdb 17130 Backup started

07:17:27.830 [6276] <2> bpbrm main: from client sapp1zdb02-ibvip.prasa.com: read client start message
07:17:27.830 [6276] <2> bpbrm write_continue_backup: wrote CONTINUE BACKUP on COMM_SOCK <7>
07:17:27.830 [6276] <2> write_file_names: buffering file name 'job_type="8"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'version="1"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'dedupe="1"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'oracle_sid="RP11"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'oracle_home="/oracle/RP1/112"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'client="sapp1zdb02-ibvip.prasa.com"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'OS_check_password="1"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'OS_authenticated="1"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'num_channels="4"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'skip_readonly="0"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'force_readonly="0"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'skip_offline="0"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'rman_defaults="1"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'obj_type="Archived Logs"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'server="prasaerpbck01.prasa.com"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'classname="sapp1-RP1-DB-RMAN"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'schedule="Archived-Redo-Log"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'backup_archlogs="101"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'auto_archdelete="2"' for output
07:17:27.831 [6276] <2> write_file_names: buffering file name 'arch_num_channels="4"' for output
07:17:27.831 [6276] <2> write_file_names: successfully wrote buffer to COMM_SOCK
07:17:27.831 [6276] <2> bpbrm main: wrote CONTINUE on COMM_SOCK
07:17:27.831 [6276] <2> bpbrm main: closing COMM_SOCK
07:17:27.831 [6276] <2> bpbrm main: Value of PFI = 0
07:17:27.900 [6276] <2> set_job_details: Tfile (376164): LOG 1473830247 4 bphdb 17130 Waiting for the child status

07:17:27.900 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 58, msg = LOG 1473830247 4 bphdb 17130 Waiting for the child status

07:17:28.838 [6276] <16> bpbrm main: from client sapp1zdb02-ibvip.prasa.com: ERR - Script exited with status = 11 <system call failed>
07:17:28.838 [6276] <2> set_job_details: Tfile (376164): LOG 1473830248 16 bpbrm 6276 from client sapp1zdb02-ibvip.prasa.com: ERR - Script exited with status = 11 <system call failed>

07:17:28.838 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 127, msg = LOG 1473830248 16 bpbrm 6276 from client sapp1zdb02-ibvip.prasa.com: ERR - Script exited with status = 11 <system call failed>

07:17:28.838 [6276] <4> db_error_add_to_file: from client sapp1zdb02-ibvip.prasa.com: ERR - Script exited with status = 11 <system call failed>
07:17:28.839 [6276] <16> bpbrm main: from client sapp1zdb02-ibvip.prasa.com: ERR - bphdb exit status = 6: the backup failed to back up the requested files
07:17:28.839 [6276] <2> set_job_details: Tfile (376164): LOG 1473830248 16 bpbrm 6276 from client sapp1zdb02-ibvip.prasa.com: ERR - bphdb exit status = 6: the backup failed to back up the requested files

07:17:28.839 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 147, msg = LOG 1473830248 16 bpbrm 6276 from client sapp1zdb02-ibvip.prasa.com: ERR - bphdb exit status = 6: the backup failed to back up the requested files

07:17:28.839 [6276] <4> db_error_add_to_file: from client sapp1zdb02-ibvip.prasa.com: ERR - bphdb exit status = 6: the backup failed to back up the requested files
07:17:28.839 [6276] <2> bpbrm main: client sapp1zdb02-ibvip.prasa.com EXIT STATUS = 6: the backup failed to back up the requested files
07:17:28.839 [6276] <2> set_job_details: Tfile (376164): LOG 1473830248 4 bphdb 17130 done. status: 6: the backup failed to back up the requested files

07:17:28.839 [6276] <2> send_job_file: job ID 376164, ftype = 3 msg len = 95, msg = LOG 1473830248 4 bphdb 17130 done. status: 6: the backup failed to back up the requested files

07:17:28.839 [6276] <2> bpbrm Exit: client backup EXIT STATUS 6: the backup failed to back up the requested files
07:17:28.840 [6276] <2> vnet_close_socket_safely: [vnet.c:1106] safe close 0 0x0
07:17:28.885 [6276] <2> job_monitoring_exex: ACK disconnect
07:17:28.885 [6276] <2> job_disconnect: Disconnected

[Michael_G_Ander]

First question has anything between changed in either the SAP Database or the Unix OS ?

Being a database server my first suspicion would be some kind of memory exhaustion

https://vox.veritas.com/t5/NetBackup/General-database-backup-error-troubleshooting/m-p/675381

Describes my way of troubleshooting database backups.

I would also get the DBA and OS administrator to monitor their systems while the backup runs, status 11 is usually caused by something outside Netbackup.

[Marianne]

Wait... a bit more info please.... 

Is this a SAP or Oracle policy?

Are you saying the clients were migrated to new hardware with newer NBU version?
Which Unix OS and version?
SAP and Oracle versions?
Have you verified Compatibility?

If the client was moved to new hardware, have you tried to treat the config same as new client?
With proper steps to link NBU agent?

If my understanding is wrong about what exactly has changed, please help me to understand...

PS:
Please copy logs to .txt files (e.g. bpbrm.txt, dbclient.txt) and use 'Choose Files' to upload logs.

[Hendrik_Bezuide]

SAP policy

It worked for more than 4 months without problem, ionly yesterday and today we are getting this error.

Not sure what logs to get and where to get it from

[Marianne · Accepted solution]

Status 11 (system call failed) says something is wrong on the client.

This is Archive Log backup that failed. 
When last did a full db backup complete successfully?
Has the dba had a look at script output files?
Is SAP/Oracle online and healthy?

Logs that need to be created/enabled on client are listed in the Troubleshooting section of NetBackup for SAP Administrator's Guide 

[Hendrik_Bezuide]

Hi Marianne

The full backup was done the day before yesterday and the archive logs yesterday (every 4 hours) the last one 15:00.I am trying to get hold of the person that is working with this clients, sound that they are working on problem with SAP.

Will update if I get hold of him.

[Hendrik_Bezuide]

Hi Marianne

We are using Oracle instances for this, I think there may be no scripts may be wrong.

[Marianne]

I am battling to understand your previous post.

What does "We are using Oracle instances for this  " mean?

Oracle Intelligent Policies (OIP) perhaps?
Not SAP?

Logs needed for OIP in Troubleshooting section of NetBackup for Oracle Administrator's Guide 

And the screenshot?
What is that all about? 

[Hendrik_Bezuide]

Does this screenshot help, it is on the netbackup administration console.

And my policy info

[Hendrik_Bezuide]

Thank you, got it working.

The person from working on SAP database was here with me, we work together and sorted this out. the problem was that the database was not fully up from their side.