Forum Discussion

rblaas's avatar
rblaas
Level 3
10 years ago

Random failure with status code 23

Hi all,

Could someone point me in the right direction? 

I have this problem which randomly occurs . 

I am backing up a physical client (mac OS X 10.6.8) and this client has 2 volumes. 

Most of the times the backup runs successfully. but now and then the backup fails with status 23. 

I can connect to the client via telnet on port 13782

DNS resolving works to. 

Here is some logging:

01/27/2015 11:10:14 - Info nbjm (pid=26156) starting backup job (jobid=4178) for client delage.xxxx.nl, policy PHYSICAL_LINUX_SPECIAL, schedule Daily_Dinc
01/27/2015 11:10:15 - Info bpbrm (pid=183283) delage.xxxx.nl is the host to backup data from
01/27/2015 11:10:15 - Info bpbrm (pid=183283) reading file list for client
01/27/2015 11:10:15 - estimated 1915622 kbytes needed
01/27/2015 11:10:15 - Info nbjm (pid=26156) started backup (backupid=delage.xxxx.nl_1422353415) job for client delage.xxxx.nl, policy PHYSICAL_LINUX_SPECIAL, schedule Daily_Dinc on storage unit STU_MSDP_NETBACKUP
01/27/2015 11:10:15 - started process bpbrm (pid=183283)
01/27/2015 11:10:16 - Info bpbrm (pid=183283) accelerator enabled
01/27/2015 11:10:16 - Error bpbrm (pid=183283) bpcd on delage.xxxx.nl exited with status 23: socket read failed
01/27/2015 11:10:16 - Info bpbkar (pid=0) done. status: 23: socket read failed
01/27/2015 11:10:16 - end writing
socket read failed  (23)

 

Client and Appliance running NetBackup v7.6.1

 

Backup and Recovery
MAC Agent
NetBackup Appliance

11 Replies

Sort By
Show More
  • rblaas's avatar
    rblaas
    Level 3
    10 years ago

    And again a failure.. (only on 1 partition.) 

    this output looks different then the last one.. 

     

    17:24:24.982 [82524] <2> bpbrm main: INITIATING (VERBOSE = 1): version NetBackup 7.6.1 2014102721
    17:24:24.982 [82524] <2> logparams: -backup -S netbackup -c delage.tros.nl -ct 0 -ru root -cl PHYSICAL_LINUX_SPECIAL -sched Daily_Dinc -bt 1422462264 -dt 3902 -st 1 -b delage.tros.nl_1422462264 -mediasvr netbackup -jobid 4585 -jobgrpid 4584 -masterversion 760000 -maxfrag 51200 -bpstart_time 1422462564 -reqid -1421688216 -mt 0 -to 0 -stunit STU_MSDP_NETBACKUP -rl 1 -rp 1209600 -eari 0 -p PureDiskVolume -mst 6 -flags 2 -storagesvr netbackup -sts_type PureDisk -use_ofb -use_otm -jm -secure 1 -kl 13 -rg root -fso -fscp -stream_count 2 -stream_number 1 -connect_options 16974338 
    17:24:24.982 [82524] <4> bpbrm main: ensured stderr cannot be used
    17:24:24.983 [82524] <2> bpbrm main: max_entries_per_add = 5000
    17:24:24.983 [82524] <2> read_client: dname=., offline=0, online_at=0 offline_at=0
    17:24:24.983 [82524] <2> read_client: dname=.., offline=0, online_at=0 offline_at=0
    17:24:24.983 [82524] <2> read_client: dname=CO_0, offline=0, online_at=0 offline_at=0
    17:24:24.983 [82524] <2> read_client: dname=OA_0, offline=0, online_at=0 offline_at=0
    17:24:24.983 [82524] <2> read_client: dname=host_info, offline=0, online_at=0 offline_at=0
    17:24:24.984 [82524] <2> read_host_info: CONNECT_NON_RES_PORT 2
    17:24:24.984 [82524] <2> read_host_info: DYNAMIC_ADDRESS 0
    17:24:24.984 [82524] <2> read_host_info: FREE_BROWSE 0
    17:24:24.984 [82524] <2> read_host_info: LIST_RESTORE 0
    17:24:24.984 [82524] <2> read_host_info: MAX_JOBS_THIS_CLIENT 3
    17:24:24.984 [82524] <2> read_host_info: NO_CALLBACK 2
    17:24:24.984 [82524] <2> read_host_info: WOFB_ENABLED 1
    17:24:24.984 [82524] <2> read_host_info: WOFB_FIM 1
    17:24:24.984 [82524] <2> read_host_info: WOFB_USAGE 0
    17:24:24.984 [82524] <2> read_host_info: WOFB_ERROR 0
    17:24:24.984 [82524] <2> read_host_info: DAEMON_PORT_TYPE 3
    17:24:24.984 [82524] <2> read_host_info: OST_CLIENT_DIRECT 0
    17:24:24.984 [82524] <2> read_host_info: OST_PROXY 0
    17:24:24.984 [82524] <2> read_host_info: OST_PROXY_SERVER *NULL*
    17:24:24.984 [82524] <2> read_host_info: OST_CLIENT_DIRECT_RESTORE 0
    17:24:24.984 [82524] <2> db_freeEXDB_INFO: ?
    17:24:24.995 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:24.996 [82524] <2> logconnections: BPCD CONNECT FROM xx.xx.xx.150.44026 TO xx.xx.yy.8.1556 fd = 4
    17:24:25.000 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.000 [82524] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD CONNECT FROM xx.xx.xx.150.37435 TO xx.xx.yy.8.1556 fd = 5
    17:24:25.001 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
    17:24:25.002 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-67842422462264999092000000344-Ps84Os
    17:24:25.082 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.082 [82524] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 4
    17:24:25.082 [82524] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 5
    17:24:25.083 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.088 [82524] <2> read_legacy_touch_file: Found /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK; requested from (miscsvr.c.521).
    17:24:25.089 [82524] <2> read_legacy_touch_file: 2097152 read ; requested from (miscsvr.c.521).
    17:24:25.090 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.090 [82524] <2> job_connect: SO_KEEPALIVE set on socket 4 for client netbackup
    17:24:25.090 [82524] <2> logconnections: BPJOBD CONNECT FROM xx.xx.xx.150.38599 TO xx.xx.xx.150.1556 fd = 4
    17:24:25.090 [82524] <2> job_authenticate_connection: ignoring VxSS authentication check for now...
    17:24:25.090 [82524] <2> job_connect: Connected to the host netbackup contype 53 jobid <4585> socket <4>
    17:24:25.090 [82524] <2> job_connect: Connected on port 38599
    17:24:25.091 [82524] <2> bpbrm main: send_3_0_client_params = 1
    17:24:25.094 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.094 [82524] <2> logconnections: BPCD CONNECT FROM xx.xx.xx.150.45849 TO xx.xx.yy.8.1556 fd = 5
    17:24:25.096 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.097 [82524] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD CONNECT FROM xx.xx.xx.150.49235 TO xx.xx.yy.8.1556 fd = 6
    17:24:25.097 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
    17:24:25.098 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-67844422462265095182000000344-Ps84Os
    17:24:25.178 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.178 [82524] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 5
    17:24:25.178 [82524] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 6
    17:24:25.180 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.181 [82524] <2> logconnections: BPCD CONNECT FROM xx.xx.xx.150.50973 TO xx.xx.yy.8.1556 fd = 8
    17:24:25.183 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.183 [82524] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD CONNECT FROM xx.xx.xx.150.33569 TO xx.xx.yy.8.1556 fd = 9
    17:24:25.184 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
    17:24:25.184 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-67846422462265181475000000344-Ps84Os
    17:24:25.262 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.262 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.269 [82524] <2> Orb::init: Enabling ORBNativeCharCodeSet UTF-8(Orb.cpp:713)
    17:24:25.271 [82524] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static Resource_Factory '-ORBNativeCharCodeSet UTF-8'" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBDefaultInitRef '' -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:916)
    17:24:25.276 [82524] <2> Orb::init: caching EndpointSelectorFactory(Orb.cpp:930)
    17:24:25.317 [82524] <4> MmDeviceMappings::GetRobotAttributes
        : <../../lib/MmDeviceMappings.cpp:976> search robot list (length=424) for OST PureDisk, type 8

    17:24:25.317 [82524] <4> MmDeviceMappings::GetRobotAttributes
        : <../../lib/MmDeviceMappings.cpp:1229> found match: "PureDisk" OST PureDisk
    17:24:25.317 [82524] <2> nb_accl_sts_support_check: accelerator support code is 1
    17:24:25.317 [82524] <2> nb_accl_validate: accelerator enabled
    17:24:25.317 [82524] <4> nb_accl_get_last_backup_byname: track_dir_win=\netbackup\netbackup\delage.tros.nl\PHYSICAL_LINUX_SPECIAL, track_dir_lnx=/netbackup/netbackup/delage.tros.nl/PHYSICAL_LINUX_SPECIAL, IN_SOCK=-1, OUT_SOCK=-1,real_client delage.tros.nl,client_param delage.tros.nl,client_hostname_param delage.tros.nl .
    17:24:25.319 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.319 [82524] <2> logconnections: BPCD CONNECT FROM xx.xx.xx.150.36196 TO xx.xx.yy.8.1556 fd = 11
    17:24:25.321 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.322 [82524] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD CONNECT FROM xx.xx.xx.150.41632 TO xx.xx.yy.8.1556 fd = 12
    17:24:25.322 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
    17:24:25.322 [82524] <8> vnet_pop_byte: [vnet.c:194] errno 0 0x0
    17:24:25.322 [82524] <2> vnet_pop_byte: vnet.c.196: 0: Function failed: 9 0x00000009
    17:24:25.322 [82524] <2> vnet_pop_string: vnet.c.276: 0: Function failed: 9 0x00000009
    17:24:25.322 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:454] status 9 0x9
    17:24:25.322 [82524] <2> bpcr_vnetd_connect_forward_socket_begin: vnet_vnetd_connect_forward_socket_begin() failed: 9
    17:24:25.322 [82524] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_socket_begin failed: 23
    17:24:25.322 [82524] <2> ConnectToBPCD: bpcd_connect_and_verify(delage.tros.nl, delage.tros.nl) failed: 23
    17:24:25.323 [82524] <16> bpbrm start_bpcd_stat: bpcd on delage.tros.nl exited with status 23: socket read failed
    17:24:25.323 [82524] <4> db_error_add_to_file: bpcd on delage.tros.nl exited with status 23: socket read failed
    17:24:25.323 [82524] <2> bpbrm start_bpcd: ../bpbrm.c.23102: start_bpcd_stat failed: 23 23 0x00000017
    17:24:25.323 [82524] <2> bpbrm kill_child_process: start
    17:24:25.323 [82524] <2> bpbrm Exit: attempting to send mail to root on delage.tros.nl
    17:24:25.325 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.325 [82524] <2> logconnections: BPCD CONNECT FROM xx.xx.xx.150.44802 TO xx.xx.yy.8.1556 fd = 11
    17:24:25.328 [82524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
    17:24:25.328 [82524] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD CONNECT FROM xx.xx.xx.150.49476 TO xx.xx.yy.8.1556 fd = 12
    17:24:25.328 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
    17:24:25.329 [82524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:457] ipc_string /usr/openv/var/tmp/vnet-67850422462265326415000000344-Ps84Os
    17:24:25.410 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.410 [82524] <2> bpbrm Exit: OUT_SOCK from bpcr = 11
    17:24:25.410 [82524] <2> bpbrm Exit: IN_SOCK from bpcr = 12
    17:24:25.410 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.411 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.437 [82524] <2> bpcr_get_version_rqst: bpcd version: 07610000
    17:24:25.451 [82524] <2> bpbrm Exit: client backup EXIT STATUS 23: socket read failed
    17:24:25.456 [82524] <2> job_monitoring_exex: ACK disconnect
    17:24:25.456 [82524] <2> job_disconnect: Disconnected