Forum Discussion

vyasmehulh's avatar
vyasmehulh
Level 4
7 years ago

Backup failing with error code 23

Hi Team,

One windows client backup is getting failed with error code 23 from today.

Below is the bptestbpcd output from master.

Tried adding master server hostname and IP on client machine and client hostname and IP on master server.

E:\Veritas\NetBackup\bin>bptestbpcd -client mispw3206 -debug
07:33:44.377 [3424.18492] <2> bptestbpcd: VERBOSE = 0
07:33:44.393 [3424.18492] <2> read_client: dname=., offline=0, online_at=0 offli
ne_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=.., offline=0, online_at=0 offl
ine_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=COA_0, offline=0, online_at=0 o
ffline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=CO_0, offline=0, online_at=0 of
fline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=host_info, offline=0, online_at
=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=OAA_0, offline=0, online_at=0 o
ffline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=OA_0, offline=0, online_at=0 of
fline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> check_vnetd_process_dup_permission: OpenProcessTok
en() failed: 5
07:33:44.393 [3424.18492] <2> vnet_check_windows_privileges: enabled SeDebugPriv
ilege privilege
07:33:44.424 [3424.18492] <2> vnet_pbxConnect_ex: pbxConnectExEx() failed: 10054

07:33:44.424 [3424.18492] <2> vnet_pbxConnect_ex: ../../libvlibs/vnet_pbx.c.691:
 pbxSetAddrEx/pbxConnectExEx return error 10054:An existing connection was forci
bly closed by the remote host.
07:33:44.424 [3424.18492] <8> do_pbx_service: [vnet_connect.c:2491] vnet_pbxConn
ect() failed, status=18, errno=0, use_vnetd=0, cr->vcr_service=bpcd
07:33:44.424 [3424.18492] <8> vnet_get_peer_sock_names: [vnet_nbrntd.c:829] gets
ockname() failed 10038 0x2736
07:33:44.424 [3424.18492] <2> vnet_proxy_get_logs: vnet_get_peer_sock_names(628)
 returned 10 10038
07:33:44.424 [3424.18492] <8> async_connect: [vnet_connect.c:2058] do_service fa
iled 18 0x12
07:33:44.486 [3424.18492] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3531] VN
_REQUEST_PBX_C_SUPPORTED 13 0xd
07:33:44.970 [3424.18492] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2053] VN_
REQUEST_SERVICE_SOCKET 6 0x6
07:33:44.970 [3424.18492] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2067] ser
vice bpcd
07:33:45.188 [3424.18492] <2> logconnections: BPCD CONNECT FROM 10.128.44.226.59
719 TO 10.128.74.45.13724 fd = 628
07:33:45.188 [3424.18492] <2> vnet_connect_to_vnetd_bpcd: js_bpcd_info: 00000000
027E1C70
07:33:45.204 [3424.18492] <2> vnet_pbxConnect_ex: pbxConnectExEx Succeeded
07:33:45.298 [3424.18492] <8> do_pbx_service: [vnet_connect.c:2581] via PBX VNET
D CONNECT FROM 10.128.44.226.59721 TO 10.128.74.45.1556 fd = 652
07:33:45.298 [3424.18492] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vne
td.c:458] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
07:33:45.547 [3424.18492] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vne
td.c:483] ipc_string 65216
07:33:45.984 [3424.18492] <2> get_short_base: (1) cannot read (byte 1) from netw
ork: An existing connection was forcibly closed by the remote host.
07:33:45.984 [3424.18492] <2> bpcr_put_vnetd_forward_socket: get_short failed: 1
0054
07:33:45.984 [3424.18492] <2> local_bpcr_connect: bpcr_put_vnetd_forward_socket
failed: 23
07:33:45.984 [3424.18492] <2> ConnectToBPCD: bpcd_connect_and_verify(mispw3206,
mispw3206) failed: 23
07:33:45.984 [3424.18492] <16> bptestbpcd main: JSON proxy message = {"allow_lar
ge_status": {"timestamp": 1530099225, "who": "vnet_tss_init", "line_number": 32,
 "comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timest
amp": 1530099225, "who": "connect_to_service", "line_number": 834, "comment": "c
onnect parameters", "data": {"who": "vnet_connect_to_vnetd_bpcd", "host": "mispw
3206", "service": "vnetd", "override_required_interface": null, "extra_tries_on_
connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_opt
ions": null}}, "proxy_request": {"timestamp": 1530099225, "who": "vnet_proxy_bui
ld_request", "line_number": 3300, "comment": "request to local proxy", "data": {
"bpcd_info": {"remote_client": "mispw3206", "remote_host": "mispw3206", "remote_
probably_dhcp": false, "proxy_flag": 32, "proxy_flag_hex": "0x00000020", "cmd_so
ck": 628}, "i_am_daemon": false, "domain_constraints": {}, "service": "vnetd", "
auth_only": false, "pid": 3424, "ssa": false, "local_user": "v565751@CORP", "hos
t": "mispw3206", "peer_host": "mispw3206", "isTestConnection": true, "peer_clien
t": "mispw3206", "local_client": "to1pw3243.corp.statestr.com", "local_dhcp_inte
rval": 0, "peer_dhcp_info": null, "local_hostname": "to1pw3243.corp.statestr.com
", "secure_peer": false, "resilient": false, "nbrntd_peer_addr": null, "nbrntd_s
ock_addr": null, "hand_back": false, "request_timeout": 1810}}, "proxy_response"
: {"timestamp": 1530099225, "who": "vnet_proxy_parse_response", "line_number": 3
472, "comment": "response from local proxy", "data": {"status": 0, "local_proxy_
info": {}, "domain_constraints_set": {"process_hint": "882e988d-3244-48e9-bb12-8
b43b6f0dac7", "process_hint_reason": "there was no inherited hint, the connected
 peer was not a master server, so defaulted to primary master", "process_hint_se
rver_name": "to1pw3243.corp.statestr.com"}, "statusmsgxml": "<?xml version=\"1.0
\" encoding=\"utf-8\"?><StatusMsgDoc><StatusMsg NBUStatus=\"0\" Severity=\"0\"><
/StatusMsg></StatusMsgDoc>"}}, "status": {"timestamp": 1530099225, "who": "conne
ct_to_service", "line_number": 981, "comment": "vnet status", "data": 0}}
<16>bptestbpcd main: Function ConnectToBPCD(mispw3206) failed: 23
07:33:45.984 [3424.18492] <16> bptestbpcd main: Function ConnectToBPCD(mispw3206
) failed: 23
<16>bptestbpcd main: socket read failed
07:33:46.000 [3424.18492] <16> bptestbpcd main: socket read failed
<2>bptestbpcd: socket read failed
07:33:46.015 [3424.18492] <2> bptestbpcd: socket read failed
<2>bptestbpcd: EXIT status = 23
07:33:46.015 [3424.18492] <2> bptestbpcd: EXIT status = 23
socket read failed

8
Windows Server 2008

15 Replies

Sort By
  • Tape_Archived's avatar
    Tape_Archived
    Moderator
    7 years ago
    1. Check in client registry and make sure it has desired entries of Master & Media Servers. You can do this same using Client Gui as well
    2. Verify you can ping from both master/media and client 
    3. Again verify correct IP's are set in host files 
    4. Use bpclntcmd -pn from client to check the connectivity, try other options of bpclntcmd as well
    5. Check this message in verbose logs - "process_hint_reason": "there was no inherited hint, the connected
       peer was not a master server, so defaulted to primary master", "process_hint_se
      rver_name": "to1pw3243.corp.statestr.com"
    6. Ensure all the settings are correct and rerun the bptestbpcd command

     

    • vyasmehulh's avatar
      vyasmehulh
      Level 4
      7 years ago

      Hi Tape_Archived,

      Have tried all the possibilities below already but still its failing

      • Nicolai's avatar
        Nicolai
        Moderator
        7 years ago

        Reboot server ..

        If that is not possible , stop the netbackup client and check that no bpbkar32.exe processes are running. Start the netbackup client again.

  • Marianne's avatar
    Marianne
    Level 6
    7 years ago

    Have you checked if NetBackup Client Services are running on the client?

    If so, create bpcd log folder on the client (under ...\netbacup\logs) and run bptestbpcd again.

    Check bpcd log on the client for errors. 

    • vyasmehulh's avatar
      vyasmehulh
      Level 4
      7 years ago

      Hi Marianne,

      Yes, the netbackup client service is running.

      C:\Program Files\Veritas\NetBackup\bin>bpps
      * MISPW3206                                              6/27/18 08:25:01.768
      COMMAND           PID      LOAD             TIME   MEM                  START
      vnetd            7108    0.000%            0.078  8.7M   6/27/18 07:32:44.245
      bpinetd          7220    0.000%            5.070   11M   6/27/18 07:32:44.291
      mtstrmd          5024    0.000%            0.046  7.3M   6/27/18 07:32:49.112
      vnetd            1864    0.000%            0.046  8.2M   6/27/18 08:24:24.359
      bpps             7724    0.000%            0.265  8.2M   6/27/18 08:25:00.489


      bpcd log
      ==================
      08:24:24.390 [1988.2044] <2> setup_debug_log: switched debug log file for bpcd
      08:24:24.390 [1988.2044] <4> bpcd main: VERBOSE=0, BPCD VERBOSE=0 Timestamp=1530098791
      08:24:24.390 [1988.2044] <2> logparams: bpcd
      08:24:24.390 [1988.2044] <2> bpcd main: Got socket for input 464
      08:24:24.390 [1988.2044] <2> process_requests: offset to GMT 18000
      08:24:24.406 [1988.2044] <2> logconnections: BPCD ACCEPT FROM 10.128.44.226.60917 TO 10.128.74.45.13724 fd = 464
      08:24:24.406 [1988.2044] <2> process_requests: setup_sockopts complete
      08:24:24.437 [1988.2044] <2> bpcd peer_hostname: Connection from host to1pw3243.statestr.com (10.128.44.226) port 60917
      08:24:24.546 [1988.2044] <2> vnet_pbxConnect: pbxConnectEx Succeeded
      08:24:24.577 [1988.2044] <8> do_pbx_service: [vnet_connect.c:2186] via PBX VNETD CONNECT FROM 10.128.74.45.49968 TO 10.128.44.226.1556 fd = 496
      08:24:24.593 [1988.2044] <8> vnet_vnetd_get_security_info: [vnet_vnetd.c:2800] VN_REQUEST_GET_SECURITY_INFO 9 0x9
      08:24:25.233 [1988.2044] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:198] VN_REQUEST_DISCONNECT 1 0x1
      08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1733] in failed file cache ERR=11001 NAME=eb60574b-b7fd-4828-a4a6-7b04cdd5d9e1 SVC=NULL
      08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1402] vnet_cached_getaddrinfo_and_update() failed 6 0x6
      08:24:25.233 [1988.2044] <8> vnet_same_host_and_update: [vnet_addrinfo.c:3013] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME1=eb60574b-b7fd-4828-a4a6-7b04cdd5d9e1
      08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1733] in failed file cache ERR=11001 NAME=keystore SVC=NULL
      08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1402] vnet_cached_getaddrinfo_and_update() failed 6 0x6
      08:24:25.233 [1988.2044] <8> vnet_same_host_and_update: [vnet_addrinfo.c:3013] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME1=keystore
      08:24:25.279 [1988.2044] <16> VssImportCredential: (../../libVnbat/vss_auth.cpp,3299): Cred file open FAILED
      08:24:25.279 [1988.2044] <8> vnet_setup_vxss_socket: [vnet_vxss_helper.c:189] vnet_vxss_import_credential() failed 36 0x24
      08:24:25.279 [1988.2044] <16> process_requests: vnet_setup_vxss_socket(to1pw3243.statestr.com) failed: 36




      • Nicolai's avatar
        Nicolai
        Moderator
        7 years ago

        What version of netbackup are you running ?

        Try on the client running bpclntcmd -clear_host_cache 

        According to the logs shown, it looks like a name resolution issue