Backup failing with error code 23

Hi Team,

One windows client backup is getting failed with error code 23 from today.

Below is the bptestbpcd output from master.

Tried adding master server hostname and IP on client machine and client hostname and IP on master server.

E:\Veritas\NetBackup\bin>bptestbpcd -client mispw3206 -debug
07:33:44.377 [3424.18492] <2> bptestbpcd: VERBOSE = 0
07:33:44.393 [3424.18492] <2> read_client: dname=., offline=0, online_at=0 offli
ne_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=.., offline=0, online_at=0 offl
ine_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=COA_0, offline=0, online_at=0 o
ffline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=CO_0, offline=0, online_at=0 of
fline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=host_info, offline=0, online_at
=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=OAA_0, offline=0, online_at=0 o
ffline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=OA_0, offline=0, online_at=0 of
fline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> check_vnetd_process_dup_permission: OpenProcessTok
en() failed: 5
07:33:44.393 [3424.18492] <2> vnet_check_windows_privileges: enabled SeDebugPriv
ilege privilege
07:33:44.424 [3424.18492] <2> vnet_pbxConnect_ex: pbxConnectExEx() failed: 10054

07:33:44.424 [3424.18492] <2> vnet_pbxConnect_ex: ../../libvlibs/vnet_pbx.c.691:
 pbxSetAddrEx/pbxConnectExEx return error 10054:An existing connection was forci
bly closed by the remote host.
07:33:44.424 [3424.18492] <8> do_pbx_service: [vnet_connect.c:2491] vnet_pbxConn
ect() failed, status=18, errno=0, use_vnetd=0, cr->vcr_service=bpcd
07:33:44.424 [3424.18492] <8> vnet_get_peer_sock_names: [vnet_nbrntd.c:829] gets
ockname() failed 10038 0x2736
07:33:44.424 [3424.18492] <2> vnet_proxy_get_logs: vnet_get_peer_sock_names(628)
 returned 10 10038
07:33:44.424 [3424.18492] <8> async_connect: [vnet_connect.c:2058] do_service fa
iled 18 0x12
07:33:44.486 [3424.18492] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3531] VN
_REQUEST_PBX_C_SUPPORTED 13 0xd
07:33:44.970 [3424.18492] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2053] VN_
REQUEST_SERVICE_SOCKET 6 0x6
07:33:44.970 [3424.18492] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2067] ser
vice bpcd
07:33:45.188 [3424.18492] <2> logconnections: BPCD CONNECT FROM 10.128.44.226.59
719 TO 10.128.74.45.13724 fd = 628
07:33:45.188 [3424.18492] <2> vnet_connect_to_vnetd_bpcd: js_bpcd_info: 00000000
027E1C70
07:33:45.204 [3424.18492] <2> vnet_pbxConnect_ex: pbxConnectExEx Succeeded
07:33:45.298 [3424.18492] <8> do_pbx_service: [vnet_connect.c:2581] via PBX VNET
D CONNECT FROM 10.128.44.226.59721 TO 10.128.74.45.1556 fd = 652
07:33:45.298 [3424.18492] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vne
td.c:458] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
07:33:45.547 [3424.18492] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vne
td.c:483] ipc_string 65216
07:33:45.984 [3424.18492] <2> get_short_base: (1) cannot read (byte 1) from netw
ork: An existing connection was forcibly closed by the remote host.
07:33:45.984 [3424.18492] <2> bpcr_put_vnetd_forward_socket: get_short failed: 1
0054
07:33:45.984 [3424.18492] <2> local_bpcr_connect: bpcr_put_vnetd_forward_socket
failed: 23
07:33:45.984 [3424.18492] <2> ConnectToBPCD: bpcd_connect_and_verify(mispw3206,
mispw3206) failed: 23
07:33:45.984 [3424.18492] <16> bptestbpcd main: JSON proxy message = {"allow_lar
ge_status": {"timestamp": 1530099225, "who": "vnet_tss_init", "line_number": 32,
 "comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timest
amp": 1530099225, "who": "connect_to_service", "line_number": 834, "comment": "c
onnect parameters", "data": {"who": "vnet_connect_to_vnetd_bpcd", "host": "mispw
3206", "service": "vnetd", "override_required_interface": null, "extra_tries_on_
connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_opt
ions": null}}, "proxy_request": {"timestamp": 1530099225, "who": "vnet_proxy_bui
ld_request", "line_number": 3300, "comment": "request to local proxy", "data": {
"bpcd_info": {"remote_client": "mispw3206", "remote_host": "mispw3206", "remote_
probably_dhcp": false, "proxy_flag": 32, "proxy_flag_hex": "0x00000020", "cmd_so
ck": 628}, "i_am_daemon": false, "domain_constraints": {}, "service": "vnetd", "
auth_only": false, "pid": 3424, "ssa": false, "local_user": "v565751@CORP", "hos
t": "mispw3206", "peer_host": "mispw3206", "isTestConnection": true, "peer_clien
t": "mispw3206", "local_client": "to1pw3243.corp.statestr.com", "local_dhcp_inte
rval": 0, "peer_dhcp_info": null, "local_hostname": "to1pw3243.corp.statestr.com
", "secure_peer": false, "resilient": false, "nbrntd_peer_addr": null, "nbrntd_s
ock_addr": null, "hand_back": false, "request_timeout": 1810}}, "proxy_response"
: {"timestamp": 1530099225, "who": "vnet_proxy_parse_response", "line_number": 3
472, "comment": "response from local proxy", "data": {"status": 0, "local_proxy_
info": {}, "domain_constraints_set": {"process_hint": "882e988d-3244-48e9-bb12-8
b43b6f0dac7", "process_hint_reason": "there was no inherited hint, the connected
 peer was not a master server, so defaulted to primary master", "process_hint_se
rver_name": "to1pw3243.corp.statestr.com"}, "statusmsgxml": "<?xml version=\"1.0
\" encoding=\"utf-8\"?><StatusMsgDoc><StatusMsg NBUStatus=\"0\" Severity=\"0\"><
/StatusMsg></StatusMsgDoc>"}}, "status": {"timestamp": 1530099225, "who": "conne
ct_to_service", "line_number": 981, "comment": "vnet status", "data": 0}}
<16>bptestbpcd main: Function ConnectToBPCD(mispw3206) failed: 23
07:33:45.984 [3424.18492] <16> bptestbpcd main: Function ConnectToBPCD(mispw3206
) failed: 23
<16>bptestbpcd main: socket read failed
07:33:46.000 [3424.18492] <16> bptestbpcd main: socket read failed
<2>bptestbpcd: socket read failed
07:33:46.015 [3424.18492] <2> bptestbpcd: socket read failed
<2>bptestbpcd: EXIT status = 23
07:33:46.015 [3424.18492] <2> bptestbpcd: EXIT status = 23
socket read failed

Tags (2)
15 Replies

Re: Backup failing with error code 23

Have you checked if NetBackup Client Services are running on the client?

If so, create bpcd log folder on the client (under ...\netbacup\logs) and run bptestbpcd again.

Check bpcd log on the client for errors. 

Re: Backup failing with error code 23

Hi Marianne,

Yes, the netbackup client service is running.

C:\Program Files\Veritas\NetBackup\bin>bpps
* MISPW3206                                              6/27/18 08:25:01.768
COMMAND           PID      LOAD             TIME   MEM                  START
vnetd            7108    0.000%            0.078  8.7M   6/27/18 07:32:44.245
bpinetd          7220    0.000%            5.070   11M   6/27/18 07:32:44.291
mtstrmd          5024    0.000%            0.046  7.3M   6/27/18 07:32:49.112
vnetd            1864    0.000%            0.046  8.2M   6/27/18 08:24:24.359
bpps             7724    0.000%            0.265  8.2M   6/27/18 08:25:00.489


bpcd log
==================
08:24:24.390 [1988.2044] <2> setup_debug_log: switched debug log file for bpcd
08:24:24.390 [1988.2044] <4> bpcd main: VERBOSE=0, BPCD VERBOSE=0 Timestamp=1530098791
08:24:24.390 [1988.2044] <2> logparams: bpcd
08:24:24.390 [1988.2044] <2> bpcd main: Got socket for input 464
08:24:24.390 [1988.2044] <2> process_requests: offset to GMT 18000
08:24:24.406 [1988.2044] <2> logconnections: BPCD ACCEPT FROM 10.128.44.226.60917 TO 10.128.74.45.13724 fd = 464
08:24:24.406 [1988.2044] <2> process_requests: setup_sockopts complete
08:24:24.437 [1988.2044] <2> bpcd peer_hostname: Connection from host to1pw3243.statestr.com (10.128.44.226) port 60917
08:24:24.546 [1988.2044] <2> vnet_pbxConnect: pbxConnectEx Succeeded
08:24:24.577 [1988.2044] <8> do_pbx_service: [vnet_connect.c:2186] via PBX VNETD CONNECT FROM 10.128.74.45.49968 TO 10.128.44.226.1556 fd = 496
08:24:24.593 [1988.2044] <8> vnet_vnetd_get_security_info: [vnet_vnetd.c:2800] VN_REQUEST_GET_SECURITY_INFO 9 0x9
08:24:25.233 [1988.2044] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:198] VN_REQUEST_DISCONNECT 1 0x1
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1733] in failed file cache ERR=11001 NAME=eb60574b-b7fd-4828-a4a6-7b04cdd5d9e1 SVC=NULL
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1402] vnet_cached_getaddrinfo_and_update() failed 6 0x6
08:24:25.233 [1988.2044] <8> vnet_same_host_and_update: [vnet_addrinfo.c:3013] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME1=eb60574b-b7fd-4828-a4a6-7b04cdd5d9e1
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1733] in failed file cache ERR=11001 NAME=keystore SVC=NULL
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1402] vnet_cached_getaddrinfo_and_update() failed 6 0x6
08:24:25.233 [1988.2044] <8> vnet_same_host_and_update: [vnet_addrinfo.c:3013] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME1=keystore
08:24:25.279 [1988.2044] <16> VssImportCredential: (../../libVnbat/vss_auth.cpp,3299): Cred file open FAILED
08:24:25.279 [1988.2044] <8> vnet_setup_vxss_socket: [vnet_vxss_helper.c:189] vnet_vxss_import_credential() failed 36 0x24
08:24:25.279 [1988.2044] <16> process_requests: vnet_setup_vxss_socket(to1pw3243.statestr.com) failed: 36




Tags (2)

Re: Backup failing with error code 23

What version of netbackup are you running ?

Try on the client running bpclntcmd -clear_host_cache 

According to the logs shown, it looks like a name resolution issue 

 

Re: Backup failing with error code 23

The Status 36 error that I see in bpcd log is new to me.

08:24:25.279 [1988.2044] <16> process_requests: vnet_setup_vxss_socket(to1pw3243.statestr.com) failed: 36

Status Code Ref Guide says this about status 36:

NetBackup status code: 36

Message: failed trying to allocate memory

Explanation: Allocation of system memory failed. This error occurs when an insufficient system memory is available. The system may be overloaded with too many processes and it does not have enough physical and virtual memory.

Recommended Action: Free up memory by terminating any unneeded processes that consume a lot of memory. Add more swap space or physical memory.

 

Can you please check if this is relevant here?

Re: Backup failing with error code 23

Hi Nicholai,

Tried clearing host cache but didn't worked.

 

NBU version on master is 8.1.1 and on client is 8.0

Master and media are same.

Tags (2)

Re: Backup failing with error code 23

  1. Check in client registry and make sure it has desired entries of Master & Media Servers. You can do this same using Client Gui as well
  2. Verify you can ping from both master/media and client 
  3. Again verify correct IP's are set in host files 
  4. Use bpclntcmd -pn from client to check the connectivity, try other options of bpclntcmd as well
  5. Check this message in verbose logs - "process_hint_reason": "there was no inherited hint, the connected
     peer was not a master server, so defaulted to primary master", "process_hint_se
    rver_name": "to1pw3243.corp.statestr.com"
  6. Ensure all the settings are correct and rerun the bptestbpcd command

 

Re: Backup failing with error code 23

Hi Marianne,

Have informed windows team to have a look into memory.

 

 

Tags (2)

Re: Backup failing with error code 23

Hi Tape_Archived,

Have tried all the possibilities below already but still its failing

Tags (2)

Re: Backup failing with error code 23

Reboot server ..

If that is not possible , stop the netbackup client and check that no bpbkar32.exe processes are running. Start the netbackup client again.

Re: Backup failing with error code 23

Hi Nicholai,

Thanks for suggesting.

Tried stopping all the backup services and restarting them but no luck.

Following with on server reboot.

 

Also we observerd that the reverse lookup for backup server is not correct.

Attached screen shot.

Tags (2)

Re: Backup failing with error code 23

That need to be fixed.

Reverse lookup is also a non-authorized answer. Netbackup will not trust a non-authorized DNS answer.

You can disable reverse name lookup on the client by running :

bpclntcmd -reverse_name_lookup  prohibited

See also:

https://www.veritas.com/support/en_US/doc/15263389-127350397-0/v14660333-127350397

Re: Backup failing with error code 23

Try to add both hostnames in the Client's SERVER  list:
to1pw3243.statestr.com
to1pw3243.corp.statestr.com

It seems that 'bpclntcmd -pn' did not produce any output?
This command will connect to the master server (1st entry in the Client's Server list) on port 1556 or 13724. 

If you are getting no response, it means that the Client cannot connect to master server. 
(Best to leave command running until it times out or produce output.)
   Check bprd log on the master to see if connection was received from client IP

The output should look like this:

expecting response from server <master-name> 
<client-name-resolved-from-IP> <client-name-in-policy> <Client-source-IP> <Client-source-port-number> 

By the looks of it, you are experiencing name lookup issues in your environment. 
Is this with one client only? Or more?
Have you asked your network/DNS admin?

 

Highlighted

Re: Backup failing with error code 23

Hi Marianne,

bpclntcmd -pn command doesn't shows any output after entering both the below entries

to1pw3243.statestr.com
to1pw3243.corp.statestr.com

Also find below answer to queries

Is this with one client only? Or more? :- One client
Have you asked your network/DNS admin? :- Yes, they are looking into it.

Tags (2)

Re: Backup failing with error code 23

Hi Nicolai,

Tried below command but no luck.

 

bpclntcmd -reverse_name_lookup  prohibited

Tags (2)

Re: Backup failing with error code 23

Please see this part of my previous post:

If you are getting no response, it means that the Client cannot connect to master server. 
(Best to leave command running until it times out or produce output.)
Check bprd log on the master to see if connection was received from client IP.