β06-27-2018 04:46 AM
Hi Team,
One windows client backup is getting failed with error code 23 from today.
Below is the bptestbpcd output from master.
Tried adding master server hostname and IP on client machine and client hostname and IP on master server.
E:\Veritas\NetBackup\bin>bptestbpcd -client mispw3206 -debug
07:33:44.377 [3424.18492] <2> bptestbpcd: VERBOSE = 0
07:33:44.393 [3424.18492] <2> read_client: dname=., offline=0, online_at=0 offli
ne_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=.., offline=0, online_at=0 offl
ine_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=COA_0, offline=0, online_at=0 o
ffline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=CO_0, offline=0, online_at=0 of
fline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=host_info, offline=0, online_at
=0 offline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=OAA_0, offline=0, online_at=0 o
ffline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> read_client: dname=OA_0, offline=0, online_at=0 of
fline_at=0 offlineres=0 onlineres_at=0 offlineres_at=0
07:33:44.393 [3424.18492] <2> check_vnetd_process_dup_permission: OpenProcessTok
en() failed: 5
07:33:44.393 [3424.18492] <2> vnet_check_windows_privileges: enabled SeDebugPriv
ilege privilege
07:33:44.424 [3424.18492] <2> vnet_pbxConnect_ex: pbxConnectExEx() failed: 10054
07:33:44.424 [3424.18492] <2> vnet_pbxConnect_ex: ../../libvlibs/vnet_pbx.c.691:
pbxSetAddrEx/pbxConnectExEx return error 10054:An existing connection was forci
bly closed by the remote host.
07:33:44.424 [3424.18492] <8> do_pbx_service: [vnet_connect.c:2491] vnet_pbxConn
ect() failed, status=18, errno=0, use_vnetd=0, cr->vcr_service=bpcd
07:33:44.424 [3424.18492] <8> vnet_get_peer_sock_names: [vnet_nbrntd.c:829] gets
ockname() failed 10038 0x2736
07:33:44.424 [3424.18492] <2> vnet_proxy_get_logs: vnet_get_peer_sock_names(628)
returned 10 10038
07:33:44.424 [3424.18492] <8> async_connect: [vnet_connect.c:2058] do_service fa
iled 18 0x12
07:33:44.486 [3424.18492] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3531] VN
_REQUEST_PBX_C_SUPPORTED 13 0xd
07:33:44.970 [3424.18492] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2053] VN_
REQUEST_SERVICE_SOCKET 6 0x6
07:33:44.970 [3424.18492] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2067] ser
vice bpcd
07:33:45.188 [3424.18492] <2> logconnections: BPCD CONNECT FROM 10.128.44.226.59
719 TO 10.128.74.45.13724 fd = 628
07:33:45.188 [3424.18492] <2> vnet_connect_to_vnetd_bpcd: js_bpcd_info: 00000000
027E1C70
07:33:45.204 [3424.18492] <2> vnet_pbxConnect_ex: pbxConnectExEx Succeeded
07:33:45.298 [3424.18492] <8> do_pbx_service: [vnet_connect.c:2581] via PBX VNET
D CONNECT FROM 10.128.44.226.59721 TO 10.128.74.45.1556 fd = 652
07:33:45.298 [3424.18492] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vne
td.c:458] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
07:33:45.547 [3424.18492] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vne
td.c:483] ipc_string 65216
07:33:45.984 [3424.18492] <2> get_short_base: (1) cannot read (byte 1) from netw
ork: An existing connection was forcibly closed by the remote host.
07:33:45.984 [3424.18492] <2> bpcr_put_vnetd_forward_socket: get_short failed: 1
0054
07:33:45.984 [3424.18492] <2> local_bpcr_connect: bpcr_put_vnetd_forward_socket
failed: 23
07:33:45.984 [3424.18492] <2> ConnectToBPCD: bpcd_connect_and_verify(mispw3206,
mispw3206) failed: 23
07:33:45.984 [3424.18492] <16> bptestbpcd main: JSON proxy message = {"allow_lar
ge_status": {"timestamp": 1530099225, "who": "vnet_tss_init", "line_number": 32,
"comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timest
amp": 1530099225, "who": "connect_to_service", "line_number": 834, "comment": "c
onnect parameters", "data": {"who": "vnet_connect_to_vnetd_bpcd", "host": "mispw
3206", "service": "vnetd", "override_required_interface": null, "extra_tries_on_
connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_opt
ions": null}}, "proxy_request": {"timestamp": 1530099225, "who": "vnet_proxy_bui
ld_request", "line_number": 3300, "comment": "request to local proxy", "data": {
"bpcd_info": {"remote_client": "mispw3206", "remote_host": "mispw3206", "remote_
probably_dhcp": false, "proxy_flag": 32, "proxy_flag_hex": "0x00000020", "cmd_so
ck": 628}, "i_am_daemon": false, "domain_constraints": {}, "service": "vnetd", "
auth_only": false, "pid": 3424, "ssa": false, "local_user": "v565751@CORP", "hos
t": "mispw3206", "peer_host": "mispw3206", "isTestConnection": true, "peer_clien
t": "mispw3206", "local_client": "to1pw3243.corp.statestr.com", "local_dhcp_inte
rval": 0, "peer_dhcp_info": null, "local_hostname": "to1pw3243.corp.statestr.com
", "secure_peer": false, "resilient": false, "nbrntd_peer_addr": null, "nbrntd_s
ock_addr": null, "hand_back": false, "request_timeout": 1810}}, "proxy_response"
: {"timestamp": 1530099225, "who": "vnet_proxy_parse_response", "line_number": 3
472, "comment": "response from local proxy", "data": {"status": 0, "local_proxy_
info": {}, "domain_constraints_set": {"process_hint": "882e988d-3244-48e9-bb12-8
b43b6f0dac7", "process_hint_reason": "there was no inherited hint, the connected
peer was not a master server, so defaulted to primary master", "process_hint_se
rver_name": "to1pw3243.corp.statestr.com"}, "statusmsgxml": "<?xml version=\"1.0
\" encoding=\"utf-8\"?><StatusMsgDoc><StatusMsg NBUStatus=\"0\" Severity=\"0\"><
/StatusMsg></StatusMsgDoc>"}}, "status": {"timestamp": 1530099225, "who": "conne
ct_to_service", "line_number": 981, "comment": "vnet status", "data": 0}}
<16>bptestbpcd main: Function ConnectToBPCD(mispw3206) failed: 23
07:33:45.984 [3424.18492] <16> bptestbpcd main: Function ConnectToBPCD(mispw3206
) failed: 23
<16>bptestbpcd main: socket read failed
07:33:46.000 [3424.18492] <16> bptestbpcd main: socket read failed
<2>bptestbpcd: socket read failed
07:33:46.015 [3424.18492] <2> bptestbpcd: socket read failed
<2>bptestbpcd: EXIT status = 23
07:33:46.015 [3424.18492] <2> bptestbpcd: EXIT status = 23
socket read failed
β06-27-2018 04:56 AM
Have you checked if NetBackup Client Services are running on the client?
If so, create bpcd log folder on the client (under ...\netbacup\logs) and run bptestbpcd again.
Check bpcd log on the client for errors.
β06-27-2018 05:29 AM
Hi Marianne,
Yes, the netbackup client service is running.
C:\Program Files\Veritas\NetBackup\bin>bpps
* MISPW3206 6/27/18 08:25:01.768
COMMAND PID LOAD TIME MEM START
vnetd 7108 0.000% 0.078 8.7M 6/27/18 07:32:44.245
bpinetd 7220 0.000% 5.070 11M 6/27/18 07:32:44.291
mtstrmd 5024 0.000% 0.046 7.3M 6/27/18 07:32:49.112
vnetd 1864 0.000% 0.046 8.2M 6/27/18 08:24:24.359
bpps 7724 0.000% 0.265 8.2M 6/27/18 08:25:00.489
bpcd log
==================
08:24:24.390 [1988.2044] <2> setup_debug_log: switched debug log file for bpcd
08:24:24.390 [1988.2044] <4> bpcd main: VERBOSE=0, BPCD VERBOSE=0 Timestamp=1530098791
08:24:24.390 [1988.2044] <2> logparams: bpcd
08:24:24.390 [1988.2044] <2> bpcd main: Got socket for input 464
08:24:24.390 [1988.2044] <2> process_requests: offset to GMT 18000
08:24:24.406 [1988.2044] <2> logconnections: BPCD ACCEPT FROM 10.128.44.226.60917 TO 10.128.74.45.13724 fd = 464
08:24:24.406 [1988.2044] <2> process_requests: setup_sockopts complete
08:24:24.437 [1988.2044] <2> bpcd peer_hostname: Connection from host to1pw3243.statestr.com (10.128.44.226) port 60917
08:24:24.546 [1988.2044] <2> vnet_pbxConnect: pbxConnectEx Succeeded
08:24:24.577 [1988.2044] <8> do_pbx_service: [vnet_connect.c:2186] via PBX VNETD CONNECT FROM 10.128.74.45.49968 TO 10.128.44.226.1556 fd = 496
08:24:24.593 [1988.2044] <8> vnet_vnetd_get_security_info: [vnet_vnetd.c:2800] VN_REQUEST_GET_SECURITY_INFO 9 0x9
08:24:25.233 [1988.2044] <8> vnet_vnetd_disconnect: [vnet_vnetd.c:198] VN_REQUEST_DISCONNECT 1 0x1
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1733] in failed file cache ERR=11001 NAME=eb60574b-b7fd-4828-a4a6-7b04cdd5d9e1 SVC=NULL
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1402] vnet_cached_getaddrinfo_and_update() failed 6 0x6
08:24:25.233 [1988.2044] <8> vnet_same_host_and_update: [vnet_addrinfo.c:3013] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME1=eb60574b-b7fd-4828-a4a6-7b04cdd5d9e1
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1733] in failed file cache ERR=11001 NAME=keystore SVC=NULL
08:24:25.233 [1988.2044] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1402] vnet_cached_getaddrinfo_and_update() failed 6 0x6
08:24:25.233 [1988.2044] <8> vnet_same_host_and_update: [vnet_addrinfo.c:3013] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME1=keystore
08:24:25.279 [1988.2044] <16> VssImportCredential: (../../libVnbat/vss_auth.cpp,3299): Cred file open FAILED
08:24:25.279 [1988.2044] <8> vnet_setup_vxss_socket: [vnet_vxss_helper.c:189] vnet_vxss_import_credential() failed 36 0x24
08:24:25.279 [1988.2044] <16> process_requests: vnet_setup_vxss_socket(to1pw3243.statestr.com) failed: 36
β06-27-2018 07:44 AM
What version of netbackup are you running ?
Try on the client running bpclntcmd -clear_host_cache
According to the logs shown, it looks like a name resolution issue
β06-27-2018 07:47 AM
The Status 36 error that I see in bpcd log is new to me.
08:24:25.279 [1988.2044] <16> process_requests: vnet_setup_vxss_socket(to1pw3243.statestr.com) failed: 36
Status Code Ref Guide says this about status 36:
Message: failed trying to allocate memory
Explanation: Allocation of system memory failed. This error occurs when an insufficient system memory is available. The system may be overloaded with too many processes and it does not have enough physical and virtual memory.
Recommended Action: Free up memory by terminating any unneeded processes that consume a lot of memory. Add more swap space or physical memory.
Can you please check if this is relevant here?
β06-27-2018 07:48 AM
Hi Nicholai,
Tried clearing host cache but didn't worked.
NBU version on master is 8.1.1 and on client is 8.0
Master and media are same.
β06-27-2018 08:18 AM
β06-27-2018 08:43 AM
Hi Marianne,
Have informed windows team to have a look into memory.
β06-27-2018 08:45 AM
Hi Tape_Archived,
Have tried all the possibilities below already but still its failing
β06-28-2018 01:30 AM
Reboot server ..
If that is not possible , stop the netbackup client and check that no bpbkar32.exe processes are running. Start the netbackup client again.
β06-28-2018 04:17 AM
Hi Nicholai,
Thanks for suggesting.
Tried stopping all the backup services and restarting them but no luck.
Following with on server reboot.
Also we observerd that the reverse lookup for backup server is not correct.
Attached screen shot.
β06-28-2018 04:50 AM - edited β06-28-2018 04:54 AM
That need to be fixed.
Reverse lookup is also a non-authorized answer. Netbackup will not trust a non-authorized DNS answer.
You can disable reverse name lookup on the client by running :
bpclntcmd -reverse_name_lookup prohibited
See also:
https://www.veritas.com/support/en_US/doc/15263389-127350397-0/v14660333-127350397
β06-28-2018 04:52 AM
Try to add both hostnames in the Client's SERVER list:
to1pw3243.statestr.com
to1pw3243.corp.statestr.com
It seems that 'bpclntcmd -pn' did not produce any output?
This command will connect to the master server (1st entry in the Client's Server list) on port 1556 or 13724.
If you are getting no response, it means that the Client cannot connect to master server.
(Best to leave command running until it times out or produce output.)
Check bprd log on the master to see if connection was received from client IP
The output should look like this:
expecting response from server <master-name>
<client-name-resolved-from-IP> <client-name-in-policy> <Client-source-IP> <Client-source-port-number>
By the looks of it, you are experiencing name lookup issues in your environment.
Is this with one client only? Or more?
Have you asked your network/DNS admin?
β06-28-2018 06:32 AM
Hi Marianne,
bpclntcmd -pn command doesn't shows any output after entering both the below entries
to1pw3243.statestr.com
to1pw3243.corp.statestr.com
Also find below answer to queries
Is this with one client only? Or more? :- One client
Have you asked your network/DNS admin? :- Yes, they are looking into it.
β06-28-2018 06:42 AM
Hi Nicolai,
Tried below command but no luck.
bpclntcmd -reverse_name_lookup prohibited
β06-28-2018 08:13 AM
Please see this part of my previous post:
If you are getting no response, it means that the Client cannot connect to master server.
(Best to leave command running until it times out or produce output.)
Check bprd log on the master to see if connection was received from client IP.