Highlighted

bptestbpcd: cannot connect on socket EXIT status = 25

Master: 7.1.0.4

Windows Server 2003

Clients are inside DMZ including the Master and telnet is not allowed. Some of the clients have issue (cannot connect on socket EXIT status = 25)

Rest of the clients are working fine.

On Clients Multiple IP Scheme is used as shown in the Problematic Client Screen Shot. But the same IP scheme is on the working clients.

Error:

errora.JPG

 

Problematic Client:

1.JPG

Troubleshooting:

Bptestbpcd –client ECOMWEBCA100 -verbose

errorb.JPG

errorc.JPG

Also Checked and Tested the Working Client as well as the Problematic

Working Client Output:

D:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -verbose -debug -clie
nt SAIWILYCOLL100
10:12:22.558 [668.5320] <2> bptestbpcd: VERBOSE = 0
10:12:22.574 [668.5320] <2> vnet_pbxConnect: pbxConnectEx Succeeded
10:12:22.574 [668.5320] <2> logconnections: BPCD CONNECT FROM 192.168.133.20.279
4 TO 192.168.132.34.1556 fd = 1636
10:12:22.589 [668.5320] <2> vnet_pbxConnect: pbxConnectEx Succeeded
10:12:22.652 [668.5320] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1776:
0: via PBX: VNETD CONNECT FROM 192.168.133.20.2795 TO 192.168.132.34.1556 fd = 1
608
10:12:22.667 [668.5320] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvl
ibs/vnet_vnetd.c.445: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
10:12:22.871 [668.5320] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvl
ibs/vnet_vnetd.c.462: 0: ipc_string: 58240
1 1 1
192.168.133.20:2794 -> 192.168.132.34:1556
192.168.133.20:2795 -> 192.168.132.34:1556
10:12:23.746 [668.5320] <2> bpcr_get_peername_rqst: Server peername length = 33
10:12:23.964 [668.5320] <2> bpcr_get_hostname_rqst: Server hostname length = 37
10:12:24.183 [668.5320] <2> bpcr_get_clientname_rqst: Server clientname length =
 37
10:12:24.402 [668.5320] <2> bpcr_get_version_rqst: bpcd version: 07500000
10:12:24.511 [668.5320] <2> bpcr_get_platform_rqst: Server platform length = 7
10:12:24.730 [668.5320] <2> bpcr_get_version_rqst: bpcd version: 07500000
10:12:24.949 [668.5320] <2> bpcr_patch_version_rqst: theRest == > <
10:12:24.949 [668.5320] <2> bpcr_get_version_rqst: bpcd version: 07500000
10:12:25.277 [668.5320] <2> bpcr_patch_version_rqst: theRest == > <
10:12:25.277 [668.5320] <2> bpcr_get_version_rqst: bpcd version: 07500000
PEER_NAME = ecombkup02.aceinet.aaaclubnet.com
HOST_NAME = saiwilycoll100.aceinet.aaaclubnet.com
CLIENT_NAME = saiwilycoll100.aceinet.aaaclubnet.com
VERSION = 0x07500000
PLATFORM = win_x64
PATCH_VERSION = 7.5.0.0
SERVER_PATCH_VERSION = 7.5.0.0
MASTER_SERVER = Ecombkup02.aceinet.aaaclubnet.com
EMM_SERVER = Ecombkup02.aceinet.aaaclubnet.com
NB_MACHINE_TYPE = CLIENT
10:12:25.511 [668.5320] <2> vnet_pbxConnect: pbxConnectEx Succeeded
10:12:25.527 [668.5320] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1776:
0: via PBX: VNETD CONNECT FROM 192.168.133.20.2796 TO 192.168.132.34.1556 fd = 1
588
10:12:25.527 [668.5320] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvl
ibs/vnet_vnetd.c.445: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
10:12:25.714 [668.5320] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvl
ibs/vnet_vnetd.c.462: 0: ipc_string: 58242
192.168.133.20:2796 -> 192.168.132.34:1556
<2>bptestbpcd: EXIT status = 0
10:12:26.136 [668.5320] <2> bptestbpcd: EXIT status = 0

 

Problematic Client Output

D:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -verbose -debug -clie
nt ECOMWEBCA100
10:14:36.291 [5824.1140] <2> bptestbpcd: VERBOSE = 0
10:15:06.306 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:15:07.259 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:15:17.212 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:15:48.212 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:16:00.212 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:16:00.212 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:16:11.258 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:16:41.258 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:16:42.195 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:16:52.273 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:17:22.273 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:17:23.210 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:17:33.273 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:18:03.272 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:18:05.272 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:18:14.194 [5824.1140] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1
376: 0: getsockopt SO_ERROR returned: 10060 0x0000274c
10:18:14.194 [5824.1140] <2> connect_to_service: ../../libvlibs/vnet_connect.c.3
82: 0: vnet_async_connect() failed: 18 0x00000012
10:18:14.194 [5824.1140] <2> vnet_connect_to_bpcd: ../../libvlibs/vnet_connect.c
.237: 0: connect_to_service() failed: 18 0x00000012
10:18:14.194 [5824.1140] <2> local_bpcr_connect: Can't connect to client ECOMWEB
CA100
10:18:14.194 [5824.1140] <2> ConnectToBPCD: bpcd_connect_and_verify(ECOMWEBCA100
, ECOMWEBCA100) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(ECOMWEBCA100) failed: 25
10:18:14.194 [5824.1140] <16> bptestbpcd main: Function ConnectToBPCD(ECOMWEBCA1
00) failed: 25
<2>bptestbpcd: cannot connect on socket
10:18:14.194 [5824.1140] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
10:18:14.194 [5824.1140] <2> bptestbpcd: EXIT status = 25
cannot connect on socket

Checking that the Daemon Binary is Not Corrupt:

bpcd log attached: 051616.log

13:30:51.148 [704.4676] <2> setup_debug_log: switched debug log file for bpcd
13:30:51.148 [704.4676] <2> bpcd main: VERBOSE = 0
13:30:51.148 [704.4676] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:30:51.148 [704.4676] <2> process_requests: offset to GMT 28800
13:30:51.163 [704.4676] <2> logconnections: BPCD ACCEPT FROM 192.168.133.20.607 TO 0.0.0.0.13782 fd = 300
13:30:51.163 [704.4676] <2> process_requests: setup_sockopts complete
13:30:51.163 [704.4676] <2> bpcd peer_hostname: Connection from host Ecombkup02.aceinet.aaaclubnet.com (192.168.133.20) port 607
13:30:51.163 [704.4676] <2> bpcd valid_server: comparing 192.168.133.20 and Ecombkup02.aceinet.aaaclubnet.com
13:30:51.163 [704.4676] <4> bpcd valid_server: hostname comparison succeeded
13:30:51.163 [704.4676] <16> process_requests: read failed: An existing connection was forcibly closed by the remote host. 
13:31:32.397 [4048.4204] <2> setup_debug_log: switched debug log file for bpcd
13:31:32.397 [4048.4204] <2> bpcd main: VERBOSE = 0
13:31:32.397 [4048.4204] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:31:32.397 [4048.4204] <2> process_requests: offset to GMT 28800
13:31:32.397 [4048.4204] <2> logconnections: BPCD ACCEPT FROM 192.168.133.20.655 TO 0.0.0.0.13782 fd = 300
13:31:32.397 [4048.4204] <2> process_requests: setup_sockopts complete
13:31:32.397 [4048.4204] <2> bpcd peer_hostname: Connection from host Ecombkup02.aceinet.aaaclubnet.com (192.168.133.20) port 655
13:31:32.397 [4048.4204] <2> bpcd valid_server: comparing 192.168.133.20 and Ecombkup02.aceinet.aaaclubnet.com
13:31:32.397 [4048.4204] <4> bpcd valid_server: hostname comparison succeeded
13:31:32.397 [4048.4204] <16> process_requests: read failed: An existing connection was forcibly closed by the remote host. 
13:34:08.676 [4900.4376] <2> setup_debug_log: switched debug log file for bpcd
13:34:08.676 [4900.4376] <2> bpcd main: VERBOSE = 0
13:34:08.676 [4900.4376] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:34:08.676 [4900.4376] <2> process_requests: offset to GMT 28800
13:34:08.676 [4900.4376] <2> logconnections: BPCD ACCEPT FROM 192.168.132.24.60242 TO 192.168.132.24.13782 fd = 300
13:34:08.676 [4900.4376] <2> process_requests: setup_sockopts complete
13:34:08.676 [4900.4376] <2> bpcd peer_hostname: Connection from host ECOMWEBCA100.aceinet.aaaclubnet.com (192.168.132.24) port 60242
13:34:08.676 [4900.4376] <2> bpcd valid_server: comparing 192.168.133.20 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:08.676 [4900.4376] <2> bpcd valid_server: comparing Ecombkup02.aceinet.aaaclubnet.com and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:08.676 [4900.4376] <2> bpcd valid_server: comparing Ecombkup02 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:08.676 [4900.4376] <4> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server
13:34:08.676 [4900.4376] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either
13:34:08.676 [4900.4376] <2> process_requests: output socket port number = 4481
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+bprd,1,1,0,1,0+.txt
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+bpdbm,1,1,0,1,0+.txt
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+vmd,1,1,0,1,0+.txt
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.676 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+tldcd,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+tl8cd,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+tlhcd,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+tl4d,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+tlmd,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+tshd,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+acsd,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+bpjobd,1,1,0,1,0+.txt
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.692 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\1ff\ffffffff+testdaemon,1,1,0,1,0+.txt
13:34:08.708 [4900.4376] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2474: 0: vnet_cached_getaddrinfo failed: 6 0x00000006
13:34:08.708 [4900.4376] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2475: 0: vnet_cached_getaddrinfo failed ret: 10109 0x0000277d
13:34:08.708 [4900.4376] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2476: 0: vnet_cached_getaddrinfo failed service: testdaemon
13:34:08.708 [4900.4376] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1827: 0: vnet_cached_get_service_port() failed: 6 0x00000006
13:34:08.708 [4900.4376] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1828: 0: service: 4481
13:34:08.708 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:08.708 [4900.4376] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\0f5\b1955ef5+4481,1,0,2,1,0+192.168.132.24.txt
13:34:08.708 [4900.4376] <2> process_requests: <---- NetBackup 7.1 0 ------------initiated
13:34:08.708 [4900.4376] <2> process_requests: VERBOSE = 0
13:34:08.708 [4900.4376] <2> bpcd exit_bpcd: exit status 46  ----------->exiting
13:34:08.708 [4900.4376] <4> bpcd exit_bpcd: FTL - BPCD EXIT STATUS 46

13:34:08.708 [4900.4376] <16> process_requests: Server access denied
13:34:11.286 [3584.2708] <2> setup_debug_log: switched debug log file for bpcd
13:34:11.286 [3584.2708] <2> bpcd main: VERBOSE = 0
13:34:11.286 [3584.2708] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:34:11.286 [3584.2708] <2> process_requests: offset to GMT 28800
13:34:11.286 [3584.2708] <2> logconnections: BPCD ACCEPT FROM 192.168.132.24.60244 TO 192.168.132.24.13782 fd = 300
13:34:11.286 [3584.2708] <2> process_requests: setup_sockopts complete
13:34:11.286 [3584.2708] <2> bpcd peer_hostname: Connection from host ECOMWEBCA100.aceinet.aaaclubnet.com (192.168.132.24) port 60244
13:34:11.286 [3584.2708] <2> bpcd valid_server: comparing 192.168.133.20 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:11.286 [3584.2708] <2> bpcd valid_server: comparing Ecombkup02.aceinet.aaaclubnet.com and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:11.286 [3584.2708] <2> bpcd valid_server: comparing Ecombkup02 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:11.286 [3584.2708] <4> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server
13:34:11.286 [3584.2708] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either
13:34:11.286 [3584.2708] <16> process_requests: read failed: The operation completed successfully. 
13:34:11.286 [3184.4200] <2> setup_debug_log: switched debug log file for bpcd
13:34:11.286 [3184.4200] <2> bpcd main: VERBOSE = 0
13:34:11.286 [3184.4200] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:34:11.286 [3184.4200] <2> process_requests: offset to GMT 28800
13:34:11.286 [3184.4200] <2> logconnections: BPCD ACCEPT FROM 192.168.132.24.60245 TO 192.168.132.24.13782 fd = 300
13:34:11.286 [3184.4200] <2> process_requests: setup_sockopts complete
13:34:11.301 [3184.4200] <2> bpcd peer_hostname: Connection from host ECOMWEBCA100.aceinet.aaaclubnet.com (192.168.132.24) port 60245
13:34:11.301 [3184.4200] <2> bpcd valid_server: comparing 192.168.133.20 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:11.301 [3184.4200] <2> bpcd valid_server: comparing Ecombkup02.aceinet.aaaclubnet.com and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:11.301 [3184.4200] <2> bpcd valid_server: comparing Ecombkup02 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:34:11.301 [3184.4200] <4> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server
13:34:11.301 [3184.4200] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either
13:34:11.301 [3184.4200] <2> process_requests: output socket port number = 4113
13:34:11.301 [3184.4200] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1527: 0: in failed file cache: 10109 0x0000277d
13:34:11.301 [3184.4200] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1528: 0: in failed file cache name: NULL
13:34:11.301 [3184.4200] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1529: 0: in failed file cache service: testdaemon
13:34:11.301 [3184.4200] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2474: 0: vnet_cached_getaddrinfo failed: 6 0x00000006
13:34:11.301 [3184.4200] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2475: 0: vnet_cached_getaddrinfo failed ret: 10109 0x0000277d
13:34:11.301 [3184.4200] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2476: 0: vnet_cached_getaddrinfo failed service: testdaemon
13:34:11.301 [3184.4200] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1827: 0: vnet_cached_get_service_port() failed: 6 0x00000006
13:34:11.301 [3184.4200] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1828: 0: service: 4113
13:34:11.317 [3184.4200] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:34:11.317 [3184.4200] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\0f5\b1955ef5+4113,1,0,2,1,0+192.168.132.24.txt
13:34:11.317 [3184.4200] <2> process_requests: <---- NetBackup 7.1 0 ------------initiated
13:34:11.317 [3184.4200] <2> process_requests: VERBOSE = 0
13:34:11.317 [3184.4200] <2> bpcd exit_bpcd: exit status 46  ----------->exiting
13:34:11.317 [3184.4200] <4> bpcd exit_bpcd: FTL - BPCD EXIT STATUS 46

13:34:11.317 [3184.4200] <16> process_requests: Server access denied
13:38:37.986 [2268.3568] <2> setup_debug_log: switched debug log file for bpcd
13:38:37.986 [2268.3568] <2> bpcd main: VERBOSE = 0
13:38:37.986 [2268.3568] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:38:37.986 [2268.3568] <2> process_requests: offset to GMT 28800
13:38:37.986 [2268.3568] <2> logconnections: BPCD ACCEPT FROM 192.168.132.24.60267 TO 192.168.132.24.13782 fd = 300
13:38:37.986 [2268.3568] <2> process_requests: setup_sockopts complete
13:38:37.986 [2268.3568] <2> bpcd peer_hostname: Connection from host ECOMWEBCA100.aceinet.aaaclubnet.com (192.168.132.24) port 60267
13:38:37.986 [2268.3568] <2> bpcd valid_server: comparing 192.168.133.20 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:37.986 [2268.3568] <2> bpcd valid_server: comparing Ecombkup02.aceinet.aaaclubnet.com and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:37.986 [2268.3568] <2> bpcd valid_server: comparing Ecombkup02 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:37.986 [2268.3568] <4> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server
13:38:37.986 [2268.3568] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either
13:38:37.986 [2268.3568] <2> process_requests: output socket port number = 4481
13:38:37.986 [2268.3568] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1527: 0: in failed file cache: 10109 0x0000277d
13:38:37.986 [2268.3568] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1528: 0: in failed file cache name: NULL
13:38:37.986 [2268.3568] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1529: 0: in failed file cache service: testdaemon
13:38:37.986 [2268.3568] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2474: 0: vnet_cached_getaddrinfo failed: 6 0x00000006
13:38:37.986 [2268.3568] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2475: 0: vnet_cached_getaddrinfo failed ret: 10109 0x0000277d
13:38:37.986 [2268.3568] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2476: 0: vnet_cached_getaddrinfo failed service: testdaemon
13:38:37.986 [2268.3568] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1827: 0: vnet_cached_get_service_port() failed: 6 0x00000006
13:38:37.986 [2268.3568] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1828: 0: service: 4481
13:38:38.001 [2268.3568] <2> process_requests: <---- NetBackup 7.1 0 ------------initiated
13:38:38.001 [2268.3568] <2> process_requests: VERBOSE = 0
13:38:38.001 [2268.3568] <2> bpcd exit_bpcd: exit status 46  ----------->exiting
13:38:38.001 [2268.3568] <4> bpcd exit_bpcd: FTL - BPCD EXIT STATUS 46

13:38:38.001 [2268.3568] <16> process_requests: Server access denied
13:38:38.017 [3692.4216] <2> setup_debug_log: switched debug log file for bpcd
13:38:38.017 [3692.4216] <2> bpcd main: VERBOSE = 0
13:38:38.017 [3692.4216] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:38:38.017 [3692.4216] <2> process_requests: offset to GMT 28800
13:38:38.017 [4668.3012] <2> setup_debug_log: switched debug log file for bpcd
13:38:38.017 [4668.3012] <2> bpcd main: VERBOSE = 0
13:38:38.017 [4668.3012] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:38:38.017 [4668.3012] <2> process_requests: offset to GMT 28800
13:38:38.017 [3692.4216] <2> logconnections: BPCD ACCEPT FROM 192.168.132.24.60270 TO 192.168.132.24.13782 fd = 300
13:38:38.017 [3692.4216] <2> process_requests: setup_sockopts complete
13:38:38.017 [4668.3012] <2> logconnections: BPCD ACCEPT FROM 192.168.132.24.60269 TO 192.168.132.24.13782 fd = 300
13:38:38.017 [4668.3012] <2> process_requests: setup_sockopts complete
13:38:38.017 [3692.4216] <2> bpcd peer_hostname: Connection from host ECOMWEBCA100.aceinet.aaaclubnet.com (192.168.132.24) port 60270
13:38:38.017 [3692.4216] <2> bpcd valid_server: comparing 192.168.133.20 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:38.017 [4668.3012] <2> bpcd peer_hostname: Connection from host ECOMWEBCA100.aceinet.aaaclubnet.com (192.168.132.24) port 60269
13:38:38.017 [4668.3012] <2> bpcd valid_server: comparing 192.168.133.20 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:38.017 [3692.4216] <2> bpcd valid_server: comparing Ecombkup02.aceinet.aaaclubnet.com and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:38.017 [3692.4216] <2> bpcd valid_server: comparing Ecombkup02 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:38.017 [4668.3012] <2> bpcd valid_server: comparing Ecombkup02.aceinet.aaaclubnet.com and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:38.017 [4668.3012] <2> bpcd valid_server: comparing Ecombkup02 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:38:38.017 [3692.4216] <4> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server
13:38:38.017 [3692.4216] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either
13:38:38.017 [3692.4216] <2> process_requests: output socket port number = 3629
13:38:38.017 [4668.3012] <4> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server
13:38:38.017 [4668.3012] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either
13:38:38.017 [4668.3012] <16> process_requests: read failed: The operation completed successfully. 
13:38:38.017 [3692.4216] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1527: 0: in failed file cache: 10109 0x0000277d
13:38:38.017 [3692.4216] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1528: 0: in failed file cache name: NULL
13:38:38.017 [3692.4216] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1529: 0: in failed file cache service: testdaemon
13:38:38.017 [3692.4216] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2474: 0: vnet_cached_getaddrinfo failed: 6 0x00000006
13:38:38.017 [3692.4216] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2475: 0: vnet_cached_getaddrinfo failed ret: 10109 0x0000277d
13:38:38.017 [3692.4216] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2476: 0: vnet_cached_getaddrinfo failed service: testdaemon
13:38:38.017 [3692.4216] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1827: 0: vnet_cached_get_service_port() failed: 6 0x00000006
13:38:38.017 [3692.4216] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1828: 0: service: 3629
13:38:38.017 [3692.4216] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:38:38.017 [3692.4216] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: D:\Program Files\Veritas\NetBackup\var\host_cache\0f5\b1955ef5+3629,1,0,2,1,0+192.168.132.24.txt
13:38:38.017 [3692.4216] <2> process_requests: <---- NetBackup 7.1 0 ------------initiated
13:38:38.017 [3692.4216] <2> process_requests: VERBOSE = 0
13:38:38.017 [3692.4216] <2> bpcd exit_bpcd: exit status 46  ----------->exiting
13:38:38.017 [3692.4216] <4> bpcd exit_bpcd: FTL - BPCD EXIT STATUS 46

13:38:38.017 [3692.4216] <16> process_requests: Server access denied
13:39:56.344 [2552.588] <2> setup_debug_log: switched debug log file for bpcd
13:39:56.344 [2552.588] <2> bpcd main: VERBOSE = 5
13:39:56.344 [2552.588] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone 
13:39:56.344 [2552.588] <2> ReadKeyfile: keyfile D:\Program Files\Veritas\NetBackup\bin\keyfile.dat does not exist

13:39:56.344 [2552.588] <2> process_requests: offset to GMT 28800
13:39:56.344 [2552.588] <2> logconnections: BPCD ACCEPT FROM 127.0.0.1.60275 TO 127.0.0.1.13782 fd = 300
13:39:56.344 [2552.588] <2> process_requests: setup_sockopts complete
13:39:56.344 [2552.588] <2> vnet_pcache_init_table: ../../libvlibs/vnet_private.c.232: 0: starting cache size: 200 0x000000c8
13:39:56.344 [2552.588] <2> vnet_cached_getnameinfo: ../../libvlibs/vnet_addrinfo.c.1837: 0: found via getnameinfo our_host: ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getnameinfo: ../../libvlibs/vnet_addrinfo.c.1838: 0: found via getnameinfo ipstr: 127.0.0.1
13:39:56.344 [2552.588] <2> bpcd peer_hostname: Connection from host ECOMWEBCA100.aceinet.aaaclubnet.com (127.0.0.1) port 60275
13:39:56.344 [2552.588] <2> bpcd valid_server: comparing 192.168.133.20 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_pcache_init_table: ../../libvlibs/vnet_private.c.232: 0: starting cache size: 200 0x000000c8
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1514: 0: found in file cache name: 192.168.133.20
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1515: 0: found in file cache service: NULL
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1514: 0: found in file cache name: ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1515: 0: found in file cache service: NULL
13:39:56.344 [2552.588] <2> bpcd valid_server: comparing Ecombkup02.aceinet.aaaclubnet.com and ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1514: 0: found in file cache name: Ecombkup02.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1515: 0: found in file cache service: NULL
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1370: 0: found in cache name: ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1371: 0: found in cache service: NULL
13:39:56.344 [2552.588] <2> bpcd valid_server: comparing Ecombkup02 and ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1514: 0: found in file cache name: Ecombkup02
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1515: 0: found in file cache service: NULL
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1370: 0: found in cache name: ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1371: 0: found in cache service: NULL
13:39:56.344 [2552.588] <4> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server
13:39:56.344 [2552.588] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1370: 0: found in cache name: ECOMWEBCA100.aceinet.aaaclubnet.com
13:39:56.344 [2552.588] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1371: 0: found in cache service: NULL
13:40:26.765 [2552.588] <16> process_requests: read failed: The operation completed successfully. 

 

Followed the Following Tech Article:

1- https://www.veritas.com/support/en_US/article.TECH141839

2- https://www.veritas.com/support/en_US/article.TECH68832

Please your solution is needed. Thanks in Advance

 

1 Solution

Accepted Solutions
Highlighted
Accepted Solution!

Hi All, Sorry for long delay

Hi All,

Sorry for long delay on this, due to customer avialability. This issue is resolved, below are the details.

- Checked bpcd logs on Master server and also on client as well.

- Analyzed Network traffic on client using Wireshark and confirmed that Master server initiates connection but client unable to reply back.

- Compared Routing Table on both clients using “netstat –r” one that is working and other one that’s not working and found that Static  routes need to be added on client (ECOMWEBCA100)  to establish successful connection with Master server.

- After Adding Static  routes on client (ECOMWEBCA100)  and establish successful connection with Master server.

This was Network issue of the Client in the DMZ.

View solution in original post

18 Replies
Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

ECOMWEBCA100.aceinet.aaaclubnet.com is not a master server 13:34:11.301 [3184.4200] <16> bpcd valid_server: ECOMWEBCA100.aceinet.aaaclubnet.com is not a media server either Add ECOMWEBCA100.aceinet.aaaclubnet.com to Server entries on the client. Double-check that port 1556 is open in both directions between client and master and client and media server. Check that Windows firewall is turned off on the client. I see connection to client is only attempted on port 13782. Why is that? NBU 7.0.1 and above will by default first try 1556, then 13724 and only then 13782. Has Connect Options been changed for this client?
Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

If the client servers are behind the firewall then you have to request the firewall team to allow the ports bidirectional with Master & Media server .
Highlighted

Hi

Hi Marianne,

Yes ECOMWEBCA100.aceinet.aaaclubnet.com it is the Problematic Client

Master is: Ecombkup02.aceinet.aaaclubnet.com

Already Added: ECOMWEBCA100.aceinet.aaaclubnet.com

1556 is open and listening both directions

Firewall on ECOMWEBCA100.aceinet.aaaclubnet.com is off by default as well as on Working Client

Today i Figure Out:

Working Client: Installed with NB 7.5 Client

Problematic Client: Installed with NB 7.1 Client

Told the Client to Install 7.5 Client

 

So lets see.

 

Highlighted

Hi John, Already did that and

Hi John,

Already did that and all ports fine in Bidirectional

Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

13:38:38.001 [2268.3568] <2> bpcd exit_bpcd: exit status 46  ----------->exiting
13:38:38.001 [2268.3568] <4> bpcd exit_bpcd: FTL - BPCD EXIT STATUS 46

13:38:38.001 [2268.3568] <16> process_requests: Server access denied

I feel you must have hosts entries for Master on client and vise versa.

Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

So, the status 46 was a connection from the client itself. Who was doing what here? As per my previous post: I see connection to client is only attempted on port 13782. Why is that? NBU 7.0.1 and above will by default first try 1556, then 13724 and only then 13782. Has Connect Options been changed for this client? Check Client Connect options on the master server with bpclient or in the GUI under Host Properties -> Master -> Client Attributes.
Highlighted

Hi Pat, On the client in

Hi Pat,

On the client in Hosts file there are Entries for the Master as well as Media server 

Highlighted

Hi Marianne, That was me to

Hi Marianne,

That was me to have logs for the bpcd standalone. Look below (followed technote: https://www.veritas.com/support/en_US/article.TECH68832)

Checking that the Daemon Binary is Not Corrupt

For vnetd and bpcd connection problems, on destination hosts, confirm that the executable binary file is not corrupt.

On the destination host, create the debug log directory for the process, set verbose to 5, and try to start the daemon from the command line.  Be sure the program executes with appropriate administrative privileges.  E.g.

<install path>/netbackup/bin/bpcd

Checking Local Connectivity
If the daemon process appears to be listening, but not accepting connections from remote hosts, then test a local connection on the destination host to itself.  Typically first to PBX, then to the specific port for the destination daemon.  E.g.

telnet 127.0.0.1 1556
telnet 127.0.0.1 13782
 

Client Connection options not changed.

I am waiitng for the Client to change the NB Client on the Problematic Client to version 7.5

Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

So, have you done the telnet tests? I hope that upgrading the client will fix your issues but I have some reservations about it. Connection port decision is made by the server. We never see any attempt on 1556 or 13724. Only 13782. This is why I asked to check Connect Options on the master server.
Highlighted

Hi Marianne, Yes i did that

Hi Marianne,

Yes i did that telnet tests and the output is in the original post. Yeah waiting for the customer to do so, will update you.

I agree with you but they are in listening state, and i will re check again when in next session.

Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

Hi Marianne,

Unfortunately installing the 7.5 Client didn't fixed the issue and it is still the same.

What's your recommendations? what should be done next?

 

Regards,

Highlighted

Is the client Windows

Is the client Windows too?

.

On master, what does this show:

"C:\Program Files\Veritas\NetBackup\bin\bpclient.exe" -L -client ECOMWEBCA100.aceinet.aaaclubnet.com

 

Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

Hi Marianne,

I am here with some screen shots which shows you the clear picture.

This is the step which was performed exat the same time when running the test. Please see Images

netstate -o -a

Captur1we.PNG

 

Untitled picture.png

Highlighted

Hi SDO, Client is Windows

Hi SDO,

Client is Windows

 

Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

As per my suggestion of 2 days ago: Check Client Connect options on the master server with bpclient command or in the GUI under Host Properties -> Master -> Client Attributes.
Highlighted

Hi Marianne, i will be

Hi Marianne,

i will be continuing this with Client on the 1st of June, so this is postpond till that time.

Highlighted

Re: bptestbpcd: cannot connect on socket EXIT status = 25

Please check output of bpclient command as suggested by sdo. It will tell us if Client Connect options has been set for the client on the master server. Check shortname and FQDN.
Highlighted
Accepted Solution!

Hi All, Sorry for long delay

Hi All,

Sorry for long delay on this, due to customer avialability. This issue is resolved, below are the details.

- Checked bpcd logs on Master server and also on client as well.

- Analyzed Network traffic on client using Wireshark and confirmed that Master server initiates connection but client unable to reply back.

- Compared Routing Table on both clients using “netstat –r” one that is working and other one that’s not working and found that Static  routes need to be added on client (ECOMWEBCA100)  to establish successful connection with Master server.

- After Adding Static  routes on client (ECOMWEBCA100)  and establish successful connection with Master server.

This was Network issue of the Client in the DMZ.

View solution in original post