cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup client connectivity issue

suparman7
Level 3

bptestbpcd from master server to client is failing with below error, although able to connect all the backup ports and name resolution is fine and also able to issue certificate successfully. Still backups are failing with error 58 can't connect client

vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.94: pbxRegisterExEx failed with error 110:Connection timed outdaemon_select_and_accept_ex: vnet_registerPBXServer() failed: 47

BPTESTBPCD

/usr/openv/netbackup/bin/admincmd/bptestbpcd -verbose -debug -client XXXXXXXX.XXXX.XXXX
14:32:29.524 [30023] <2> bptestbpcd: VERBOSE = 0
14:32:29.525 [30023] <2> ConnectionCache::connectAndCache: Acquiring new connection for host XXXXXXXX.XX.XXXX, query type 223
14:32:29.528 [30023] <2> logconnections: BPDBM CONNECT FROM XXX.XXX.XXX.XXX TO XXX.XXX.XXX.XXX fd = 3
14:32:29.529 [30023] <2> db_CLIENTsend: reset client protocol version from 0 to 9
14:32:29.533 [30023] <2> db_getCLIENT: db_CLIENTreceive: no entity was found 227
14:32:29.533 [30023] <2> closeConnection: Caching connection for query type 223 for reuse
14:32:29.542 [30023] <2> vnet_pbxConnect_ex: pbxConnectExEx() failed: 104
14:32:29.542 [30023] <2> vnet_pbxConnect_ex: ../../libvlibs/vnet_pbx.c.674: pbxSetAddrEx/pbxConnectExEx return error 104:Connection reset by peer
14:32:29.542 [30023] <8> do_pbx_service: [vnet_connect.c:4012] vnet_pbxConnect() failed, status=18, errno=2, use_vnetd=0, cr->vcr_service=bpcd
14:32:29.542 [30023] <8> async_connect: [vnet_connect.c:3543] do_service failed 18 0x12
14:32:29.544 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:30.547 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:32.549 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:36.551 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:44.554 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:44.554 [30023] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED status: FAILED, (24) BAD_VERSION; system: (2) No such file or directory; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA pbx status: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA vnetd
14:32:44.554 [30023] <16> connect_to_service: JSON data = {"allow_large_status": {"timestamp": 1672218149, "who": "vnet_tss_init", "line_number": 32, "comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timestamp": 1672218149, "who": "connect_to_service", "line_number": 2199, "comment": "connect parameters", "data": {"who": "vnet_connect_to_bpcd", "host": "XXXXXXXX.XXXX.XXXX", "service": "bpcd", "override_required_interface": null, "extra_tries_on_connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_options": {"server": null, "callback_kind": {"number": 1, "symbol": "NBCONF_CALLBACK_KIND_VNETD", "description": "Vnetd"}, "daemon_port_type": {"number": 0, "symbol": "NBCONF_DAEMON_PORT_TYPE_AUTOMATIC", "description": "Automatic"}, "reserved_port_kind": {"number": 0, "symbol": "NBCONF_RESERVED_PORT_KIND_LEGACY", "description": "Legacy"}}}}, "status": {"timestamp": 1672218164, "who": "connect_to_service", "line_number": 2465, "comment": "vnet status", "data": 18}, "connect_recs": {"timestamp": 1672218164, "who": "vnet_tss_get", "line_number": 97, "comment": "connect rec status messages", "data": "connect failed STATUS (18) CONNECT_FAILED\n\tstatus: FAILED, (24) BAD_VERSION; system: (2) No such file or directory; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA pbx\n\tstatus: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA vnetd"}}
14:32:44.554 [30023] <8> vnet_connect_to_bpcd: [vnet_connect.c:623] connect_to_service() failed 18 0x12
14:32:44.554 [30023] <16> local_bpcr_connect: vnet_connect_to_bpcd(XXXXXXXX.XXXX.XXXX) failed: 18
14:32:44.554 [30023] <2> local_bpcr_connect: Can't connect to client XXXXXXXX.XXXX.XXXX
14:32:44.554 [30023] <2> ConnectToBPCD: bpcd_connect_and_verify(XXXXXXXX.XXXX.XXXX, XXXXXXXX.XXXX.XXXX) failed: 25
14:32:44.554 [30023] <16> bptestbpcd main: JSON proxy message = {"allow_large_status": {"timestamp": 1672218149, "who": "vnet_tss_init", "line_number": 32, "comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timestamp": 1672218149, "who": "connect_to_service", "line_number": 2199, "comment": "connect parameters", "data": {"who": "vnet_connect_to_bpcd", "host": "XXXXXXXX.XXXX.XXXX", "service": "bpcd", "override_required_interface": null, "extra_tries_on_connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_options": {"server": null, "callback_kind": {"number": 1, "symbol": "NBCONF_CALLBACK_KIND_VNETD", "description": "Vnetd"}, "daemon_port_type": {"number": 0, "symbol": "NBCONF_DAEMON_PORT_TYPE_AUTOMATIC", "description": "Automatic"}, "reserved_port_kind": {"number": 0, "symbol": "NBCONF_RESERVED_PORT_KIND_LEGACY", "description": "Legacy"}}}}, "status": {"timestamp": 1672218164, "who": "connect_to_service", "line_number": 2465, "comment": "vnet status", "data": 18}, "connect_recs": {"timestamp": 1672218164, "who": "vnet_tss_get", "line_number": 97, "comment": "connect rec status messages", "data": "connect failed STATUS (18) CONNECT_FAILED\n\tstatus: FAILED, (24) BAD_VERSION; system: (2) No such file or directory; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA pbx\n\tstatus: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA vnetd"}}
<16>bptestbpcd main: Function ConnectToBPCD(XXXXXXXX.XXXX.XXXX) failed: 25
14:32:44.554 [30023] <16> bptestbpcd main: Function ConnectToBPCD(XXXXXXXX.XXXX.XXXX) failed: 25
<16>bptestbpcd main: cannot connect on socket
14:32:44.559 [30023] <16> bptestbpcd main: cannot connect on socket
<2>bptestbpcd: cannot connect on socket
14:32:44.559 [30023] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
14:32:44.559 [30023] <2> bptestbpcd: EXIT status = 25

 

 

7 REPLIES 7

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @suparman7 

What version of Netbackup is listing at the client ?

Is there firewall between NBU and the client ?

What OS is the client ?

Netbackup will per default connect on PBX port 1556, is that port isn't available, Netbackup will try 13724 (vnetd) and 13782 in that order.  

I see both connection to PBX (1556) and 13724 (Vnetd) failing.

Try from the client side to use wireshark or tcpdump to see if connection requests is really reaching the client, it doesn't look like it.

 

Hamza_H
Moderator
Moderator
   VIP   
Hi,
Was this servers ip address used for another server before? If yes, check the mapping if the bad server’s name is mapped to your server.
We had a similar issue for this.. there was no FW, telnet was OK in both directions.. nbu commands were working fine (bpclntcmd, bptestnetconn..) but bptestbpcd was failing and backups as well..
There were no actions and it worked by itself.. (one of netbackups mysteries..)
What netbackup version you are using (master, media and client)?

I have telnet all NBU ports and are connecting fine both ways from client to master and vice versa... I have tried to reinstall and was able to issue a certificate too but when i tried to connect to client or bptestbpcd it fails...

All Master, Media and client version 9.1.0.1

All required ports are connecting and i have tried to run a

traceroute, nslookup and ping both ways and everything seems working perfectly fine..

Exactly, I am also victim of same scenario now.. I have checked mapping too there are no conflicts.

Hamza_H
Moderator
Moderator
   VIP   

Do you use NBAC?

check if it is enabled 

Hamza_H
Moderator
Moderator
   VIP   

also check vnetd logs on the client with high verbose, I think you will notice an error such as failed 0x9 or something like it.. 

for FW, even if ports are opened, try to disable it and test it again

14:56:49.675 [32203.32203] <4> ProcessContext: ProcessName:[vnetd], FipsMode:[DISABLED], Username:[root], IsServiceAdmin:[0], UserID:[0], GroupID:[0]
14:56:49.675 [32203.32203] <2> setup_debug_log: switched debug log file for vnetd
14:56:49.676 [32203.32203] <2> logparams: /usr/openv/netbackup/bin/vnetd -terminate
14:56:49.688 [32203.32203] <4> main: VERBOSE=0, VNETD VERBOSE=0 Timestamp=1672133035
14:56:50.696 [32203.32203] <2> vnet_pop_byte: vnet.c.211: 140371869648704: Function failed: 11 0x0000000b
14:56:50.696 [32203.32203] <2> vnet_pop_string: vnet.c.306: 140371869648704: Function failed: 11 0x0000000b
14:56:50.696 [32203.32203] <2> vnet_pop_signed: vnet.c.349: 140371869648704: Function failed: 11 0x0000000b
14:56:50.696 [32203.32203] <16> daemon_terminate_local: vnet_pop_signed failed: 11
14:56:50.696 [32203.32203] <8> try_terminate: [vnetd.c:3035] daemon_terminate_local() failed 23 0x17
14:56:50.696 [32203.32203] <16> main: EXIT status = 23
14:56:54.118 [32298.32298] <4> ProcessContext: ProcessName:[vnetd], FipsMode:[DISABLED], Username:[root], IsServiceAdmin:[0], UserID:[0], GroupID:[0]
14:56:54.118 [32298.32298] <2> setup_debug_log: switched debug log file for vnetd
14:56:54.118 [32298.32298] <2> logparams: /usr/openv/netbackup/bin/vnetd -standalone
14:56:54.130 [32298.32298] <4> main: VERBOSE=0, VNETD VERBOSE=0 Timestamp=1672133035
14:56:54.131 [32298.32298] <2> cs_cache_create: opened sqlite3 database
14:56:54.131 [32298.32298] <2> cs_cache_create: Created table (hostid_to_alias)
14:56:54.131 [32298.32298] <2> cs_cache_create: Created table (hostname_to_hostid)
14:56:54.131 [32298.32298] <2> cs_cache_create: Created table (generic_key_value)
14:56:54.131 [32298.32298] <2> cs_cache_create: closed sqlite3 database
14:56:54.131 [32298.32298] <4> cs_cache_create: /usr/openv/var/vnetd/allowedlist_cache.db permission changed
14:56:54.143 [32300.32300] <2> setup_debug_log: switching debug log file for vnetd
14:56:54.143 [32300.32300] <4> ProcessContext: ProcessName:[vnetd], FipsMode:[DISABLED], Username:[root], IsServiceAdmin:[0], UserID:[0], GroupID:[0]
14:56:54.143 [32300.32300] <2> setup_debug_log: switched debug log file for vnetd
14:56:59.144 [32300.32300] <16> daemon_lock: ManageReadableLockFile(/usr/openv/var/vnetd/vnetd.lock) failed: -3
14:56:59.144 [32300.32300] <8> main: [vnetd.c:579] daemon_lock failed, retval=145, VNETD_LOCKPATH=/usr/openv/var/vnetd/vnetd.lock
14:56:59.144 [32300.32300] <16> main: EXIT status = 145
15:04:36.932 [12801.12801] <4> ProcessContext: ProcessName:[vnetd], FipsMode:[DISABLED], Username:[root], IsServiceAdmin:[0], UserID:[0], GroupID:[0]
15:04:36.932 [12801.12801] <2> setup_debug_log: switched debug log file for vnetd
15:04:36.933 [12801.12801] <2> logparams: /usr/openv/netbackup/bin/vnetd -terminate -proxy inbound_proxy
15:04:36.945 [12801.12801] <4> main: VERBOSE=0, VNETD VERBOSE=0 Timestamp=1672133035
15:04:36.970 [12806.12806] <4> ProcessContext: ProcessName:[vnetd], FipsMode:[DISABLED], Username:[root], IsServiceAdmin:[0], UserID:[0], GroupID:[0]
15:04:36.970 [12806.12806] <2> setup_debug_log: switched debug log file for vnetd
15:04:36.970 [12806.12806] <2> logparams: /usr/openv/netbackup/bin/vnetd -terminate -proxy outbound_proxy
15:04:36.982 [12806.12806] <4> main: VERBOSE=0, VNETD VERBOSE=0 Timestamp=1672133035
15:04:36.999 [12801.12801] <2> log_set_callback: switched to a new logging callback
15:04:36.999 [12801.12801] <4> log_set_callback: ProcessName:[vnetd], FipsMode:[DISABLED], Username:[root], IsServiceAdmin:[0], UserID:[0], GroupID:[0]
15:04:37.025 [12806.12806] <2> log_set_callback: switched to a new logging callback
15:04:37.025 [12806.12806] <4> log_set_callback: ProcessName:[vnetd], FipsMode:[DISABLED], Username:[root], IsServiceAdmin:[0], UserID:[0], GroupID:[0]