Forum Discussion

suparman7's avatar
suparman7
Level 3
3 years ago

Netbackup client connectivity issue

bptestbpcd from master server to client is failing with below error, although able to connect all the backup ports and name resolution is fine and also able to issue certificate successfully. Still backups are failing with error 58 can't connect client

vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.94: pbxRegisterExEx failed with error 110:Connection timed outdaemon_select_and_accept_ex: vnet_registerPBXServer() failed: 47

BPTESTBPCD

/usr/openv/netbackup/bin/admincmd/bptestbpcd -verbose -debug -client XXXXXXXX.XXXX.XXXX
14:32:29.524 [30023] <2> bptestbpcd: VERBOSE = 0
14:32:29.525 [30023] <2> ConnectionCache::connectAndCache: Acquiring new connection for host XXXXXXXX.XX.XXXX, query type 223
14:32:29.528 [30023] <2> logconnections: BPDBM CONNECT FROM XXX.XXX.XXX.XXX TO XXX.XXX.XXX.XXX fd = 3
14:32:29.529 [30023] <2> db_CLIENTsend: reset client protocol version from 0 to 9
14:32:29.533 [30023] <2> db_getCLIENT: db_CLIENTreceive: no entity was found 227
14:32:29.533 [30023] <2> closeConnection: Caching connection for query type 223 for reuse
14:32:29.542 [30023] <2> vnet_pbxConnect_ex: pbxConnectExEx() failed: 104
14:32:29.542 [30023] <2> vnet_pbxConnect_ex: ../../libvlibs/vnet_pbx.c.674: pbxSetAddrEx/pbxConnectExEx return error 104:Connection reset by peer
14:32:29.542 [30023] <8> do_pbx_service: [vnet_connect.c:4012] vnet_pbxConnect() failed, status=18, errno=2, use_vnetd=0, cr->vcr_service=bpcd
14:32:29.542 [30023] <8> async_connect: [vnet_connect.c:3543] do_service failed 18 0x12
14:32:29.544 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:30.547 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:32.549 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:36.551 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:44.554 [30023] <8> async_connect: [vnet_connect.c:3566] getsockopt SO_ERROR returned 111 0x6f
14:32:44.554 [30023] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED status: FAILED, (24) BAD_VERSION; system: (2) No such file or directory; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA pbx status: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA vnetd
14:32:44.554 [30023] <16> connect_to_service: JSON data = {"allow_large_status": {"timestamp": 1672218149, "who": "vnet_tss_init", "line_number": 32, "comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timestamp": 1672218149, "who": "connect_to_service", "line_number": 2199, "comment": "connect parameters", "data": {"who": "vnet_connect_to_bpcd", "host": "XXXXXXXX.XXXX.XXXX", "service": "bpcd", "override_required_interface": null, "extra_tries_on_connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_options": {"server": null, "callback_kind": {"number": 1, "symbol": "NBCONF_CALLBACK_KIND_VNETD", "description": "Vnetd"}, "daemon_port_type": {"number": 0, "symbol": "NBCONF_DAEMON_PORT_TYPE_AUTOMATIC", "description": "Automatic"}, "reserved_port_kind": {"number": 0, "symbol": "NBCONF_RESERVED_PORT_KIND_LEGACY", "description": "Legacy"}}}}, "status": {"timestamp": 1672218164, "who": "connect_to_service", "line_number": 2465, "comment": "vnet status", "data": 18}, "connect_recs": {"timestamp": 1672218164, "who": "vnet_tss_get", "line_number": 97, "comment": "connect rec status messages", "data": "connect failed STATUS (18) CONNECT_FAILED\n\tstatus: FAILED, (24) BAD_VERSION; system: (2) No such file or directory; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA pbx\n\tstatus: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA vnetd"}}
14:32:44.554 [30023] <8> vnet_connect_to_bpcd: [vnet_connect.c:623] connect_to_service() failed 18 0x12
14:32:44.554 [30023] <16> local_bpcr_connect: vnet_connect_to_bpcd(XXXXXXXX.XXXX.XXXX) failed: 18
14:32:44.554 [30023] <2> local_bpcr_connect: Can't connect to client XXXXXXXX.XXXX.XXXX
14:32:44.554 [30023] <2> ConnectToBPCD: bpcd_connect_and_verify(XXXXXXXX.XXXX.XXXX, XXXXXXXX.XXXX.XXXX) failed: 25
14:32:44.554 [30023] <16> bptestbpcd main: JSON proxy message = {"allow_large_status": {"timestamp": 1672218149, "who": "vnet_tss_init", "line_number": 32, "comment": "allow vnet status > 255", "data": true}, "direct_connect": {"timestamp": 1672218149, "who": "connect_to_service", "line_number": 2199, "comment": "connect parameters", "data": {"who": "vnet_connect_to_bpcd", "host": "XXXXXXXX.XXXX.XXXX", "service": "bpcd", "override_required_interface": null, "extra_tries_on_connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_options": {"server": null, "callback_kind": {"number": 1, "symbol": "NBCONF_CALLBACK_KIND_VNETD", "description": "Vnetd"}, "daemon_port_type": {"number": 0, "symbol": "NBCONF_DAEMON_PORT_TYPE_AUTOMATIC", "description": "Automatic"}, "reserved_port_kind": {"number": 0, "symbol": "NBCONF_RESERVED_PORT_KIND_LEGACY", "description": "Legacy"}}}}, "status": {"timestamp": 1672218164, "who": "connect_to_service", "line_number": 2465, "comment": "vnet status", "data": 18}, "connect_recs": {"timestamp": 1672218164, "who": "vnet_tss_get", "line_number": 97, "comment": "connect rec status messages", "data": "connect failed STATUS (18) CONNECT_FAILED\n\tstatus: FAILED, (24) BAD_VERSION; system: (2) No such file or directory; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA pbx\n\tstatus: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO XXXXXXXX.XXXX.XXXX XXX.XXX.XXX.XXX bpcd VIA vnetd"}}
<16>bptestbpcd main: Function ConnectToBPCD(XXXXXXXX.XXXX.XXXX) failed: 25
14:32:44.554 [30023] <16> bptestbpcd main: Function ConnectToBPCD(XXXXXXXX.XXXX.XXXX) failed: 25
<16>bptestbpcd main: cannot connect on socket
14:32:44.559 [30023] <16> bptestbpcd main: cannot connect on socket
<2>bptestbpcd: cannot connect on socket
14:32:44.559 [30023] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
14:32:44.559 [30023] <2> bptestbpcd: EXIT status = 25

 

 

7 Replies

  • Hi suparman7 

    What version of Netbackup is listing at the client ?

    Is there firewall between NBU and the client ?

    What OS is the client ?

    Netbackup will per default connect on PBX port 1556, is that port isn't available, Netbackup will try 13724 (vnetd) and 13782 in that order.  

    I see both connection to PBX (1556) and 13724 (Vnetd) failing.

    Try from the client side to use wireshark or tcpdump to see if connection requests is really reaching the client, it doesn't look like it.

     

    • suparman7's avatar
      suparman7
      Level 3

      I have telnet all NBU ports and are connecting fine both ways from client to master and vice versa... I have tried to reinstall and was able to issue a certificate too but when i tried to connect to client or bptestbpcd it fails...

      All Master, Media and client version 9.1.0.1

      All required ports are connecting and i have tried to run a

      traceroute, nslookup and ping both ways and everything seems working perfectly fine..

  • Hi,
    Was this servers ip address used for another server before? If yes, check the mapping if the bad server’s name is mapped to your server.
    We had a similar issue for this.. there was no FW, telnet was OK in both directions.. nbu commands were working fine (bpclntcmd, bptestnetconn..) but bptestbpcd was failing and backups as well..
    There were no actions and it worked by itself.. (one of netbackups mysteries..)
    What netbackup version you are using (master, media and client)?
    • suparman7's avatar
      suparman7
      Level 3

      Exactly, I am also victim of same scenario now.. I have checked mapping too there are no conflicts.

      • Hamza_H's avatar
        Hamza_H
        Moderator

        Do you use NBAC?

        check if it is enabled