bptestbpcd command failed

[tarmizi]

Hi, I got a problem with one of my AIX client. After install the client, I ran " bptestbpcd -client " on the master server it return with " bptestbpcd main: Function ConnectToBPCD(client name) failed: 24 socket write failed". What should I do next? Since I have ask our firewall team to open up the 1556,13724 & 13782 ports but still no go. Thank you

[Marianne]

Have you confirmed forward and reverse name lookup between client and server?
Does OS comms like ping and traceroute work? Telnet to ports from both ends?

Please create bpcd log folder on the client under /usr/openv/netbackup/logs.

Run bptestbpcd command again with verbose and debug options:

bptestbpcd -client <client-name> -verbose -debug

Please post output of command as well as bpcd log on the client.

If nothing is written to bpcd log, there is probably a port connection problem.

[tarmizi]

Hi Marianne,

I can ping using IP and hostname to the client also telnet to the three ports given earlier post.

And this is what I got after run the command given by you:

C:\Documents and Settings\rdpadmin>bptestbpcd -client <client_name> -verbose -debug
16:26:06.703 [5296.17388] <2> bptestbpcd: VERBOSE = 0
16:26:06.703 [5296.17388] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.656: pbxSetAddrEx/pbxConnectEx return error 10054:An existing connection was forcibly closed by the remote host.
16:26:06.703 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1658: 0: vnet_pbxConnect() failed: 18 0x00000012
16:26:06.703 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1659: 0: save_errno: 10054 0x00002746
16:26:06.703 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1660: 0: use_vnetd: 0 0x00000000
16:26:06.718 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1661: 0: cr->vcr_service: bpcd
16:26:06.718 [5296.17388] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 18 0x00000012
16:26:06.734 [5296.17388] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2196: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
16:26:06.734 [5296.17388] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2210: 0: service: bpcd
16:26:06.859 [5296.17388] <2> logconnections: BPCD CONNECT FROM 10.10.100.176.55540 TO 10.10.70.15.13724 fd = 1616
16:26:06.875 [5296.17388] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.656: pbxSetAddrEx/pbxConnectEx return error 10054:An existing connection was forcibly closed by the remote host.
16:26:06.875 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1658: 0: vnet_pbxConnect() failed: 18 0x00000012
16:26:06.875 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1659: 0: save_errno: 10054 0x00002746
16:26:06.875 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1660: 0: use_vnetd: 1 0x00000001
16:26:06.875 [5296.17388] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1661: 0: cr->vcr_service: vnetd
16:26:06.875 [5296.17388] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1293: 0: do_service failed: 18 0x00000012
16:26:06.890 [5296.17388] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1615: 0: connect: VNETD CONNECT FROM 10.10.100.176.55542 TO 10.10.70.15.13724 fd = 1632
16:26:06.890 [5296.17388] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.540: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
16:26:07.046 [5296.17388] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.557: 0: ipc_string: /tmp/vnet-26292374567944596011000000000-X-axqa
16:26:07.250 [5296.17388] <2> put_string: cannot write data to network:  An established connection was aborted by the software in your host machine.
16:26:07.250 [5296.17388] <2> bpcr_put_vnetd_forward_socket: put_string /tmp/vnet-26292374567944596011000000000-X-axqa failed: 10053
16:26:07.250 [5296.17388] <2> local_bpcr_connect: bpcr_put_vnetd_forward_socket failed: 24
16:26:07.250 [5296.17388] <2> ConnectToBPCD: bpcd_connect_and_verify(SACBA315, SACBA315) failed: 24
<16>bptestbpcd main: Function ConnectToBPCD(SACBA315) failed: 24
16:26:07.250 [5296.17388] <16> bptestbpcd main: Function ConnectToBPCD(SACBA315) failed: 24
<2>bptestbpcd: socket write failed
16:26:07.250 [5296.17388] <2> bptestbpcd: socket write failed
<2>bptestbpcd: EXIT status = 24
16:26:07.250 [5296.17388] <2> bptestbpcd: EXIT status = 24
socket write failed

About the bpcd log, it does not contain any log file.

Thank you

[Possible]

Netbackup is not able use the socket for the communication.

16:26:07.250 [5296.17388] <2> put_string: cannot write data to network:  An established connection was aborted by the software in your host machine.

Error message is saying "Connection was aborted by SOFTWARE in ur host machine" ...Some other thing disturbing the socket...probably Firewall...have a check for it.

[tarmizi]

Hi Gary,

I have ask the firewall team to allow any comms from the master to my client. I event asking the server admin if there are any firewall in the OS level. The answer I received is there is no internal/OS firewall but they have done OS hardening recently. It could be the cause of my problem. What should I do?ask them to open the ports? 

Thank you

[Marianne]

We see connection attempt to port 13724 (vnetd), no attempt to connect to 1556 (pbx) or 13782 (bpcd).

This made me wonder which NBU version we are dealing with? On client and server?

Please check process table (ps -ef) on the client to see if vnetd and bpcd are running (depending on the NBU version). If not, check 'netstat -a' to see if vnetd and bpcd are LISTENING. 

Please show us output of telnet attempts? 
It seems weird that telnet is working and bptestbpcd not.

[Possible]

What should I do?ask them to open the ports? 

Are those still not opened...if not get them opened.

as marianne suggested use netstat -an to get rid of open ports.

Try opening in standalone mode and test the backup.

/usr/openv/netbackup/bin/vnetd -standalone

/usr/openv/netbackup/bin bpcd -standalone

[tarmizi]

Hi Marianne and Gary, The Master version is 7.0.1 and client is 7.0. this is what I get after checking the process: [Client]:/>ps -ef | grep vnetd root 5439902 1 0 Jul 18 - 0:00 /usr/openv/netbackup/bin/vnetd -standalone [Client]:/>ps -ef | grep pbx root 3670090 1 0 Jul 18 - 0:00 /opt/VRTSpbx/bin/pbx_exchange [Client]:/>ps -ef | grep bpcd root 5308592 1 0 Jul 18 - 0:00 /usr/openv/netbackup/bin/bpcd -standalone And this what I get after checking the open port: tcp4 0 0 *.vnetd *.* LISTEN tcp4 0 0 *.bpcd *.* LISTEN Thank you

[Marianne]

You forgot about this request?

Please show us output of telnet attempts? 
It seems weird that telnet is working and bptestbpcd not.

Have you spoken to system administrators about the OS hardening?

They have obviously done 'something' that rejects port connections. 
Right now we only see connection attempt to port 13724 (vnetd).
 

[Possible]

Can u go for bpclntcmd -reverse_name_lookup allowed on client

[Marianne]

Thank you for your quick response. 

LOL! It only took you 3 months to reply!

Seems the 1st line in my 1st reply would've solved the problem:

Have you confirmed forward and reverse name lookup between client and server?

[tarmizi · Accepted solution]

Thank you for your quick response. However I managed to find the cause of this issue. Its seem that I mistakenly put the wrong ip of the master server in the AIX client hosts file. Everything is okay now after I put the correct IP.
 

Thank you again..

[tarmizi]

Hi Marianne,

Sorry for the late reply as I just started working after 2.5 months being hospitalize. hehehe