Unable to communicate with client server with status 25

[prakash_sharma]

Hi All,

Master server:

OS: Solaris 10

Netbackup: NetBackup 7.6.1.2

Client server :

OS: AIX

Netbackup: NetBackup 7.5

I am unable to communicate with a client server. When tried doing bptestbpcd for the server, I am getting the mentioned error:

bash-3.2$ bptestbpcd -client IHYDIBMP780-06-LP5 -verbose -debug
23:54:56.011 [1188] <2> bptestbpcd: VERBOSE = 0
23:55:06.021 [1188] <2> vnet_pbxConnect: pbxConnectEx Succeeded
23:55:06.023 [1188] <2> logconnections: BPCD CONNECT FROM 10.16.13.163.53022 TO 172.16.14.132.1556 fd = 5
23:55:16.026 [1188] <2> vnet_pop_byte: vnet.c.181: 0: Function failed: 11 0x0000000b
23:55:16.026 [1188] <2> vnet_pop_string: vnet.c.276: 0: Function failed: 11 0x0000000b
23:55:16.027 [1188] <2> vnet_pop_signed: vnet.c.319: 0: Function failed: 11 0x0000000b
23:55:16.027 [1188] <2> vnet_version_connect: vnet_vnetd.c.1663: 0: Function failed: 11 0x0000000b
23:55:16.027 [1188] <8> do_vnetd_service: [vnet_connect.c:1951] vnet_version_connect failed 11 0xb
23:55:16.027 [1188] <8> async_connect: [vnet_connect.c:1677] do_service failed 11 0xb
23:55:26.027 [1188] <8> async_connect: [vnet_connect.c:1235] ran out of time before connect 20 0x14
23:55:26.027 [1188] <8> async_connect: [vnet_connect.c:1235] ran out of time before connect 20 0x14
23:55:26.028 [1188] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED
        status: FAILED, (44) CONNECT_TIMEOUT; system: (150) Operation now in progress; FROM 0.0.0.0 TO 172.16.14.132 172.16.14.132 vnetd VIA pbx
        status: FAILED, (44) CONNECT_TIMEOUT; system: (150) Operation now in progress; FROM 0.0.0.0 TO 172.16.14.132 172.16.14.132 vnetd VIA vnetd
23:55:26.028 [1188] <8> vnet_connect_to_vnetd_ipi: [vnet_connect.c:275] connect_to_service() failed 18 0x12
23:55:26.028 [1188] <16> bpcr_vnetd_connect_forward_socket_begin: vnet_connect_to_vnetd_ipi(172.16.14.132) failed: 18
23:55:26.028 [1188] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_socket_begin failed: 25
23:55:26.028 [1188] <2> ConnectToBPCD: bpcd_connect_and_verify(IHYDIBMP780-06-LP5, IHYDIBMP780-06-LP5) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(IHYDIBMP780-06-LP5) failed: 25
23:55:26.029 [1188] <16> bptestbpcd main: Function ConnectToBPCD(IHYDIBMP780-06-LP5) failed: 25
<16>bptestbpcd main: cannot connect on socket
23:55:26.069 [1188] <16> bptestbpcd main: cannot connect on socket
<2>bptestbpcd: cannot connect on socket
23:55:26.069 [1188] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
23:55:26.069 [1188] <2> bptestbpcd: EXIT status = 25
cannot connect on socket

When tried bpclntcmd from client as well as from the master server, I am getting the desired output:

From Client:

bash-4.2# ./bpclntcmd -pn
expecting response from server hydt5440-nbms
IHYDIBMP780-06-LP5 IHYDIBMP780-06-LP5 172.16.14.132 63262
 

From Master server:

bash-3.2$ bpclntcmd -hn IHYDIBMP780-06-LP5
host IHYDIBMP780-06-LP5: IHYDIBMP780-06-LP5 at 172.16.14.132
host IHYDIBMP780-06-LP5: IHYDIBMP780-06-LP5 at 172.16.14.132
aliases:     IHYDIBMP780-06-LP5     172.16.14.132
bash-3.2$ bpclntcmd -ip 172.16.14.132
host 172.16.14.132: IHYDIBMP780-06-LP5 at 172.16.14.132
host 172.16.14.132: IHYDIBMP780-06-LP5 at 172.16.14.132
aliases:     IHYDIBMP780-06-LP5     172.16.14.132
bash-3.2$ 

The BPCD logs on the client server are as follows:

23:55:06.022 [11599884] <2> vnet_pbxAcceptSocket: Accepted sock[10] from 10.16.13.163:53022
23:55:06.023 [11599884] <2> bpcd main: accept sock = 10
23:55:06.025 [5701892] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:808] the socket is  0 0x0
23:55:06.026 [5701892] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:808] the socket is  1 0x1
23:55:06.026 [5701892] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:808] the socket is  2 0x2
23:55:06.026 [5701892] <2> ReadKeyfile: keyfile /usr/openv/netbackup/keyfile does not exist

23:55:06.026 [5701892] <2> process_requests: offset to GMT -19800
23:55:06.026 [5701892] <2> logconnections: BPCD ACCEPT FROM 10.16.13.163.53022 TO 172.16.14.132.1556 fd = 0
23:55:06.026 [5701892] <2> process_requests: setup_sockopts complete
23:55:06.027 [5701892] <2> vnet_pcache_init_table: [vnet_private.c:235] starting cache size 200 0xc8
23:55:06.027 [5701892] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:1895] found via getnameinfo OUR_HOST=HYDT5440-NBMS IPSTR=10.16.13.163
23:55:06.032 [5701892] <2> print_arp_info: 0821-237 arp: Entry HYDT5440-NBMS (10.16.13.163) was not found in local arp table.
23:55:06.033 [5701892] <2> bpcd peer_hostname: Connection from host HYDT5440-NBMS (10.16.13.163) port 53022
23:55:06.033 [5701892] <2> bpcd valid_server: comparing hydt5440-nbms and HYDT5440-NBMS
23:55:06.033 [5701892] <4> bpcd valid_server: hostname comparison succeeded
23:55:06.033 [5701892] <2> process_requests: output socket port number = 1
23:55:26.029 [5701892] <2> get_long_base: (2) premature end of file (byte 1)
23:55:26.029 [5701892] <2> get_vnetd_forward_socket: get_string ipc_string failed: 5
23:55:26.029 [5701892] <16> process_requests: get_vnetd_forward_socket failed: 23
23:55:26.031 [11599884] <2> daemon_check_for_zombies: waited for 1 child processes including: 5701892

Kindly help, to resolve the issue.
 

[Will_Restore]

https://www.veritas.com/support/en_US/article.000009351

Troubleshooting procedures for client Status 25 errors.

( use the link above -- seems you have tried a few already but go through the doc step by step )

[prakash_sharma]

Hi Will, I tried the Troubleshooting procedures:

Step 1: Yes , It was working previously
Step 2: bptestbpcd not happening, the output is mentioned.
Step 3: bpclntcmd happening with Hostname and IP. We are able to ping the Client's iP fom Master server.
Step 4: bpclntcmd happening from both client.
Step 5: bpcd and vnetd are in listen mode.
Step 6: bpcd logs cretaed with verbose 5 and its updated above.
Step 7: Telnet happening for 13782, 13724 and 1556 from both client as well as the master server.

[Marianne]

Errors in bpcd log looks similar to this issue with Windows client: http://www.veritas.com/docs/000005913 I suggest you ask the AIX admin to investigate memory usage on this system. A reboot will probably fix the issue.

[prakash_sharma]

Hi All,

Have asked user to take a reboot of the server.