Issue : (25) cannot connect on socket

bash-3.2$ sudo /usr/openv/netbackup/bin/admincmd/bptestbpcd -client saaclalt1-nbu -debug
11:17:30.811 [19924] <2> bptestbpcd: VERBOSE = 0
11:17:40.849 [19924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 62:Timer expired
11:17:40.849 [19924] <8> do_pbx_service: [vnet_connect.c:2034] vnet_pbxConnect() failed 18 0x12
11:17:40.850 [19924] <8> do_pbx_service: [vnet_connect.c:2035] save_errno 62 0x3e
11:17:40.850 [19924] <8> do_pbx_service: [vnet_connect.c:2036] use_vnetd 0 0x0
11:17:40.850 [19924] <8> do_pbx_service: [vnet_connect.c:2037] cr->vcr_service bpcd
11:17:40.850 [19924] <8> async_connect: [vnet_connect.c:1630] do_service failed 18 0x12
11:18:03.212 [19924] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3485] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
11:18:27.212 [19924] <8> do_vnetd_service: [vnet_connect.c:1955] remote host supports PBX, but PBX is not running 0 0x0
11:18:27.212 [19924] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:1996] VN_REQUEST_SERVICE_SOCKET 6 0x6
11:18:27.212 [19924] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2010] service bpcd
11:19:15.215 [19924] <2> logconnections: BPCD CONNECT FROM 10.30.90.1.60982 TO 10.30.18.189.13724 fd = 4
11:19:25.232 [19924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 62:Timer expired
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2034] vnet_pbxConnect() failed 18 0x12
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2035] save_errno 62 0x3e
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2036] use_vnetd 1 0x1
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2037] cr->vcr_service vnetd
11:19:25.232 [19924] <8> async_connect: [vnet_connect.c:1630] do_service failed 18 0x12
11:19:47.714 [19924] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3485] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
11:20:11.715 [19924] <8> do_vnetd_service: [vnet_connect.c:1955] remote host supports PBX, but PBX is not running 0 0x0
11:20:11.716 [19924] <8> do_vnetd_service: [vnet_connect.c:1991] connect VNETD CONNECT FROM 10.30.90.1.52382 TO 10.30.18.189.13724 fd = 5
11:20:11.719 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
11:20:59.717 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /tmp/vnet-94376441275599789505000000001--Vam7b
1 1 1
10.30.90.1:60982 -> 10.30.18.189:13724
10.30.90.1:52382 -> 10.30.18.189:13724
11:22:13.734 [19924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 62:Timer expired
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2034] vnet_pbxConnect() failed 18 0x12
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2035] save_errno 62 0x3e
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2036] use_vnetd 1 0x1
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2037] cr->vcr_service vnetd
11:22:13.734 [19924] <8> async_connect: [vnet_connect.c:1630] do_service failed 18 0x12
11:22:36.220 [19924] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3485] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
11:23:00.221 [19924] <8> do_vnetd_service: [vnet_connect.c:1955] remote host supports PBX, but PBX is not running 0 0x0
11:23:00.222 [19924] <8> do_vnetd_service: [vnet_connect.c:1991] connect VNETD CONNECT FROM 10.30.90.1.58242 TO 10.30.18.189.13724 fd = 6
11:23:00.222 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
11:23:48.223 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /tmp/vnet-82900441275768292915000000001-5Dapqb
10.30.90.1:58242 -> 10.30.18.189:13724
<2>bptestbpcd: EXIT status = 0
11:25:56.228 [19924] <2> bptestbpcd: EXIT status = 0
bash-3.2$

1 Solution

Accepted Solutions
Accepted Solution!

telnet is not successfull

telnet is not successfull from both sides.

No need to perform any further troublehooting  from NBU side.
NBU is not going to work if network connection at TCP-level is unsuccessful.

In which direction is telnet unsuccessful?
On which port?

Have you escalated to OS and network admins?

View solution in original post

8 Replies

 remote host supports PBX,

 remote host supports PBX, but PBX is not running 

Which OS on Client? 
Which NBU version on client? 

Check output of 'ps -ef' on client and look for these processes:

/usr/openv/netbackup/bin/bpcd -standalone
/usr/openv/netbackup/bin/vnetd -standalone

/opt/VRTSpbx/bin/pbx_exchange

Before you test again with bptestbpcd, create bpcd log folder on the client.
See if anything is logged in bpcd after failed test.

If not, check for firewall - including OS firewall such as OS 'hardening' and iptables on Linux clients.

Hi Marianne, Thanks for the

Hi Marianne,

Thanks for the reply. And sorry for late response. Backup is still failing.

Below are the list of output which you asked for.

bash-4.2$ uname -a
AIX saaclalt1 1 6 000076FBD400

bash-4.2$ cat /usr/openv/netbackup/bin/version
NetBackup-AIX53 7.5.0.6
 

bash-4.2$ ps -ef | grep bpcd
    root 13762766        1   0   Sep 03      -  0:02 /usr/openv/netbackup/bin/bpcd -standalone
nbuadmin 35127478 33095794   0 09:41:13  pts/0  0:00 grep bpcd
 

bash-4.2$ ps -ef | grep vnetd
    root 28377178        1   0   Sep 03      -  0:01 /usr/openv/netbackup/bin/vnetd -standalone
nbuadmin 36110460 33095794   0 09:41:30  pts/0  0:00 grep vnetd
 

bash-4.2$ ps -ef | grep pbx
    root  5112002        1   0   Sep 03      -  0:00 /opt/VRTSpbx/bin/pbx_exchange
nbuadmin 27001084 33095794   0 09:42:16  pts/0  0:00 grep pbx
 

bash-4.2$ sudo /usr/openv/netbackup/bin/bpps -a
NB Processes
------------
    root 13762766        1   0   Sep 03      -  0:02 /usr/openv/netbackup/bin/bpcd -standalone
    root 28377178        1   0   Sep 03      -  0:01 /usr/openv/netbackup/bin/vnetd -standalone
bash-4.2$
 

Last failed weekly full backup is on 09/09/2015

Please find attached BPCD log file output.

Please let me know if any other information required.

 

 

 

Find out from OS admins if

Find out from OS admins if any security-related OS-hardening was done.

It seems that client is receiving incoming connections, but is unable to connect back on 1556, 13724 or 13782.

The same error can be seen for connect-back attempts for all 3 ports:

17:28:33.370 [24379512] <16> get_vnetd_socket: vnet_end_connect_back failed: 10
17:28:33.370 [24379512] <16> process_requests: get_vnetd_socket failed: 25
Highlighted

Hmm. The bpcd log is full of

Hmm. The bpcd log is full of ERRNO=2 messages. That is a "File not found" thing.

 

Just on a hunch. Have you tried to do "bpclntcmd -clear_host_cache"?

Not tried...still struggling

Not tried...still struggling with the issue.

Host cache refreshes

Host cache refreshes automatically once per hour. This means that if the backup is still failing with the same error, the problem is not with host cache.

Have you spoken to OS admins as per my previous post?

Can you telnet from the client to the master and/or media server on port 1556 and 13724?

What does this command on client return? 

bpclntcmd -pn

Hi Marianne, telnet is not

Hi Marianne,

telnet is not successfull from both sides.

Media Server Logs :

bpcd logs : PFA

bpbrm logs : PFA

Client logs :

bpcd logs : PFA

bpbkar logs : No logs generated.

Job detail of failed job as mentioned below :

09/18/2015 12:52:53 - Info nbjm (pid=6637) starting backup job (jobid=5819622) for client saaclalt1-nbu, policy APP_FS_SWIFT_TEST, schedule Weekly_Full

09/18/2015 12:52:53 - Info nbjm (pid=6637) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=5819622, request id:{CB7C708E-5DFB-11E5-8304-00144F43C41C})

09/18/2015 12:52:53 - requesting resource OM-9940B-TSUG-ALLIB-1

09/18/2015 12:52:53 - requesting resource nbumaster.NBU_CLIENT.MAXJOBS.saaclalt1-nbu

09/18/2015 12:52:53 - requesting resource nbumaster.NBU_POLICY.MAXJOBS.APP_FS_SWIFT_TEST

09/18/2015 12:52:54 - granted resource  nbumaster.NBU_CLIENT.MAXJOBS.saaclalt1-nbu

09/18/2015 12:52:54 - granted resource  nbumaster.NBU_POLICY.MAXJOBS.APP_FS_SWIFT_TEST

09/18/2015 12:52:54 - granted resource  D00431

09/18/2015 12:52:54 - granted resource  AL_00-01-01-15_803

09/18/2015 12:52:54 - granted resource  OMG-NBUMED-01-TSU-ALLIB

09/18/2015 12:52:54 - estimated 0 kbytes needed

09/18/2015 12:52:54 - Info nbjm (pid=6637) started backup (backupid=saaclalt1-nbu_1442577174) job for client saaclalt1-nbu, policy APP_FS_SWIFT_TEST, schedule Weekly_Full on storage unit OMG-NBUMED-01-TSU-ALLIB

09/18/2015 13:00:14 - Info bpbrm (pid=1756) saaclalt1-nbu is the host to backup data from

09/18/2015 13:00:14 - Info bpbrm (pid=1756) telling media manager to start backup on client

09/18/2015 13:00:15 - Info bptm (pid=7580) using 262144 data buffer size

09/18/2015 13:00:15 - Info bptm (pid=7580) using 128 data buffers

09/18/2015 13:17:58 - Info bpbrm (pid=6848) sending bpsched msg: CONNECTING TO CLIENT FOR saaclalt1-nbu_1442577174

09/18/2015 13:17:59 - connecting

09/18/2015 13:24:30 - Error bpbrm (pid=6848) cannot create data socket, The operation completed successfully.  (0)

09/18/2015 13:24:30 - Info bpbrm (pid=1756) child done, status 21

09/18/2015 13:24:30 - Info bpbrm (pid=1756) sending message to media manager: STOP BACKUP saaclalt1-nbu_1442577174

09/18/2015 13:24:31 - Info bpbrm (pid=1756) media manager for backup id saaclalt1-nbu_1442577174 exited with status 150: termination requested by administrator

09/18/2015 13:24:31 - end writing

socket open failed  (21)

Accepted Solution!

telnet is not successfull

telnet is not successfull from both sides.

No need to perform any further troublehooting  from NBU side.
NBU is not going to work if network connection at TCP-level is unsuccessful.

In which direction is telnet unsuccessful?
On which port?

Have you escalated to OS and network admins?

View solution in original post