Forum Discussion

amit4465's avatar
amit4465
Level 3
9 years ago

Issue : (25) cannot connect on socket

bash-3.2$ sudo /usr/openv/netbackup/bin/admincmd/bptestbpcd -client saaclalt1-nbu -debug
11:17:30.811 [19924] <2> bptestbpcd: VERBOSE = 0
11:17:40.849 [19924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 62:Timer expired
11:17:40.849 [19924] <8> do_pbx_service: [vnet_connect.c:2034] vnet_pbxConnect() failed 18 0x12
11:17:40.850 [19924] <8> do_pbx_service: [vnet_connect.c:2035] save_errno 62 0x3e
11:17:40.850 [19924] <8> do_pbx_service: [vnet_connect.c:2036] use_vnetd 0 0x0
11:17:40.850 [19924] <8> do_pbx_service: [vnet_connect.c:2037] cr->vcr_service bpcd
11:17:40.850 [19924] <8> async_connect: [vnet_connect.c:1630] do_service failed 18 0x12
11:18:03.212 [19924] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3485] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
11:18:27.212 [19924] <8> do_vnetd_service: [vnet_connect.c:1955] remote host supports PBX, but PBX is not running 0 0x0
11:18:27.212 [19924] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:1996] VN_REQUEST_SERVICE_SOCKET 6 0x6
11:18:27.212 [19924] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2010] service bpcd
11:19:15.215 [19924] <2> logconnections: BPCD CONNECT FROM 10.30.90.1.60982 TO 10.30.18.189.13724 fd = 4
11:19:25.232 [19924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 62:Timer expired
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2034] vnet_pbxConnect() failed 18 0x12
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2035] save_errno 62 0x3e
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2036] use_vnetd 1 0x1
11:19:25.232 [19924] <8> do_pbx_service: [vnet_connect.c:2037] cr->vcr_service vnetd
11:19:25.232 [19924] <8> async_connect: [vnet_connect.c:1630] do_service failed 18 0x12
11:19:47.714 [19924] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3485] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
11:20:11.715 [19924] <8> do_vnetd_service: [vnet_connect.c:1955] remote host supports PBX, but PBX is not running 0 0x0
11:20:11.716 [19924] <8> do_vnetd_service: [vnet_connect.c:1991] connect VNETD CONNECT FROM 10.30.90.1.52382 TO 10.30.18.189.13724 fd = 5
11:20:11.719 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
11:20:59.717 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /tmp/vnet-94376441275599789505000000001--Vam7b
1 1 1
10.30.90.1:60982 -> 10.30.18.189:13724
10.30.90.1:52382 -> 10.30.18.189:13724
11:22:13.734 [19924] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 62:Timer expired
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2034] vnet_pbxConnect() failed 18 0x12
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2035] save_errno 62 0x3e
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2036] use_vnetd 1 0x1
11:22:13.734 [19924] <8> do_pbx_service: [vnet_connect.c:2037] cr->vcr_service vnetd
11:22:13.734 [19924] <8> async_connect: [vnet_connect.c:1630] do_service failed 18 0x12
11:22:36.220 [19924] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3485] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
11:23:00.221 [19924] <8> do_vnetd_service: [vnet_connect.c:1955] remote host supports PBX, but PBX is not running 0 0x0
11:23:00.222 [19924] <8> do_vnetd_service: [vnet_connect.c:1991] connect VNETD CONNECT FROM 10.30.90.1.58242 TO 10.30.18.189.13724 fd = 6
11:23:00.222 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
11:23:48.223 [19924] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string /tmp/vnet-82900441275768292915000000001-5Dapqb
10.30.90.1:58242 -> 10.30.18.189:13724
<2>bptestbpcd: EXIT status = 0
11:25:56.228 [19924] <2> bptestbpcd: EXIT status = 0
bash-3.2$

AIX
Backup and Recovery
Error messages
NetBackup
  • Marianne's avatar
    Marianne
    9 years ago

    telnet is not successfull from both sides.

    No need to perform any further troublehooting  from NBU side.
    NBU is not going to work if network connection at TCP-level is unsuccessful.

    In which direction is telnet unsuccessful?
    On which port?

    Have you escalated to OS and network admins?

8 Replies

Sort By
  • Marianne's avatar
    Marianne
    Level 6
    9 years ago

     remote host supports PBX, but PBX is not running 

    Which OS on Client? 
    Which NBU version on client? 

    Check output of 'ps -ef' on client and look for these processes:

    /usr/openv/netbackup/bin/bpcd -standalone
    /usr/openv/netbackup/bin/vnetd -standalone

    /opt/VRTSpbx/bin/pbx_exchange

    Before you test again with bptestbpcd, create bpcd log folder on the client.
    See if anything is logged in bpcd after failed test.

    If not, check for firewall - including OS firewall such as OS 'hardening' and iptables on Linux clients.

  • amit4465's avatar
    amit4465
    Level 3
    9 years ago

    Hi Marianne,

    Thanks for the reply. And sorry for late response. Backup is still failing.

    Below are the list of output which you asked for.

    bash-4.2$ uname -a
    AIX saaclalt1 1 6 000076FBD400

    bash-4.2$ cat /usr/openv/netbackup/bin/version
    NetBackup-AIX53 7.5.0.6
     

    bash-4.2$ ps -ef | grep bpcd
        root 13762766        1   0   Sep 03      -  0:02 /usr/openv/netbackup/bin/bpcd -standalone
    nbuadmin 35127478 33095794   0 09:41:13  pts/0  0:00 grep bpcd
     

    bash-4.2$ ps -ef | grep vnetd
        root 28377178        1   0   Sep 03      -  0:01 /usr/openv/netbackup/bin/vnetd -standalone
    nbuadmin 36110460 33095794   0 09:41:30  pts/0  0:00 grep vnetd
     

    bash-4.2$ ps -ef | grep pbx
        root  5112002        1   0   Sep 03      -  0:00 /opt/VRTSpbx/bin/pbx_exchange
    nbuadmin 27001084 33095794   0 09:42:16  pts/0  0:00 grep pbx
     

    bash-4.2$ sudo /usr/openv/netbackup/bin/bpps -a
    NB Processes
    ------------
        root 13762766        1   0   Sep 03      -  0:02 /usr/openv/netbackup/bin/bpcd -standalone
        root 28377178        1   0   Sep 03      -  0:01 /usr/openv/netbackup/bin/vnetd -standalone
    bash-4.2$
     

    Last failed weekly full backup is on 09/09/2015

    Please find attached BPCD log file output.

    Please let me know if any other information required.

     

     

     

  • Marianne's avatar
    Marianne
    Level 6
    9 years ago

    Find out from OS admins if any security-related OS-hardening was done.

    It seems that client is receiving incoming connections, but is unable to connect back on 1556, 13724 or 13782.

    The same error can be seen for connect-back attempts for all 3 ports:

    17:28:33.370 [24379512] <16> get_vnetd_socket: vnet_end_connect_back failed: 10
17:28:33.370 [24379512] <16> process_requests: get_vnetd_socket failed: 25
  • Jaime_Vazquez's avatar
    Jaime_Vazquez
    Level 6
    9 years ago

    Hmm. The bpcd log is full of ERRNO=2 messages. That is a "File not found" thing.

     

    Just on a hunch. Have you tried to do "bpclntcmd -clear_host_cache"?

  • Marianne's avatar
    Marianne
    Level 6
    9 years ago

    Host cache refreshes automatically once per hour. This means that if the backup is still failing with the same error, the problem is not with host cache.

    Have you spoken to OS admins as per my previous post?

    Can you telnet from the client to the master and/or media server on port 1556 and 13724?

    What does this command on client return? 

    bpclntcmd -pn

  • amit4465's avatar
    amit4465
    Level 3
    9 years ago

    Hi Marianne,

    telnet is not successfull from both sides.

    Media Server Logs :

    bpcd logs : PFA

    bpbrm logs : PFA

    Client logs :

    bpcd logs : PFA

    bpbkar logs : No logs generated.

    Job detail of failed job as mentioned below :

    09/18/2015 12:52:53 - Info nbjm (pid=6637) starting backup job (jobid=5819622) for client saaclalt1-nbu, policy APP_FS_SWIFT_TEST, schedule Weekly_Full

    09/18/2015 12:52:53 - Info nbjm (pid=6637) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=5819622, request id:{CB7C708E-5DFB-11E5-8304-00144F43C41C})

    09/18/2015 12:52:53 - requesting resource OM-9940B-TSUG-ALLIB-1

    09/18/2015 12:52:53 - requesting resource nbumaster.NBU_CLIENT.MAXJOBS.saaclalt1-nbu

    09/18/2015 12:52:53 - requesting resource nbumaster.NBU_POLICY.MAXJOBS.APP_FS_SWIFT_TEST

    09/18/2015 12:52:54 - granted resource  nbumaster.NBU_CLIENT.MAXJOBS.saaclalt1-nbu

    09/18/2015 12:52:54 - granted resource  nbumaster.NBU_POLICY.MAXJOBS.APP_FS_SWIFT_TEST

    09/18/2015 12:52:54 - granted resource  D00431

    09/18/2015 12:52:54 - granted resource  AL_00-01-01-15_803

    09/18/2015 12:52:54 - granted resource  OMG-NBUMED-01-TSU-ALLIB

    09/18/2015 12:52:54 - estimated 0 kbytes needed

    09/18/2015 12:52:54 - Info nbjm (pid=6637) started backup (backupid=saaclalt1-nbu_1442577174) job for client saaclalt1-nbu, policy APP_FS_SWIFT_TEST, schedule Weekly_Full on storage unit OMG-NBUMED-01-TSU-ALLIB

    09/18/2015 13:00:14 - Info bpbrm (pid=1756) saaclalt1-nbu is the host to backup data from

    09/18/2015 13:00:14 - Info bpbrm (pid=1756) telling media manager to start backup on client

    09/18/2015 13:00:15 - Info bptm (pid=7580) using 262144 data buffer size

    09/18/2015 13:00:15 - Info bptm (pid=7580) using 128 data buffers

    09/18/2015 13:17:58 - Info bpbrm (pid=6848) sending bpsched msg: CONNECTING TO CLIENT FOR saaclalt1-nbu_1442577174

    09/18/2015 13:17:59 - connecting

    09/18/2015 13:24:30 - Error bpbrm (pid=6848) cannot create data socket, The operation completed successfully.  (0)

    09/18/2015 13:24:30 - Info bpbrm (pid=1756) child done, status 21

    09/18/2015 13:24:30 - Info bpbrm (pid=1756) sending message to media manager: STOP BACKUP saaclalt1-nbu_1442577174

    09/18/2015 13:24:31 - Info bpbrm (pid=1756) media manager for backup id saaclalt1-nbu_1442577174 exited with status 150: termination requested by administrator

    09/18/2015 13:24:31 - end writing

    socket open failed  (21)

  • Marianne's avatar
    Marianne
    Level 6
    9 years ago

    telnet is not successfull from both sides.

    No need to perform any further troublehooting  from NBU side.
    NBU is not going to work if network connection at TCP-level is unsuccessful.

    In which direction is telnet unsuccessful?
    On which port?

    Have you escalated to OS and network admins?