Forum Discussion

Twinkle_Sapra's avatar
12 years ago

NB processes are not coming up

OS :- Red Hat Linux

Master/Media Server

All netbackup processes are not coming. How to troubleshoot further.

 

NB Processes
------------
root     16832     1  0 14:02 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root     16835     1  0 14:02 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root     16901     1  0 14:02 ?        00:00:01 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root     16979     1  0 14:02 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root     16985     1  0 14:02 pts/3    00:00:00 /usr/openv/netbackup/bin/bpdbm
root     17584     1  0 14:09 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 17581 noUserCredentialsFile
root     17587 17584  0 14:09 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 17581 noUserCredentialsFile
root     17593 17584  0 14:09 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 17581 noUserCredentialsFile


MM Processes
------------
root     16965     1  0 14:02 pts/3    00:00:00 /usr/openv/volmgr/bin/ltid
root     16971     1  0 14:02 pts/3    00:00:00 vmd


Shared Symantec Processes
-------------------------
root     17280     1  0 14:02 ?        00:00:00 /opt/VRTSpbx/bin/pbx_exchange

 

BPRD logs

14:35:25.282 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
14:35:25.283 [16974] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 104:Connection reset by peer
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1718: 0: vnet_pbxConnect() failed: 18 0x00000012
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1719: 0: save_errno: 104 0x00000068
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1720: 0: use_vnetd: 0 0x00000000
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1721: 0: cr->vcr_service: bpdbm
14:35:25.283 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 18 0x00000012
14:35:25.291 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
14:35:25.372 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
14:35:25.372 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:35:25.372 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
14:35:25.414 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
14:35:25.414 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
14:35:25.414 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:26.415 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:28.416 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:32.417 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:40.418 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:40.420 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
14:35:40.420 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
14:35:40.420 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
14:35:40.420 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
14:35:40.420 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
14:35:40.420 [16974] <2> db_begin: Could not connect  and returned with exceptionNo connection established
14:35:40.420 [16974] <2> bprd: cannot contact database daemon...exiting
 

PBX is listening

bash-3.2$ netstat -a|grep pbx
tcp        0      0 *:veritas_pbx               *:*                         LISTEN
tcp        0      0 *:veritas_pbx               *:*                         LISTEN

bash-3.2$ netstat -an|grep pbx
bash-3.2$ netstat -anp|grep 1556
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:1556                0.0.0.0:*                   LISTEN      -
tcp        0      0 127.0.0.1:5167              127.0.0.1:1556              ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:41748             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:50202          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:51737             ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:5167              ESTABLISHED -
tcp        0      0 13.215.68.30:1556           13.215.68.30:12072          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:2615              TIME_WAIT   -
tcp        0      0 13.215.68.30:12072          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:55355             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:37720          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:24683             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39030             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39028             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:49779          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39038             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39036             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39034             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39032             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39046             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39044             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39042             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39040             TIME_WAIT   -
tcp        0      0 13.215.68.30:37720          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39050             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:53130          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39048             TIME_WAIT   -
tcp        0      0 13.215.68.30:39669          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:45815          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:22959             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:20935             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:35532             TIME_WAIT   -
tcp        0      0 13.215.68.30:50202          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:1556           13.215.68.30:12499          TIME_WAIT   -
tcp        0      0 13.215.68.30:49779          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:53225          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:53130          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:51737             127.0.0.1:1556              ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:44005             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:53225          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:22006             TIME_WAIT   -
tcp        0     34 13.215.68.30:1556           13.215.68.30:45815          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:60660             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:39669          ESTABLISHED -
tcp        0     24 13.215.68.30:1556           13.215.68.30:46332          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:22523             TIME_WAIT   -
tcp        0      0 :::1556                     :::*                        LISTEN      -

 

PBX settings

Auth User:0 : root
Secure Mode: false
Debug Level: 10
Port Number: 1556
PBX service is not cluster configured

bp.conf

SERVER = gbrhou01lx100
SERVER = gbrhou01lx100.eu.xerox.net
#SERVER = usa0300as736.na.xerox.net
SERVER = usa0300as736
MEDIA_SERVER = gbrhou01lx100
CLIENT_NAME = gbrhou01lx100
USE_VXSS = PROHIBITED
VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY
EMMSERVER = gbrhou01lx100
VXDBMS_NB_DATA = /usr/openv/db/data
CLIENT_CONNECT_TIMEOUT = 2000
CLIENT_READ_TIMEOUT = 2000
ALLOW_MEDIA_OVERWRITE = DBR
ALLOW_MEDIA_OVERWRITE = TAR
ALLOW_MEDIA_OVERWRITE = CPIO
ALLOW_MEDIA_OVERWRITE = ANSI
ALLOW_MEDIA_OVERWRITE = AOS/VS
ALLOW_MEDIA_OVERWRITE = MTF1
ALLOW_MEDIA_OVERWRITE = RS-MTF1
ALLOW_MEDIA_OVERWRITE = BE-MTF1
OPS_CENTER_SERVER_NAME = usa0300as736.na.xerox.net
#VERBOSE = 5
#CONNECT_OPTIONS = localhost 1 0 2
 

 

 

 

6 Replies

  • Services are coming up now but bprd daemon is not listening

     

    -bash-3.2$ sudo bpps -x
    NB Processes
    ------------
    root     20384     1  0 14:47 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
    root     20389     1  0 14:47 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
    root     20460     1  0 14:48 ?        00:00:02 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
    root     20499     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbevtmgr
    root     20519     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbaudit
    root     20567     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbemm
    root     20577     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbrb
    root     20605     1  0 14:48 pts/2    00:00:00 /usr/openv/netbackup/bin/bprd
    root     20639     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
    root     20644     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbjm
    root     20649     1  0 14:48 pts/2    00:00:00 /usr/openv/netbackup/bin/bpdbm
    root     20679     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbpem
    root     20715 20679  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root     20765     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbstserv
    root     20784     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbrmms
    root     20847     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbsl
    root     20878 20644  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbjm
    root     20942     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbars
    root     20992     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbvault
    root     21005     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbsvcmon
    root     21368     1  0 14:48 pts/2    00:00:00 /usr/openv/netbackup/bin/bprd
    root     21373     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
    root     21670     1  0 14:49 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
    root     21674 21670  0 14:49 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
    root     21706 21670  0 14:49 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
    root     21810 21670  0 14:51 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
    root     22287 20847  0 14:54 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -StorageService-2.ior.mgr
    root     22292 20847  0 14:55 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -PolicyManager-2.ior.mgr
    root     22443 20847  0 14:55 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -CatalogManager-2.ior.mgr


    MM Processes
    ------------
    root     20585     1  0 14:48 pts/2    00:00:00 /usr/openv/volmgr/bin/ltid
    root     20777     1  0 14:48 pts/2    00:00:00 vmd
    root     20998 20585  0 14:48 pts/2    00:00:00 tldd
    root     21055 20585  0 14:48 pts/2    00:00:01 avrd
    root     21058     1  0 14:48 pts/2    00:00:00 tldcd


    Shared Symantec Processes
    -------------------------
    root     17280     1  0 14:02 ?        00:00:00 /opt/VRTSpbx/bin/pbx_exchange

    BPRD logs

    14:58:40.386 [20605] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
    14:58:40.387 [20605] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 104:Connection reset by peer
    14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1718: 0: vnet_pbxConnect() failed: 18 0x00000012
    14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1719: 0: save_errno: 104 0x00000068
    14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1720: 0: use_vnetd: 0 0x00000000
    14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1721: 0: cr->vcr_service: bpdbm
    14:58:40.387 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 18 0x00000012
    14:58:40.388 [20605] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
    14:58:40.468 [20605] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
    14:58:40.468 [20605] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
    14:58:40.468 [20605] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
    14:58:40.510 [20605] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
    14:58:40.510 [20605] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
    14:58:40.510 [20605] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
    14:58:40.510 [20605] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
    14:58:40.510 [20605] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
    14:58:40.510 [20605] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
    14:58:40.510 [20605] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
    14:58:40.510 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
    14:58:40.510 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:58:41.511 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
     

     

     

  • All daemons seem to be running now. 

    What happened between 1st and 2nd attempt? Or before 1st attempt that would cause daemons not to come up?

    You won't see bprd 'listening' on 7.x installation.

    To troubleshoot the first bpps output when daemons did not start, you need the '14:02' section of bprd and bpdbm logs, not the '14:35' section.

  • After running /usr/openv/netbackup/bin/bp.start_all  , all daemon comes up.

     

    BPRD logs on 14:02

     14:02:11.658 [16974] <2> bprd: INITIATING bprd (VERBOSE = 0): NetBackup 7.1 2011082514 on gbrhou01lx100
    14:02:11.658 [16974] <2> bprd: Now initializing logging for libcorbaobj
    14:02:11.658 [16974] <2> bprd: the request timeout value is 300 seconds
    14:02:18.187 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
    14:02:18.226 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
    14:02:18.226 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+veritas_pbx,1,20,2,1,0+gbrhou0
    1lx100.txt
    14:02:18.238 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
    14:02:18.238 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+vnetd,1,20,2,1,0+gbrhou01lx100
    .txt
    14:02:18.248 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
    14:02:18.248 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+bpdbm,1,20,2,1,0+gbrhou01lx100
    .txt
    14:02:18.263 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:18.272 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
    14:02:18.353 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
    14:02:18.353 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
    14:02:18.353 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
    14:02:18.393 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
    14:02:18.393 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
    14:02:18.393 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
    14:02:18.393 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
    14:02:18.393 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
    14:02:18.393 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
    14:02:18.393 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
    14:02:18.393 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
    14:02:18.393 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:19.394 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:21.395 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:25.396 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:33.398 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:33.398 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
    14:02:33.398 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
    14:02:33.398 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
    14:02:33.398 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
    14:02:33.398 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
    14:02:33.401 [16974] <2> db_begin: Could not connect  and returned with exceptionNo connection established
    14:02:33.401 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
    14:02:33.402 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:33.405 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
    14:02:33.485 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
    14:02:33.485 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
    14:02:33.485 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
    14:02:33.526 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
    14:02:33.526 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
    14:02:33.526 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
    14:02:33.526 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
    14:02:33.526 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
    14:02:33.526 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
    14:02:33.526 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
    14:02:33.526 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
    14:02:33.526 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:34.527 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:36.527 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:40.529 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:48.532 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
    14:02:48.533 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
    14:02:48.533 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
    14:02:48.533 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
    14:02:48.533 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
    14:02:48.533 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
    14:02:48.533 [16974] <2> db_begin: Could not connect  and returned with exceptionNo connection established
     

    We have 2 other master server with same OS platform and Netbackup version.

    bprd is listen on these master server but not on problemtic server.

    tcp        0      0 *:bprd                      *:*                         LISTEN
    unix  2      [ ACC ]     STREAM     LISTENING     631030693 /usr/openv/var/vnetd/bprd.uds
    unix  3      [ ]         STREAM     CONNECTED     631030707 /tmp/PBXPIPEbprd
     

    One more thing I can see with vnetd daemon has more on working server than problemtic one.

     

    Netstat of vnetd on working server

    tcp        0      0 *:vnetd                     *:*                         LISTEN
    tcp        7      0 13.245.32.15:41169          13.245.32.90:vnetd          CLOSE_WAIT
    unix  2      [ ACC ]     STREAM     LISTENING     905721741 /usr/openv/var/vnetd/vmd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     905721927 /usr/openv/var/vnetd/tldcd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     631027146 /usr/openv/var/vnetd/terminate_vnetd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     631027164 /usr/openv/var/vnetd/terminate_bpcd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     631027173 /usr/openv/var/vnetd/bpcd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     631028346 /usr/openv/var/vnetd/bpcompatd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     631028385 /usr/openv/var/vnetd/bpdbm.uds
    unix  2      [ ACC ]     STREAM     LISTENING     631028609 /usr/openv/var/vnetd/bpjobd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     631030693 /usr/openv/var/vnetd/bprd.uds
    unix  3      [ ]         STREAM     CONNECTED     631027157 /tmp/PBXPIPEvnetd
     

    Netstat of vnetd of problemtic server

     

    tcp        0      0 *:vnetd                     *:*                         LISTEN
    tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:61239         TIME_WAIT
    tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:22100         TIME_WAIT
    tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:citysearch    TIME_WAIT
    tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:30963         TIME_WAIT
    tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:38871         TIME_WAIT
    unix  2      [ ACC ]     STREAM     LISTENING     817425 /usr/openv/var/vnetd/terminate_vnetd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     817449 /usr/openv/var/vnetd/terminate_bpcd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     817457 /usr/openv/var/vnetd/bpcd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     818172 /usr/openv/var/vnetd/bpcompatd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     818518 /usr/openv/var/vnetd/vmd.uds
    unix  2      [ ACC ]     STREAM     LISTENING     819771 /usr/openv/var/vnetd/tldcd.uds
    unix  3      [ ]         STREAM     CONNECTED     867854 /usr/openv/var/vnetd/bpcompatd.uds
    unix  3      [ ]         STREAM     CONNECTED     817437 /tmp/PBXPIPEvnetd
     

     

     

     

     

     

     

  • bprd and bpcompatd are running twice.

    try to stop netbackup, kill remaining processes, restart PBX, and restart NetBackup. Or reboot your sustem.

  • Reboot seems like a good idea.

    We still don't know what happened prior to first failed attempt to start NBU....

  • I did below steps

    1. Stop netbackup services but it didn't kill all processes. Manually killed all processes.

    2. Delete lock files in /usr/openv/var/vnetd and /usr/openv/netbackup/bin.

    3.Restart netbackup services  but it didn't help.

    4. Reboot the system but didn't help.

    5. Decided to reinstall netbackup but renaming /usr/openv to /usr/openv_old and created new /usr/openv

    6. Copy image database, configuration files and EMM database files. It works well afterthat.

     

    !! Thanks a lot to all for your support !!