cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Veritas.com
Support

NB processes are not coming up

Twinkle_Sapra
Level 5
Certified

‎12-18-2012 05:53 AM

OS :- Red Hat Linux

Master/Media Server

All netbackup processes are not coming. How to troubleshoot further.

 

NB Processes
------------
root     16832     1  0 14:02 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root     16835     1  0 14:02 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root     16901     1  0 14:02 ?        00:00:01 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root     16979     1  0 14:02 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root     16985     1  0 14:02 pts/3    00:00:00 /usr/openv/netbackup/bin/bpdbm
root     17584     1  0 14:09 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 17581 noUserCredentialsFile
root     17587 17584  0 14:09 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 17581 noUserCredentialsFile
root     17593 17584  0 14:09 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 17581 noUserCredentialsFile


MM Processes
------------
root     16965     1  0 14:02 pts/3    00:00:00 /usr/openv/volmgr/bin/ltid
root     16971     1  0 14:02 pts/3    00:00:00 vmd


Shared Symantec Processes
-------------------------
root     17280     1  0 14:02 ?        00:00:00 /opt/VRTSpbx/bin/pbx_exchange

 

BPRD logs

14:35:25.282 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
14:35:25.283 [16974] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 104:Connection reset by peer
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1718: 0: vnet_pbxConnect() failed: 18 0x00000012
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1719: 0: save_errno: 104 0x00000068
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1720: 0: use_vnetd: 0 0x00000000
14:35:25.283 [16974] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1721: 0: cr->vcr_service: bpdbm
14:35:25.283 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 18 0x00000012
14:35:25.291 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
14:35:25.372 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
14:35:25.372 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:35:25.372 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
14:35:25.414 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
14:35:25.414 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
14:35:25.414 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
14:35:25.414 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:26.415 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:28.416 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:32.417 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:40.418 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:35:40.420 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
14:35:40.420 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
14:35:40.420 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
14:35:40.420 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
14:35:40.420 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
14:35:40.420 [16974] <2> db_begin: Could not connect  and returned with exceptionNo connection established
14:35:40.420 [16974] <2> bprd: cannot contact database daemon...exiting
 

PBX is listening

bash-3.2$ netstat -a|grep pbx
tcp        0      0 *:veritas_pbx               *:*                         LISTEN
tcp        0      0 *:veritas_pbx               *:*                         LISTEN
bash-3.2$ netstat -an|grep pbx
bash-3.2$ netstat -anp|grep 1556
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:1556                0.0.0.0:*                   LISTEN      -
tcp        0      0 127.0.0.1:5167              127.0.0.1:1556              ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:41748             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:50202          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:51737             ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:5167              ESTABLISHED -
tcp        0      0 13.215.68.30:1556           13.215.68.30:12072          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:2615              TIME_WAIT   -
tcp        0      0 13.215.68.30:12072          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:55355             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:37720          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:24683             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39030             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39028             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:49779          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39038             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39036             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39034             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39032             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39046             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39044             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39042             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39040             TIME_WAIT   -
tcp        0      0 13.215.68.30:37720          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39050             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:53130          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:39048             TIME_WAIT   -
tcp        0      0 13.215.68.30:39669          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:45815          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:22959             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:20935             TIME_WAIT   -
tcp        0      0 127.0.0.1:1556              127.0.0.1:35532             TIME_WAIT   -
tcp        0      0 13.215.68.30:50202          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:1556           13.215.68.30:12499          TIME_WAIT   -
tcp        0      0 13.215.68.30:49779          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:53225          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 13.215.68.30:53130          13.215.68.30:1556           ESTABLISHED -
tcp        0      0 127.0.0.1:51737             127.0.0.1:1556              ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:44005             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:53225          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:22006             TIME_WAIT   -
tcp        0     34 13.215.68.30:1556           13.215.68.30:45815          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:60660             TIME_WAIT   -
tcp        0      0 13.215.68.30:1556           13.215.68.30:39669          ESTABLISHED -
tcp        0     24 13.215.68.30:1556           13.215.68.30:46332          ESTABLISHED -
tcp        0      0 127.0.0.1:1556              127.0.0.1:22523             TIME_WAIT   -
tcp        0      0 :::1556                     :::*                        LISTEN      -

 

PBX settings

Auth User:0 : root
Secure Mode: false
Debug Level: 10
Port Number: 1556
PBX service is not cluster configured

bp.conf

SERVER = gbrhou01lx100
SERVER = gbrhou01lx100.eu.xerox.net
#SERVER = usa0300as736.na.xerox.net
SERVER = usa0300as736
MEDIA_SERVER = gbrhou01lx100
CLIENT_NAME = gbrhou01lx100
USE_VXSS = PROHIBITED
VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY
EMMSERVER = gbrhou01lx100
VXDBMS_NB_DATA = /usr/openv/db/data
CLIENT_CONNECT_TIMEOUT = 2000
CLIENT_READ_TIMEOUT = 2000
ALLOW_MEDIA_OVERWRITE = DBR
ALLOW_MEDIA_OVERWRITE = TAR
ALLOW_MEDIA_OVERWRITE = CPIO
ALLOW_MEDIA_OVERWRITE = ANSI
ALLOW_MEDIA_OVERWRITE = AOS/VS
ALLOW_MEDIA_OVERWRITE = MTF1
ALLOW_MEDIA_OVERWRITE = RS-MTF1
ALLOW_MEDIA_OVERWRITE = BE-MTF1
OPS_CENTER_SERVER_NAME = usa0300as736.na.xerox.net
#VERBOSE = 5
#CONNECT_OPTIONS = localhost 1 0 2
 

 

 

 

6 REPLIES 6

Twinkle_Sapra
Level 5
Certified

‎12-18-2012 06:00 AM

Services are coming up now but bprd daemon is not listening

 

-bash-3.2$ sudo bpps -x
NB Processes
------------
root     20384     1  0 14:47 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root     20389     1  0 14:47 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root     20460     1  0 14:48 ?        00:00:02 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root     20499     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbevtmgr
root     20519     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbaudit
root     20567     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbemm
root     20577     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbrb
root     20605     1  0 14:48 pts/2    00:00:00 /usr/openv/netbackup/bin/bprd
root     20639     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root     20644     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbjm
root     20649     1  0 14:48 pts/2    00:00:00 /usr/openv/netbackup/bin/bpdbm
root     20679     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbpem
root     20715 20679  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root     20765     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbstserv
root     20784     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbrmms
root     20847     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbsl
root     20878 20644  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbjm
root     20942     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbars
root     20992     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbvault
root     21005     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/nbsvcmon
root     21368     1  0 14:48 pts/2    00:00:00 /usr/openv/netbackup/bin/bprd
root     21373     1  0 14:48 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root     21670     1  0 14:49 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
root     21674 21670  0 14:49 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
root     21706 21670  0 14:49 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
root     21810 21670  0 14:51 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc og88427 -1 -1 en_US /usr/openv/java/auth.conf 1 -1 21667 noUserCredentialsFile
root     22287 20847  0 14:54 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -StorageService-2.ior.mgr
root     22292 20847  0 14:55 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -PolicyManager-2.ior.mgr
root     22443 20847  0 14:55 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib -mgrIORFile -CatalogManager-2.ior.mgr


MM Processes
------------
root     20585     1  0 14:48 pts/2    00:00:00 /usr/openv/volmgr/bin/ltid
root     20777     1  0 14:48 pts/2    00:00:00 vmd
root     20998 20585  0 14:48 pts/2    00:00:00 tldd
root     21055 20585  0 14:48 pts/2    00:00:01 avrd
root     21058     1  0 14:48 pts/2    00:00:00 tldcd


Shared Symantec Processes
-------------------------
root     17280     1  0 14:02 ?        00:00:00 /opt/VRTSpbx/bin/pbx_exchange

BPRD logs

14:58:40.386 [20605] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
14:58:40.387 [20605] <2> vnet_pbxConnect: ../../libvlibs/vnet_pbx.c.666: pbxSetAddrEx/pbxConnectEx return error 104:Connection reset by peer
14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1718: 0: vnet_pbxConnect() failed: 18 0x00000012
14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1719: 0: save_errno: 104 0x00000068
14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1720: 0: use_vnetd: 0 0x00000000
14:58:40.387 [20605] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1721: 0: cr->vcr_service: bpdbm
14:58:40.387 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 18 0x00000012
14:58:40.388 [20605] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
14:58:40.468 [20605] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
14:58:40.468 [20605] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:58:40.468 [20605] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
14:58:40.510 [20605] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
14:58:40.510 [20605] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
14:58:40.510 [20605] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
14:58:40.510 [20605] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
14:58:40.510 [20605] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
14:58:40.510 [20605] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
14:58:40.510 [20605] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
14:58:40.510 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
14:58:40.510 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:58:41.511 [20605] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
 

 

 

0 Kudos

Marianne
Level 6
Partner    VIP    Accredited Certified

‎12-18-2012 06:20 AM

All daemons seem to be running now. 

What happened between 1st and 2nd attempt? Or before 1st attempt that would cause daemons not to come up?

You won't see bprd 'listening' on 7.x installation.

To troubleshoot the first bpps output when daemons did not start, you need the '14:02' section of bprd and bpdbm logs, not the '14:35' section.


Handy NetBackup Links
0 Kudos

Twinkle_Sapra
Level 5
Certified

‎12-18-2012 06:59 AM

After running /usr/openv/netbackup/bin/bp.start_all  , all daemon comes up.

 

BPRD logs on 14:02

 14:02:11.658 [16974] <2> bprd: INITIATING bprd (VERBOSE = 0): NetBackup 7.1 2011082514 on gbrhou01lx100
14:02:11.658 [16974] <2> bprd: Now initializing logging for libcorbaobj
14:02:11.658 [16974] <2> bprd: the request timeout value is 300 seconds
14:02:18.187 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
14:02:18.226 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
14:02:18.226 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+veritas_pbx,1,20,2,1,0+gbrhou0
1lx100.txt
14:02:18.238 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
14:02:18.238 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+vnetd,1,20,2,1,0+gbrhou01lx100
.txt
14:02:18.248 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
14:02:18.248 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+bpdbm,1,20,2,1,0+gbrhou01lx100
.txt
14:02:18.263 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:18.272 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
14:02:18.353 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
14:02:18.353 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:02:18.353 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
14:02:18.393 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
14:02:18.393 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
14:02:18.393 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:19.394 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:21.395 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:25.396 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:33.398 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:33.398 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
14:02:33.398 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
14:02:33.398 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
14:02:33.398 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
14:02:33.398 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
14:02:33.401 [16974] <2> db_begin: Could not connect  and returned with exceptionNo connection established
14:02:33.401 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
14:02:33.402 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:33.405 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
14:02:33.485 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
14:02:33.485 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:02:33.485 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
14:02:33.526 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
14:02:33.526 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
14:02:33.526 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:34.527 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:36.527 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:40.529 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:48.532 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:48.533 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
14:02:48.533 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
14:02:48.533 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
14:02:48.533 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
14:02:48.533 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
14:02:48.533 [16974] <2> db_begin: Could not connect  and returned with exceptionNo connection established
 

We have 2 other master server with same OS platform and Netbackup version.

bprd is listen on these master server but not on problemtic server.

tcp        0      0 *:bprd                      *:*                         LISTEN
unix  2      [ ACC ]     STREAM     LISTENING     631030693 /usr/openv/var/vnetd/bprd.uds
unix  3      [ ]         STREAM     CONNECTED     631030707 /tmp/PBXPIPEbprd
 

One more thing I can see with vnetd daemon has more on working server than problemtic one.

 

Netstat of vnetd on working server

tcp        0      0 *:vnetd                     *:*                         LISTEN
tcp        7      0 13.245.32.15:41169          13.245.32.90:vnetd          CLOSE_WAIT
unix  2      [ ACC ]     STREAM     LISTENING     905721741 /usr/openv/var/vnetd/vmd.uds
unix  2      [ ACC ]     STREAM     LISTENING     905721927 /usr/openv/var/vnetd/tldcd.uds
unix  2      [ ACC ]     STREAM     LISTENING     631027146 /usr/openv/var/vnetd/terminate_vnetd.uds
unix  2      [ ACC ]     STREAM     LISTENING     631027164 /usr/openv/var/vnetd/terminate_bpcd.uds
unix  2      [ ACC ]     STREAM     LISTENING     631027173 /usr/openv/var/vnetd/bpcd.uds
unix  2      [ ACC ]     STREAM     LISTENING     631028346 /usr/openv/var/vnetd/bpcompatd.uds
unix  2      [ ACC ]     STREAM     LISTENING     631028385 /usr/openv/var/vnetd/bpdbm.uds
unix  2      [ ACC ]     STREAM     LISTENING     631028609 /usr/openv/var/vnetd/bpjobd.uds
unix  2      [ ACC ]     STREAM     LISTENING     631030693 /usr/openv/var/vnetd/bprd.uds
unix  3      [ ]         STREAM     CONNECTED     631027157 /tmp/PBXPIPEvnetd
 

Netstat of vnetd of problemtic server

 

tcp        0      0 *:vnetd                     *:*                         LISTEN
tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:61239         TIME_WAIT
tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:22100         TIME_WAIT
tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:citysearch    TIME_WAIT
tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:30963         TIME_WAIT
tcp        0      0 gbrhou01lx100:vnetd         gbrhou01lx100:38871         TIME_WAIT
unix  2      [ ACC ]     STREAM     LISTENING     817425 /usr/openv/var/vnetd/terminate_vnetd.uds
unix  2      [ ACC ]     STREAM     LISTENING     817449 /usr/openv/var/vnetd/terminate_bpcd.uds
unix  2      [ ACC ]     STREAM     LISTENING     817457 /usr/openv/var/vnetd/bpcd.uds
unix  2      [ ACC ]     STREAM     LISTENING     818172 /usr/openv/var/vnetd/bpcompatd.uds
unix  2      [ ACC ]     STREAM     LISTENING     818518 /usr/openv/var/vnetd/vmd.uds
unix  2      [ ACC ]     STREAM     LISTENING     819771 /usr/openv/var/vnetd/tldcd.uds
unix  3      [ ]         STREAM     CONNECTED     867854 /usr/openv/var/vnetd/bpcompatd.uds
unix  3      [ ]         STREAM     CONNECTED     817437 /tmp/PBXPIPEvnetd
 

 

 

 

 

 

 

0 Kudos

Yasuhisa_Ishika
Level 6
Partner Accredited Certified

‎12-18-2012 04:51 PM

bprd and bpcompatd are running twice.

try to stop netbackup, kill remaining processes, restart PBX, and restart NetBackup. Or reboot your sustem.

0 Kudos

Marianne
Level 6
Partner    VIP    Accredited Certified

‎12-18-2012 08:11 PM

Reboot seems like a good idea.

We still don't know what happened prior to first failed attempt to start NBU....


Handy NetBackup Links
0 Kudos

Twinkle_Sapra
Level 5
Certified

‎12-18-2012 10:22 PM

I did below steps

1. Stop netbackup services but it didn't kill all processes. Manually killed all processes.

2. Delete lock files in /usr/openv/var/vnetd and /usr/openv/netbackup/bin.

3.Restart netbackup services  but it didn't help.

4. Reboot the system but didn't help.

5. Decided to reinstall netbackup but renaming /usr/openv to /usr/openv_old and created new /usr/openv

6. Copy image database, configuration files and EMM database files. It works well afterthat.

 

!! Thanks a lot to all for your support !!

 

 

0 Kudos