After running /usr/openv/netbackup/bin/bp.start_all , all daemon comes up.
BPRD logs on 14:02
14:02:11.658 [16974] <2> bprd: INITIATING bprd (VERBOSE = 0): NetBackup 7.1 2011082514 on gbrhou01lx100
14:02:11.658 [16974] <2> bprd: Now initializing logging for libcorbaobj
14:02:11.658 [16974] <2> bprd: the request timeout value is 300 seconds
14:02:18.187 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
14:02:18.226 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
14:02:18.226 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+veritas_pbx,1,20,2,1,0+gbrhou0
1lx100.txt
14:02:18.238 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
14:02:18.238 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+vnetd,1,20,2,1,0+gbrhou01lx100
.txt
14:02:18.248 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
14:02:18.248 [16974] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: /usr/openv/var/host_cache/0e6/f380aee6+bpdbm,1,20,2,1,0+gbrhou01lx100
.txt
14:02:18.263 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:18.272 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
14:02:18.353 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
14:02:18.353 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:02:18.353 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
14:02:18.393 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
14:02:18.393 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
14:02:18.393 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
14:02:18.393 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:19.394 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:21.395 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:25.396 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:33.398 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:33.398 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
14:02:33.398 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
14:02:33.398 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
14:02:33.398 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
14:02:33.398 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
14:02:33.401 [16974] <2> db_begin: Could not connect and returned with exceptionNo connection established
14:02:33.401 [16974] <2> ConnectionCache::connectAndCache: Acquiring new connection for host gbrhou01lx100, query type 98
14:02:33.402 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:33.405 [16974] <2> vnet_vnetd_pbx_c_supported: ../../libvlibs/vnet_vnetd.c.3435: 0: VN_REQUEST_PBX_C_SUPPORTED: 13 0x0000000d
14:02:33.485 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000
14:02:33.485 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1957: 0: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:02:33.485 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1971: 0: service: bpdbm
14:02:33.526 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2 0x00000002
14:02:33.526 [16974] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Function failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.1987: 0: status: 9 0x00000009
14:02:33.526 [16974] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1664: 0: vnet_vnetd_service_socket failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353: 0: do_service failed: 9 0x00000009
14:02:33.526 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:34.527 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:36.527 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:40.529 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:48.532 [16974] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 111 0x0000006f
14:02:48.533 [16974] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
14:02:48.533 [16974] <2> vnet_connect_to_service: ../../libvlibs/vnet_connect.c.174: 0: connect_to_service() failed: 18 0x00000012
14:02:48.533 [16974] <2> ConnectionCache::connectToBpdbm: vnet_connect_to_service(gbrhou01lx100) failed: Operation now in progress (115), vnet status 18
14:02:48.533 [16974] <2> ConnectionCache::connectAndCache: connect_to_bpdbm(gbrhou01lx100) failed status 25 err 115
14:02:48.533 [16974] <2> ConnectionCache::connectAndCache: Return value is (21)
14:02:48.533 [16974] <2> db_begin: Could not connect and returned with exceptionNo connection established
We have 2 other master server with same OS platform and Netbackup version.
bprd is listen on these master server but not on problemtic server.
tcp 0 0 *:bprd *:* LISTEN
unix 2 [ ACC ] STREAM LISTENING 631030693 /usr/openv/var/vnetd/bprd.uds
unix 3 [ ] STREAM CONNECTED 631030707 /tmp/PBXPIPEbprd
One more thing I can see with vnetd daemon has more on working server than problemtic one.
Netstat of vnetd on working server
tcp 0 0 *:vnetd *:* LISTEN
tcp 7 0 13.245.32.15:41169 13.245.32.90:vnetd CLOSE_WAIT
unix 2 [ ACC ] STREAM LISTENING 905721741 /usr/openv/var/vnetd/vmd.uds
unix 2 [ ACC ] STREAM LISTENING 905721927 /usr/openv/var/vnetd/tldcd.uds
unix 2 [ ACC ] STREAM LISTENING 631027146 /usr/openv/var/vnetd/terminate_vnetd.uds
unix 2 [ ACC ] STREAM LISTENING 631027164 /usr/openv/var/vnetd/terminate_bpcd.uds
unix 2 [ ACC ] STREAM LISTENING 631027173 /usr/openv/var/vnetd/bpcd.uds
unix 2 [ ACC ] STREAM LISTENING 631028346 /usr/openv/var/vnetd/bpcompatd.uds
unix 2 [ ACC ] STREAM LISTENING 631028385 /usr/openv/var/vnetd/bpdbm.uds
unix 2 [ ACC ] STREAM LISTENING 631028609 /usr/openv/var/vnetd/bpjobd.uds
unix 2 [ ACC ] STREAM LISTENING 631030693 /usr/openv/var/vnetd/bprd.uds
unix 3 [ ] STREAM CONNECTED 631027157 /tmp/PBXPIPEvnetd
Netstat of vnetd of problemtic server
tcp 0 0 *:vnetd *:* LISTEN
tcp 0 0 gbrhou01lx100:vnetd gbrhou01lx100:61239 TIME_WAIT
tcp 0 0 gbrhou01lx100:vnetd gbrhou01lx100:22100 TIME_WAIT
tcp 0 0 gbrhou01lx100:vnetd gbrhou01lx100:citysearch TIME_WAIT
tcp 0 0 gbrhou01lx100:vnetd gbrhou01lx100:30963 TIME_WAIT
tcp 0 0 gbrhou01lx100:vnetd gbrhou01lx100:38871 TIME_WAIT
unix 2 [ ACC ] STREAM LISTENING 817425 /usr/openv/var/vnetd/terminate_vnetd.uds
unix 2 [ ACC ] STREAM LISTENING 817449 /usr/openv/var/vnetd/terminate_bpcd.uds
unix 2 [ ACC ] STREAM LISTENING 817457 /usr/openv/var/vnetd/bpcd.uds
unix 2 [ ACC ] STREAM LISTENING 818172 /usr/openv/var/vnetd/bpcompatd.uds
unix 2 [ ACC ] STREAM LISTENING 818518 /usr/openv/var/vnetd/vmd.uds
unix 2 [ ACC ] STREAM LISTENING 819771 /usr/openv/var/vnetd/tldcd.uds
unix 3 [ ] STREAM CONNECTED 867854 /usr/openv/var/vnetd/bpcompatd.uds
unix 3 [ ] STREAM CONNECTED 817437 /tmp/PBXPIPEvnetd