Unable to connect to NBY Client

[Nerdling]

Hi Guys,

I am unable to connect to a new client thatr we have added to our network.  The client resides on the other side of a fire wall.

It looks like to required ports for NBU are open on the fire wall.  Can anyone confirm the required ports?

I can ping from the master/media server to the client but I can't ping the NBU server from the client.

If I attempt to connect to the client from the NBU server I get a 'can not connect on socket (25)' error.

If I run a manual backup of the host this is the out put:

13/12/2011 5:43:04 PM - requesting resource mastersrv-hcart-robot-tld-0
13/12/2011 5:43:04 PM - requesting resource mastersrv.NBU_CLIENT.MAXJOBS.clientsrv
13/12/2011 5:43:04 PM - requesting resource mastersrv.NBU_POLICY.MAXJOBS.FSGE-W2K8
13/12/2011 5:43:04 PM - granted resource mastersrv.NBU_CLIENT.MAXJOBS.clientsrv
13/12/2011 5:43:04 PM - granted resource mastersrv.NBU_POLICY.MAXJOBS.FSGE-W2K8
13/12/2011 5:43:04 PM - granted resource KP3498
13/12/2011 5:43:04 PM - granted resource IBM.ULTRIUM-HH4.000
13/12/2011 5:43:04 PM - granted resource mastersrv-hcart-robot-tld-0
13/12/2011 5:43:04 PM - estimated 0 Kbytes needed
13/12/2011 5:43:05 PM - started process bpbrm (5452)
13/12/2011 5:45:51 PM - Error bpbrm(pid=5452) Cannot connect to clientsrv        
13/12/2011 5:45:51 PM - end writing
can't connect to client(58)

Any help would be fantastic.

[Marianne]

You need port 13724 (vnetd) open in both directions as a minimum.

As from NBU 7.0.1, first connection attempt will be on port 1556 (pbx). If connection cannot be established, the server will then failback to port 13724.

Please test connectivity as follows:

Create bpcd log folder on client under netbackup/logs.

Run the following on the master from cmd (in admincmd folder):

bptestbpcd -client <client-name> -verbose -debug

Check client's bpcd log file - you will see connection attempt from master's IP address.
Next, the client needs to resolve the IP address to a hostname that corresponds with the NBU SERVER entry on the client.

If this reverse lookup fails, you will see a status 58.

Add server's IP and hostname to Client's hosts file.

If you need us to assist with further troubleshooting, please post output of bptestbpcd as well as client's bpcd log.

Please also mention NBU version as well as OS on master and client.

[Nerdling]

Hi Marianne,

here is the out put:

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client NBU_CLIENT -verbose -debug
16:49:50.466 [2752.2648] <2> bptestbpcd: VERBOSE = 5
16:49:50.481 [2752.2648] <2> db_freeEXDB_INFO: ?
16:49:50.481 [2752.2648] <2> local_bpcr_connect: bpcr.c.276: connect_opts = 0x01030202 connect_opts2 = 0x01000100
16:49:50.481 [2752.2648] <2> local_bpcr_connect: bpcr.c.283: connect_opts = 0x01000100
16:49:50.481 [2752.2648] <2> local_bpcr_connect: bpcr.c.322: daemon_port_type =0
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1046: host_cache_size: 200 0x000000c8
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1047: cache_time: 3600 0x00000e10
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1061: host_failed_cache_size: 40 0x00000028
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1062: cache_time: 3600 0x00000e10
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1046: host_cache_size: 200 0x000000c8
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1047: cache_time: 3600 0x00000e10
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1061: host_failed_cache_size: 40 0x00000028
16:49:50.481 [2752.2648] <2> init_cache: ../../libvlibs/vnet_hosts.c.1062: cache_time: 3600 0x00000e10
16:49:50.481 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 0 0x00000000
16:50:00.496 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 1 0x00000001
16:50:12.540 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:50:21.510 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:50:22.524 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 0 0x00000000
16:50:32.539 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 1 0x00000001
16:50:43.552 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:50:53.568 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 0 0x00000000
16:51:03.583 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:51:14.596 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:51:14.596 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 1 0x00000001
16:51:24.612 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 0 0x00000000
16:51:35.610 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:51:45.640 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:51:45.640 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 1 0x00000001
16:51:55.656 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 0 0x00000000
16:52:06.654 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:52:16.684 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:52:16.684 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4052: connect in progress: 1 0x00000001
16:52:37.713 [2752.2648] <2> vnet_async_connect: ../../libvlibs/vnet_vnetd.c.4250: getsockopt SO_ERROR returned: 10060 0x0000274c
16:52:37.713 [2752.2648] <2> vnet_connect_to_service_or_vnetd: ../../libvlibs/vnet_vnetd.c.3584: vnet_async_connect failed: 18 0x00000012
16:52:37.713 [2752.2648] <2> nb_connect_to_vnetd_or_legacy: ../../../nb/lib/comm.c.2028: vnet_connect_to_vnetd_or_service failed: 18
16:52:37.713 [2752.2648] <2> local_bpcr_connect: nb_connect_to_vnetd_service(NBU_CLIENT) failed: 25
16:52:37.713 [2752.2648] <2> local_bpcr_connect: Can't connect to client NBU_CLIENT
16:52:37.713 [2752.2648] <2> ConnectToBPCD: bpcd_connect_and_verify(NBU_CLIENT, NBU_CLIENT) failed: 25<16>bptestbpcd main: Function ConnectToBPCD(NBU_CLIENT) failed: 25
16:52:37.713 [2752.2648] <16> bptestbpcd main: Function ConnectToBPCD(NBU_CLIENT) failed: 25
<2>bptestbpcd: cannot connect on socket
16:52:37.713 [2752.2648] <2> bptestbpcd: cannot connect on socket<2>bptestbpcd: EXIT status = 25
16:52:37.713 [2752.2648] <2> bptestbpcd: EXIT status = 25
cannot connect on socket

NBU is 7.0 running on a Win2K8r2 server.

[Marianne]

As per my previous post, we also need the bpcd log on the client to see what happend on that side.

"

Create bpcd log folder on client under netbackup/logs.

Run the following on the master from cmd (in admincmd folder):

bptestbpcd -client <client-name> -verbose -debug

Check client's bpcd log file - you will see connection attempt from master's IP address.
Next, the client needs to resolve the IP address to a hostname that corresponds with the NBU SERVER entry on the client.

"

Can you verify that vnetd is open in both directions between master and client?

[Nerdling]

Sorry, I mis read your post.

I'm advised that both ports 13724 and 1556 are open for the hosts in question.

Here is the out put from the bpcd logs folder:

13:53:34.605 [4120.4672] <2> setup_debug_log: switched debug log file for bpcd
13:53:34.605 [4120.4672] <2> bpcd main: VERBOSE = 0
13:53:34.605 [4120.4672] <2> logparams: bpcd
13:53:34.605 [4120.4672] <2> bpcd main: Got socket for input 304
13:53:34.605 [4120.4672] <2> process_requests: offset to GMT -36000
13:53:34.605 [4120.4672] <2> logconnections: BPCD ACCEPT FROM MASTER_IP_AD.58506 TO CLIENT_IP_AD.13724 fd = 304
13:53:34.605 [4120.4672] <2> process_requests: setup_sockopts complete
13:53:34.605 [4120.4672] <2> bpcd peer_hostname: Connection from host MASTER_NBU (MASTER_IP_AD) port 58506
13:53:34.605 [4120.4672] <2> bpcd valid_server: comparing MASTER_NBU and MASTER_NBU
13:53:34.605 [4120.4672] <4> bpcd valid_server: hostname comparison succeeded
13:53:34.777 [4120.4672] <2> process_requests: output socket port number = 1
13:53:35.245 [4120.4672] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1527: 0: in failed file cache: 10109 0x0000277d
13:53:35.245 [4120.4672] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1528: 0: in failed file cache name: NULL
13:53:35.245 [4120.4672] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1529: 0: in failed file cache service: testdaemon
13:53:35.245 [4120.4672] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2474: 0: vnet_cached_getaddrinfo failed: 6 0x00000006
13:53:35.245 [4120.4672] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2475: 0: vnet_cached_getaddrinfo failed ret: 10109 0x0000277d
13:53:35.245 [4120.4672] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2476: 0: vnet_cached_getaddrinfo failed service: testdaemon
13:53:35.245 [4120.4672] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1827: 0: vnet_cached_get_service_port() failed: 6 0x00000006
13:53:35.245 [4120.4672] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1828: 0: service: 49516
13:53:35.245 [4120.4672] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
13:53:35.245 [4120.4672] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: C:\Program Files\Veritas\NetBackup\var\host_cache\14f\6cf9fd4f+49516,1,0,2,1,0+127.0.0.1.txt
13:53:35.260 [4120.4672] <16> process_requests: setsockopt failed: h_errno 10038
13:53:35.260 [4120.4672] <2> process_requests: Duplicated vnetd socket on stderr
13:53:35.260 [4120.4672] <2> process_requests: <---- NetBackup 7.1 0 ------------initiated
13:53:35.260 [4120.4672] <2> process_requests: VERBOSE = 0
13:53:35.260 [4120.4672] <2> process_requests: Not using VxSS authentication with MASTER_NBU
13:53:35.260 [4120.4672] <2> put_string: cannot write data to network:  An operation was attempted on something that is not a socket.
13:53:35.260 [4120.4672] <16> process_requests: couldn't write connected message
 

Can I assume from the above that I have problems talking on the following ports?
304
58506
1527
1528
1529
2474
2475
2476
1827
1828
6635
6636
10038
 

[Nerdling]

Further update: we have another server in the same network, behind the same firewall and everything is working for it.  We have also run a 'packet capture' on the fire wall and were seeing comms between both hosts.

[Marianne · Accepted solution]

We still don't know which OS on client....

If Windows, please confirm that Windows firewall has been disabled.

We see the following in bpcd log:

process_requests: setsockopt failed: h_errno 10038

'Something' on your client is preventing socket setup.

We have seen similar posts where reinstallation of NBU client software has solved the problem, for example this one: https://www-secure.symantec.com/connect/forums/backup-no-done

[Nerdling]

Thanks Marianne,

Will give that a go.  Sorry (again) OS is Win2K8r2 Std on both Master and client servers.

[Nerdling]

Happy New Year Marianne,

thankyou for your help, I think this is the second problem that you have helped me with.  Sorry for the slow responce.  The last thing I did last year before going on holidays was to reinstall this NBU client, this has resolved the problem.

thanks