Connectivity issue from Netbackup 6.0 Master to linux client desktop

[dbeck]

Folks,

Getting Error (24) socket write failure when running backup job on linux client.

When running this command bptestbpcd -client licorice -debug from the master server (candystore) to the client (licorice) this is the output.

[root@candystore admincmd]# ./bptestbpcd -client licorice
<16>bptestbpcd main: Function ConnectToBPCD(licorice) failed: 24
socket write failed
[root@candystore admincmd]# ./bptestbpcd -client licorice -debug
08:56:06.009 [10907] <2> ParseConfigExA: Option (CLIENT_NAME) redefined on line 3, discarding definition on line 2.
08:56:06.042 [10907] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2033: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
08:56:06.042 [10907] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: service: bpcd
08:56:06.083 [10907] <2> logconnections: BPCD CONNECT FROM 10.243.223.103.60393 TO 10.243.223.97.13724
08:56:06.083 [10907] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.178: msg: VNETD CONNECT FROM 10.243.223.103.42775 TO 10.243.223.97.13724 fd = 4
08:56:06.113 [10907] <2> vnet_vnetd_connect_forward_socket_end: vnet_vnetd.c.528: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
08:56:06.155 [10907] <2> vnet_vnetd_connect_forward_socket_end: vnet_vnetd.c.546: ipc_string: /tmp/vnet-06952302699358376053000000000-bnUzKB
08:56:06.195 [10907] <2> put_long: (11) network write() error: Connection reset by peer (104); socket = 3
08:56:06.195 [10907] <2> bpcr_put_vnetd_forward_socket: put_string /tmp/vnet-06952302699358376053000000000-bnUzKB failed: 104
08:56:06.195 [10907] <2> bpcr_connect: bpcr_put_vnetd_forward_socket failed: 24
08:56:06.195 [10907] <2> ConnectToBPCD: bpcd_connect_and_verify(licorice, licorice) failed: 24
<16>bptestbpcd main: Function ConnectToBPCD(licorice) failed: 24
08:56:06.196 [10907] <16> bptestbpcd main: Function ConnectToBPCD(licorice) failed: 24
<2>bptestbpcd: socket write failed
08:56:06.196 [10907] <2> bptestbpcd: socket write failed
<2>bptestbpcd: EXIT status = 24
08:56:06.196 [10907] <2> bptestbpcd: EXIT status = 24
socket write failed

On the client (licorice) and the master server (candystore) I have modified the host files to include the ip's of both.  I can ping the client desktop from the master server and can ping the master sever from the client.  

Any inputs would be most helpful.

Thanks, Dave

[Marianne]

Please verify that the Linux client also contains an entry for itself in it's /etc/hosts file.

Check netstat -a on client - verify that bpcd and vnetd are LISTENING.

Please create bpcd log dir on client an run bptestbpcd again. Post bpcd log file.

Also let us know which Linux version.

[dbeck]

Marianne,

This is the host file of the linux client (licorice) and has a entry for itself

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost
10.243.223.97   licorice
10.243.212.14   wonka
10.243.212.15   mars
10.243.230.69   term1
10.243.210.21   dc-mda01
10.243.212.20   candystore

This is a partical copy of netstat -a ran on linux client (licorice) and shows bpcd and vnetd listing

[root@licorice /]# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State      
tcp        0      0 localhost:2208              *:*                         LISTEN      
tcp        0      0 *:56165                     *:*                         LISTEN      
tcp        0      0 *:5900                      *:*                         LISTEN      
tcp        0      0 *:sunrpc                    *:*                         LISTEN      
tcp        0      0 *:881                       *:*                         LISTEN      
tcp        0      0 *:bpcd                      *:*                         LISTEN      
tcp        0      0 *:ha-cluster                *:*                         LISTEN      
tcp        0      0 *:vopied                    *:*                         LISTEN      
tcp        0      0 localhost:ipp               *:*                         LISTEN      
tcp        0      0 *:bpjava-msvc               *:*                         LISTEN      
tcp        0      0 *:vnetd                     *:*                         LISTEN      
tcp        0      0 localhost:2207              *:*                         LISTEN      
tcp        0      0 licorice:908                wonka:nfs                   ESTABLISHED
tcp        0      0 licorice:55658              candystore:ssh              ESTABLISHED
tcp        0      0 licorice:57185              a184-24-84-111.deploy:https ESTABLISHED
tcp        0      0 licorice:57184              a184-24-84-111.deploy:https ESTABLISHED
tcp        0      0 *:ssh                       *:*                         LISTEN      
udp        0      0 *:46852                     *:*                                     
udp        0      0 *:realm-rusd                *:*                                     
udp        0      0 *:msexch-routing            *:*                                     
udp        0      0 *:bootpc                    *:*                                     
udp        0      0 *:50268                     *:*                                     
udp        0      0 *:mdns                      *:*                                     
udp        0      0 *:878                       *:*                                     
udp        0      0 *:sunrpc                    *:*                                     
udp        0      0 *:3com-amp3                 *:*                                     
udp        0      0 *:ipp                       *:*                                     
udp        0      0 *:mdns                      *:*                                     
udp        0      0 *:60268                     *:*                                     
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     7048   @/var/run/hald/dbus-IcYs75Rn1j
unix  2      [ ACC ]     STREAM     LISTENING     8748   @/tmp/fam-root-
unix  2      [ ACC ]     STREAM  

Linux client (licorice) is at the following version. Same version running on Master Server (candystore)

NetBackup-RedHat2.4 6.0MP7S01

The client (licorice) does not have the bptestbpcd command.  Not sure where to create the bpcd log directory. Does this get created on the master server.? Only see the bptestbpcd command in the /usr/openv/netbackup/bin/admincmd directory on the master server. How does the ouput get directed to this directory when created?

Dave

[dbeck]

Marianne,

Also ran the ./bpclntcmd -hn command from the linux client (licorice) to the master server (candystore)

/bpclntcmd -hn candystore

host candystore: candystore at 10.243.212.20

Ran the same command from the master server to the linux client (licorice)

./bpclntcmd -hn licorice

host licorice: licorice at 10.243.223.97

Just want to pass this along as well.

-D-

[dbeck]

Marianne,

This is the contents of the bpcd log file on client licorice after running the bptestbpcd command.

17:57:55.356 [8372] <2> bpcd main: offset to GMT 18000
17:57:55.356 [8372] <2> logconnections: BPCD ACCEPT FROM 10.243.223.103.52477 TO 10.243.223.97.13724
17:57:55.356 [8372] <2> bpcd main: setup_sockopts complete
17:57:55.367 [8372] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
17:57:55.367 [8372] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1
17:57:55.367 [8372] <16> bpcd main: Couldn't get peer hostname

Dave

[Marianne]

You need to create bpcd log dir under /usr/openv/netbackup/logs on the client, then run bptestbpcd again from the server (exactly as per your opening post), then post the log file that should be in the client's bpcd folder.

[dbeck]

Marianne,

This is the results after running the bptestbpcd.

17:57:55.356 [8372] <2> bpcd main: offset to GMT 18000
17:57:55.356 [8372] <2> logconnections: BPCD ACCEPT FROM 10.243.223.103.52477 TO 10.243.223.97.13724
17:57:55.356 [8372] <2> bpcd main: setup_sockopts complete
17:57:55.367 [8372] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
17:57:55.367 [8372] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1
17:57:55.367 [8372] <16> bpcd main: Couldn't get peer hostname
18:28:19.179 [8795] <2> bpcd main: offset to GMT 18000
18:28:19.179 [8795] <2> logconnections: BPCD ACCEPT FROM 10.243.223.103.51710 TO 10.243.223.97.13724
18:28:19.179 [8795] <2> bpcd main: setup_sockopts complete
18:28:19.190 [8795] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
18:28:19.190 [8795] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1
18:28:19.190 [8795] <16> bpcd main: Couldn't get peer hostname

[dbeck]

Marrianne,

Think I may have found the problem.  From the BPCD log file it shows BPCD ACCEPT FROM 10.243.223.103 TO 10.243.223.97.13724.  The 10.243.223.97 ip address is licorice the linux client on nic eth0.  The Netbackup Master Server (candystore) has two nic's connected. Nic eth0 is ip 10.243.212.20 and is defined in the host files and NIS.  The 10.243.223.103 is not defined in the host files anywhere and is nic eth1 on the master server which allows connection to the 10.243.223.x subnet.  I think everything will resolve by changing the Master servers ip to the 10.243.223.103  which is the same subnet the linux client is on.  Hope this makes sense.

Dave

[AbdulRasheed]

Hi Dave,

  As you can see in the bpcd log, your Linux desktop is unable to resolve the IP address 10.243.223.103  (which must be your master server) to hostname. You can fix this by adding the hostname and address of the master server to /etc/hosts file in you Linux system. Or you can ask the Linux desktop user to point the system to a DNS server that knows about the master server.

  If this fixes the issue, please mark Marianne's response as a solution. She walked you through the entire process, I wouldn't want to hijack her brilliant guidance.

[AbdulRasheed]

You don't need to change the IP address of the master. Just make sure that the name associated with the IP address of the second interface on master is added to /etc/hosts on Linux client. This name should also be added to the bp.conf of the Linux client.

[Marianne · Accepted solution]

Hi Dave

I'm glad you figured it out once you saw the incoming IP address in bpcd log.
licorice could not resolve this IP to a hostname: bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)

Please let us know if all is well after adding hosts and bp.conf entries on licorice. (I love your hostnames!)

Rasheed, thanks for carrying on when I 'faded' (it was midnight for me).

[dbeck]

All,

Everything is working!!!  Backed up some directories on linux client (licorice) to the disk array on the master server (candystore).

Marked Marianne's post as the solution.

Thanks everyone for all your inputs.

Dave