04-13-2011 06:36 AM
Folks,
Getting Error (24) socket write failure when running backup job on linux client.
When running this command bptestbpcd -client licorice -debug from the master server (candystore) to the client (licorice) this is the output.
[root@candystore admincmd]# ./bptestbpcd -client licorice
<16>bptestbpcd main: Function ConnectToBPCD(licorice) failed: 24
socket write failed
[root@candystore admincmd]# ./bptestbpcd -client licorice -debug
08:56:06.009 [10907] <2> ParseConfigExA: Option (CLIENT_NAME) redefined on line 3, discarding definition on line 2.
08:56:06.042 [10907] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2033: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
08:56:06.042 [10907] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: service: bpcd
08:56:06.083 [10907] <2> logconnections: BPCD CONNECT FROM 10.243.223.103.60393 TO 10.243.223.97.13724
08:56:06.083 [10907] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.178: msg: VNETD CONNECT FROM 10.243.223.103.42775 TO 10.243.223.97.13724 fd = 4
08:56:06.113 [10907] <2> vnet_vnetd_connect_forward_socket_end: vnet_vnetd.c.528: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
08:56:06.155 [10907] <2> vnet_vnetd_connect_forward_socket_end: vnet_vnetd.c.546: ipc_string: /tmp/vnet-06952302699358376053000000000-bnUzKB
08:56:06.195 [10907] <2> put_long: (11) network write() error: Connection reset by peer (104); socket = 3
08:56:06.195 [10907] <2> bpcr_put_vnetd_forward_socket: put_string /tmp/vnet-06952302699358376053000000000-bnUzKB failed: 104
08:56:06.195 [10907] <2> bpcr_connect: bpcr_put_vnetd_forward_socket failed: 24
08:56:06.195 [10907] <2> ConnectToBPCD: bpcd_connect_and_verify(licorice, licorice) failed: 24
<16>bptestbpcd main: Function ConnectToBPCD(licorice) failed: 24
08:56:06.196 [10907] <16> bptestbpcd main: Function ConnectToBPCD(licorice) failed: 24
<2>bptestbpcd: socket write failed
08:56:06.196 [10907] <2> bptestbpcd: socket write failed
<2>bptestbpcd: EXIT status = 24
08:56:06.196 [10907] <2> bptestbpcd: EXIT status = 24
socket write failed
On the client (licorice) and the master server (candystore) I have modified the host files to include the ip's of both. I can ping the client desktop from the master server and can ping the master sever from the client.
Any inputs would be most helpful.
Thanks, Dave
Solved! Go to Solution.
04-13-2011 10:51 PM
Hi Dave
I'm glad you figured it out once you saw the incoming IP address in bpcd log.
licorice could not resolve this IP to a hostname: bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
Please let us know if all is well after adding hosts and bp.conf entries on licorice. (I love your hostnames!)
Rasheed, thanks for carrying on when I 'faded' (it was midnight for me).
04-13-2011 07:02 AM
Please verify that the Linux client also contains an entry for itself in it's /etc/hosts file.
Check netstat -a on client - verify that bpcd and vnetd are LISTENING.
Please create bpcd log dir on client an run bptestbpcd again. Post bpcd log file.
Also let us know which Linux version.
04-13-2011 10:15 AM
Marianne,
This is the host file of the linux client (licorice) and has a entry for itself
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost
10.243.223.97 licorice
10.243.212.14 wonka
10.243.212.15 mars
10.243.230.69 term1
10.243.210.21 dc-mda01
10.243.212.20 candystore
This is a partical copy of netstat -a ran on linux client (licorice) and shows bpcd and vnetd listing
[root@licorice /]# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:2208 *:* LISTEN
tcp 0 0 *:56165 *:* LISTEN
tcp 0 0 *:5900 *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 *:881 *:* LISTEN
tcp 0 0 *:bpcd *:* LISTEN
tcp 0 0 *:ha-cluster *:* LISTEN
tcp 0 0 *:vopied *:* LISTEN
tcp 0 0 localhost:ipp *:* LISTEN
tcp 0 0 *:bpjava-msvc *:* LISTEN
tcp 0 0 *:vnetd *:* LISTEN
tcp 0 0 localhost:2207 *:* LISTEN
tcp 0 0 licorice:908 wonka:nfs ESTABLISHED
tcp 0 0 licorice:55658 candystore:ssh ESTABLISHED
tcp 0 0 licorice:57185 a184-24-84-111.deploy:https ESTABLISHED
tcp 0 0 licorice:57184 a184-24-84-111.deploy:https ESTABLISHED
tcp 0 0 *:ssh *:* LISTEN
udp 0 0 *:46852 *:*
udp 0 0 *:realm-rusd *:*
udp 0 0 *:msexch-routing *:*
udp 0 0 *:bootpc *:*
udp 0 0 *:50268 *:*
udp 0 0 *:mdns *:*
udp 0 0 *:878 *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:3com-amp3 *:*
udp 0 0 *:ipp *:*
udp 0 0 *:mdns *:*
udp 0 0 *:60268 *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 7048 @/var/run/hald/dbus-IcYs75Rn1j
unix 2 [ ACC ] STREAM LISTENING 8748 @/tmp/fam-root-
unix 2 [ ACC ] STREAM
Linux client (licorice) is at the following version. Same version running on Master Server (candystore)
NetBackup-RedHat2.4 6.0MP7S01
The client (licorice) does not have the bptestbpcd command. Not sure where to create the bpcd log directory. Does this get created on the master server.? Only see the bptestbpcd command in the /usr/openv/netbackup/bin/admincmd directory on the master server. How does the ouput get directed to this directory when created?
Dave
04-13-2011 02:05 PM
Marianne,
Also ran the ./bpclntcmd -hn command from the linux client (licorice) to the master server (candystore)
/bpclntcmd -hn candystore
host candystore: candystore at 10.243.212.20
Ran the same command from the master server to the linux client (licorice)
./bpclntcmd -hn licorice
host licorice: licorice at 10.243.223.97
Just want to pass this along as well.
-D-
04-13-2011 03:03 PM
Marianne,
This is the contents of the bpcd log file on client licorice after running the bptestbpcd command.
17:57:55.356 [8372] <2> bpcd main: offset to GMT 18000
17:57:55.356 [8372] <2> logconnections: BPCD ACCEPT FROM 10.243.223.103.52477 TO 10.243.223.97.13724
17:57:55.356 [8372] <2> bpcd main: setup_sockopts complete
17:57:55.367 [8372] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
17:57:55.367 [8372] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1
17:57:55.367 [8372] <16> bpcd main: Couldn't get peer hostname
Dave
04-13-2011 03:03 PM
You need to create bpcd log dir under /usr/openv/netbackup/logs on the client, then run bptestbpcd again from the server (exactly as per your opening post), then post the log file that should be in the client's bpcd folder.
04-13-2011 03:29 PM
Marianne,
This is the results after running the bptestbpcd.
17:57:55.356 [8372] <2> bpcd main: offset to GMT 18000
17:57:55.356 [8372] <2> logconnections: BPCD ACCEPT FROM 10.243.223.103.52477 TO 10.243.223.97.13724
17:57:55.356 [8372] <2> bpcd main: setup_sockopts complete
17:57:55.367 [8372] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
17:57:55.367 [8372] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1
17:57:55.367 [8372] <16> bpcd main: Couldn't get peer hostname
18:28:19.179 [8795] <2> bpcd main: offset to GMT 18000
18:28:19.179 [8795] <2> logconnections: BPCD ACCEPT FROM 10.243.223.103.51710 TO 10.243.223.97.13724
18:28:19.179 [8795] <2> bpcd main: setup_sockopts complete
18:28:19.190 [8795] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
18:28:19.190 [8795] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1
18:28:19.190 [8795] <16> bpcd main: Couldn't get peer hostname
04-13-2011 04:24 PM
Marrianne,
Think I may have found the problem. From the BPCD log file it shows BPCD ACCEPT FROM 10.243.223.103 TO 10.243.223.97.13724. The 10.243.223.97 ip address is licorice the linux client on nic eth0. The Netbackup Master Server (candystore) has two nic's connected. Nic eth0 is ip 10.243.212.20 and is defined in the host files and NIS. The 10.243.223.103 is not defined in the host files anywhere and is nic eth1 on the master server which allows connection to the 10.243.223.x subnet. I think everything will resolve by changing the Master servers ip to the 10.243.223.103 which is the same subnet the linux client is on. Hope this makes sense.
Dave
04-13-2011 04:31 PM
Hi Dave,
As you can see in the bpcd log, your Linux desktop is unable to resolve the IP address 10.243.223.103 (which must be your master server) to hostname. You can fix this by adding the hostname and address of the master server to /etc/hosts file in you Linux system. Or you can ask the Linux desktop user to point the system to a DNS server that knows about the master server.
If this fixes the issue, please mark Marianne's response as a solution. She walked you through the entire process, I wouldn't want to hijack her brilliant guidance.
04-13-2011 04:35 PM
You don't need to change the IP address of the master. Just make sure that the name associated with the IP address of the second interface on master is added to /etc/hosts on Linux client. This name should also be added to the bp.conf of the Linux client.
04-13-2011 10:51 PM
Hi Dave
I'm glad you figured it out once you saw the incoming IP address in bpcd log.
licorice could not resolve this IP to a hostname: bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
Please let us know if all is well after adding hosts and bp.conf entries on licorice. (I love your hostnames!)
Rasheed, thanks for carrying on when I 'faded' (it was midnight for me).
04-15-2011 08:53 AM
All,
Everything is working!!! Backed up some directories on linux client (licorice) to the disk array on the master server (candystore).
Marked Marianne's post as the solution.
Thanks everyone for all your inputs.
Dave