Restore fails with error 25 : socket write failed

[Balaji_RT]

The restore is failing again and again with the same mediaserver.

The client runs on 6.5 and master is on 6.5.6.  Aug 24, 2011 2:42:44 AM - begin Restore

job details

==========

Aug 24, 2011 2:42:47 AM - media needed: AAxxx

Aug 24, 2011 2:42:47 AM - restoring from image servername_1309494520

Aug 24, 2011 2:42:47 AM - requesting resource @aaaaL

Aug 24, 2011 2:43:07 AM - Error bpbrm (pid=13469) bpcd on servername exited with status 24: socket write failed

Aug 24, 2011 2:43:08 AM - Error bpbrm (pid=13469) client restore EXIT STATUS 24: socket write failed

Aug 24, 2011 2:43:04 AM - granted resource  MediaID=@aaaaL;DiskVolume=/backup_disk/disk01;DiskPool=pronbu24_ad_pool;Path=/backup_disk/disk01;StorageServer=pronbu24;MediaServer=pronbu24

Aug 24, 2011 3:45:08 AM - restored from image servername_1309494520; restore time: 1:02:21

Aug 24, 2011 3:45:08 AM - restoring from image servername_1311233415

Aug 24, 2011 3:45:10 AM - Error bpbrm (pid=15941) bpcd on servername exited with status 24: socket write failed

Aug 24, 2011 3:45:10 AM - Error bpbrm (pid=15941) client restore EXIT STATUS 24: socket write failed

[Marianne]

Check/verify comms between media server and client.

Run the following on media server and client for the other machine:

bpclntcmd -self
bpclntcmd -hn <hostname>
bpclntcmd ip <ip-address>

If any of the above fails, check DNS and/or hosts entries.

Next, test connectivity. Create bpcd log dir on client.

Run the following command on the media server:

bptestbpcd -client <client-name> -verbose

Please post output as well as bpcd log.

[pikachu]

setup logging for bpcd and also run a 'bptestbpcd -client CLIENTNAME -debug -verbose' from the media server

[Balaji_RT]

I ve found the backup is failing only with this media server among 4 UNIX media servers in our environment.Other backups are running fine on all media servers.

All the 3 bpclntcmd commands resolves correctly.

Here comes out for bptestbpcd -client clientname -debug -verbose

04:05:41.415 [20580] <2> bptestbpcd: VERBOSE = 0

04:05:41.484 [20580] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2046: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006

04:05:41.484 [20580] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2060: service: bpdbm

04:05:41.598 [20580] <2> logconnections: BPDBM CONNECT FROM 10.133.113.148.56474 TO 10.133.225.84.13724

04:05:41.967 [20580] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2046: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006

04:05:41.967 [20580] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2060: service: bpcd

04:05:42.008 [20580] <2> logconnections: BPCD CONNECT FROM 10.133.114.27.56475 TO 10.133.114.127.13724

04:05:42.010 [20580] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.180: msg: VNETD CONNECT FROM 10.133.114.27.56476 TO 10.133.114.127.13724 fd = 8

04:05:42.023 [20580] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.533: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

04:05:42.067 [20580] <2> vnet_vnetd_connect_forward_socket_begin: vnet_vnetd.c.550: ipc_string: /tmp/vnet-13593314615942064263000000000-PryAab

04:05:42.070 [20580] <2> put_long: (11) network write() error: Broken pipe (32); socket = 7

04:05:42.070 [20580] <2> bpcr_put_vnetd_forward_socket: put_string /tmp/vnet-13593314615942064263000000000-PryAab failed: 32

04:05:42.070 [20580] <2> local_bpcr_connect: bpcr_put_vnetd_forward_socket failed: 24

04:05:42.070 [20580] <2> ConnectToBPCD: bpcd_connect_and_verify(devesp014, devesp014) failed: 24

<16>bptestbpcd main: Function ConnectToBPCD(devesp014) failed: 24

04:05:42.071 [20580] <16> bptestbpcd main: Function ConnectToBPCD(devesp014) failed: 24

<2>bptestbpcd: socket write failed

04:05:42.071 [20580] <2> bptestbpcd: socket write failed

<2>bptestbpcd: EXIT status = 24

04:05:42.072 [20580] <2> bptestbpcd: EXIT status = 24

socket write failed

bpcd log from Client :

04:05:42.017 [13592] <2> ReadKeyfile: keyfile /usr/openv/netbackup/keyfile does not exist

04:05:42.017 [13592] <2> bpcd main: ReadKeyfile failed

04:05:42.017 [13592] <2> bpcd main: offset to GMT 28800

04:05:42.018 [13592] <2> logconnections: BPCD ACCEPT FROM 10.133.114.27.56475 TO 10.133.114.127.13724

04:05:42.018 [13592] <2> bpcd main: setup_sockopts complete

04:05:42.019 [13592] <2> vauth_acceptor: vauth_comm.c.337: no methods for address: no authentication required

04:05:42.019 [13592] <2> bpcd main: no authentication required

04:05:42.021 [13592] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)

04:05:42.021 [13592] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1

04:05:42.021 [13592] <16> bpcd main: Couldn't get peer hostname

[Dipendra_Singh]

If the restore is failing for the client only with a particular media server then you need to check the network connectivity and any firewall rules set on this media server for communiction with this client subnet. I am sure the backup might also be failing, when tried to run using the same media server.

[pikachu]

gethostbyaddr is reverse lookup

gethostbyname is fwd lookup.

setup a hosts file on the client

[Marianne]

Hopefully you have managed to resolve the reverse lookup problem by now and successfully restored?

As per pikachu's post:

Client is receiving connection request from IP address "10.133.114.27". This should be the media server, right?

Client now needs to resolve IP address to a valid hostname that appears in its SERVER list. Client is unable to do so:
gethostbyaddr failed : HOST_NOT_FOUND (1)

Only when hostname lookup and valid SERVER entry is confirmed, will client accept connection and connect back.
 

[Tmy_70]

 
      Commands you should use for everyday trouble shooting.
1.  /usr/openv/netbackup/bin/bpclntcmd -hn client
            (This command is extremely helpful, it tells you if Netbackup can do a successful lookup of the dns name)
2. bpclntcmd -ip ip_address
            (This command is also very helpfuls, it tells you if Netbackup can do a reverse dns lookup)
 

Another Commnad:

telnet <nombre_media> bpcd
bpclntcmd -self
bpclntcmd -hn <hostname>
bpclntcmd -server <NBU master>
bpclntcmd -ip <ipaddress>