Backup always failing with socket error

[BharadwajR]

Backup is always getting failed for one client with socket error even after reInstalling Netbackup client and rebooting Server.

From master server,we are able to do resolve ip and host name with bpcIntcmd command.

We are able to telnet of the client with bpcd ,PBX and other ports from master server.

But bptestbpcd  is always getting socket error.

In client bpcd logs it is showing 

18:11:41.147 [17672.18124] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 0

18:11:41.147 [17672.18124] <16> process_requests: Couldn't get peer hostname

18:14:48.363 [15240.16364] <2> setup_debug_log: switched debug log file for bpcd

18:14:48.363 [15240.16364] <2> bpcd main: VERBOSE = 0

18:14:48.363 [15240.16364] <2> logparams: C:\Program Files\veritas\Netbackup\bin\BPCD.EXE

18:14:48.363 [15240.16364] <2> bpcd main: Got socket for input 540

18:14:48.363 [15240.16364] <2> process_requests: offset to GMT -19800

18:14:48.363 [15240.16364] <2> logconnections: BPCD ACCEPT FROM 10.10.3.22.4087 TO 10.10.3.22.13724

18:14:48.363 [15240.16364] <2> process_requests: setup_sockopts complete

18:14:50.613 [15240.16364] <2> bpcd peer_hostname: gethostbyaddr failed:The requested name is valid, but no data of the requested type was found.

It seems like Host name issue.

But Every config setting is fine in client and master server.

EXCEPT one problem.

We were able to telnet Master Server IP from client.

But we are unable to take telnet of master server Host name from client.

In HKEY_LOCAL_MACHINE files,Host files every thing is fine.

But still getting the socket error.

Please help.

[sdo]

On master:

bptestnetconn -v -a -s

...and do these twice, to see if you have dual revolving names in DNS:

nslookup client-name

nslookup client-name

...and these reverse IP lookup of the IP of the client at least twice too, to double check for dual reverse entries:

nslookup c.c.c.c

nslookup c.c.c.c

...and then try:

bpclntcmd -self

bpclntcmd -hn client-name

bpclntcmd -ip c.c.c.c

bptestbpcd -verbose -client client-name

bpgetconfig -g client-name

bpgetconfig -M client-name SERVER

...and then on client, same commands - but the other way, but with a few extra bpclntcmd commands:

bptestnetconn -v -a -s

nslookup master-name

nslookup master-name

nslookup m.m.m.m

nslookup m.m.m.m

bpclntcmd -self

bpclntcmd -sv

bpclntcmd -pn

bpclntcmd -hn master-name

bpclntcmd -ip m.m.m.m

bptestbpcd -verbose -client master-name

(the commands without folder path prefix above assume that you have 'PATH' on either Windows or *n*x defined to the various NetBackup binaries folders - let me know if not).

[BharadwajR]

Thank you for prompt response.

From master server, i got error for this command

bash-3.00# ./bpgetconfig -M d265xx SERVER
socket read failed

In client i am unable to find bpcIntcmd command.For bptestnetconn, i got below error.

bptestnetconn -v -a -s

SERVER = P20001

Forward lookup times:

host: P20001[0.0.0.0]: 2250ms

Reverse lookup times:

host: P20001[0.0.0.0]: unknown host: 0ms

Total elapsed time : 4515 msec

[sdo]

It's strange that you don't have bpclntcmd on the client.

On my test Windows 2008 R2 SP1 machine with NetBackup Client v7.6.0.2 (installed from binaries kit downloaded from an appliance), I can see:

 Directory of C:\Program Files\Veritas\NetBackup\bin
04/24/2014  04:46 PM            60,792 bpclntcmd.exe

You say that you've re-installed the client.  If bpclntcmd really is missing, then this makes we want to ask what else could be missing.

How about checking this:

1) On the problem client, total up the number of files in the binaries tree:

cd /d "C:\Program Files\Veritas\NetBackup\bin"
dir /s

2) Do the same on a another plain client of the same O/S and same client version.

Are the numbers wildly different?

Your bptestnetconn response times of 2 and 4 seconds for DNS lookups is quite poor.  Just from those I would check the DNS config on the problem client.

Just wondering if there's a reason you didn't do all the commands I listed?  There is a reason for doing them in that order, as they progressively expose communications issues.

[BharadwajR]

Sorry for typo.

I have checked all the instructed commands on master server(Sun OS). It seems fine except for bpgetconfig file.

In client server(WINDOWS 2003),i compared the results of instructed commands with similiar client server.

1st image is command results of Error server.

2nd image is the command results of Similar client server. Thank you for support.

[Marianne]

Add hosts entry for the server on this client.

Or check DNS settings on the client.

[NBU35]

it looks like Name resolution issue.

Try to add all the interfaces name & IP of client in to master's hosts file, do same for master server.

[Marianne]

You have pretty much identified the problem when you said in your opening post: 

It seems like Host name issue.

Have you ever tried to ping the master server from the client using hostname?

Your master server is Solaris and probably not added to DNS.
This means that you have to add hosts entries on clients for the server.

So, navigate to C:\Windows\System32\drivers\etc on the client in Windows Explorer, open hosts file with Notepad, add entry for master:

10.10.4.15  T20000

Save the file. 
Take care that Notepad does add .txt to the file.

Ensure that T20000 is the 1st Server entry in Client's NetBackup registry setting.

Test 'ping T20000' from the client and run bpclntcmd commands as previously.

Lookup should now be fine and connectivity from the master as well...... 

[BharadwajR]

I have checked the drivers\etc\hosts and Netbackup registry setting.

Master server entry is properly done.

I have again re-Installed client also .

Still getting same error.

We are unable to ping / telnet /nslookup Master server Hostname from client server.(With Master server IP we can ping,telnet)

Since it is server name resolution issue,I will contact Windows team for this issue.

[Marianne · Accepted solution]

We are unable to ping / telnet /nslookup Master server Hostname from client server

Please double-check that Notepad did not add .txt to the hosts filename. (Extensions are normally hidden for known file types.)

If you are unable to ping, telnet, nslookup, you are right - this is OS issue. Your Windows team need to resolve.

One more thing:
I had a similar issue when I added hosts file entry on my own laptop for our in-house demo appliance.
The anti-virus software on my laptop prevented hosts file to be edited and hosts file was saved as hosts.txt.
I had to temporarily disable AV in order to rename hosts file.

Something else to check when OS lookup is sorted out:

Ensure that Windows firewall is turned off on the client.

[maurijo]

This is hosts file for sure if the master is Solaris. Like Marianne explained they are not added in DNS.

Most of my connection problems with Solaris masters are also related to IPv6. Sometimes it communicates using ipv6 and the Windows servers have this disabled (not by default but most Windows engineers I know disable it on setup).

[Mark_Solutions]

What is also worth a test is whether the client can actually resolve its own name ... short and FQDN  Its is worth also doing all of your tests with both the short and FQDN.

If the client only has one IP address then put itself into its own hosts file as well as the Masters hosts file  .. but both using short and FQDN as it may just be that DNS is missing one of those

Hope this helps

[V4]

post updating correct host entries clear DNS cache on master and affected client

ipconfig /flushdns

ipconfig /registerdns

then try with bpclntcmd from client.

Also from master /bin/admincmd/bptestbpcd -client <clientname> -debug -verbose

You can also repeat above after re-installing netbackup client if commands are missing.... (Download binaries from fileconnect ... it will be huge approx.1-2GB complete package of server, Remote admin console,search server and clients.)

[backupguy2014]

If you use uppercase or camelcase names in your hosts file, you may want to have an alias that is strictly lowercase, as in:

dotted.ip.address \t fqdn \t shortname \t alias \t alias

so:

10.10.1.45       T20000.some.domain     T20000      t20000

[V4]

Linux and Unix are known for case sensitivity.. windows is exempted... 

[Marianne]

I know this one keeps on popping up under 'Can you solve these?', but you will notice that this user has not been back to look for answers or mark/share solution.

[BharadwajR]

We have moved the application to new server and taking up the backup.

Sorry for late reply.

Thank you all for sharing valuable knowledge.