bpclntcmd returns error code 25

[kbrandt]

I changed the IP of the master server and some of the backups have stopped working.

When I run from the problem clients:
/usr/openv/netbackup/bin/bpclntcmd -pn
It returns nothing, and the error code is 25.

A problem is that I have an old domain that some of the machines are on.  On some, the bp.conf points to the old domain, and for some the new domain.  I created a reverse entry for the new ip of the master server and it points to the new domain.  I tried updating the bp.conf on the client with the new domain but it still doesn't work.  Am I going about this the wrong way, is there a command I need to run after updating the bp.conf?

[Andy_Welburn]

by hostname as well as IP?

Have you any local hosts entries on the client pointing to the old IP?

[Marianne]

Check on the clients if there are local /etc/hosts entries. Hosts entries are normally looked at 1st, then DNS.
Do bpclntcmd -hn <master> from the clients to see what IP address the client is using to connect to the master.
No need to update bp.conf if only IP address for master is changed - the name remained the same, right?
Therefore no change is required to SERVER entries.

If I may ask - what did you change in bp.conf?

[kbrandt]

I can't ping because ICMP is filtered for these machines.  I guess maybe pn never worked before, what command can I test these after updaing any hosts file from the master server?

[kbrandt]

I changed the bp.conf file because it points to an old domain, so I thought maybe the reverse lookup of the ip needs to match the FQDN?

[Marianne]

bpclntcmd -pn is trying to connect to bprd (via vnetd) on the master. It should work, even through firewalls.
Have you tried bpclntcmd -hn <master> yet?
Have you checked for local /etc/hosts entries on the clients?

[kbrandt]

I have checked the hosts files and they seem okay.  -hn and -ip seem to work okay form the client.  I should also add I get errors 23,24,58,59 when I try to run the backup from the master.  Is there a command I can run to test this that will be faster than running the backup (since that has to mount a tape etc).

[kbrandt]

Run from the Master (If this Helps:)

[root@backup1 admincmd]$ ./bptestbpcd -client dmzvm1 -verbose -debug
10:10:27.558 [13648] <2> read_client: ?
10:10:27.558 [13648] <2> read_client: cannot access() dmzvm1: No such file or directory (2)
10:10:27.558 [13648] <2> ConnectToBPCD: db_getCLIENT(dmzvm1) failed: 227
10:10:27.565 [13648] <2> nbconf_get_info: nbconf_glue.cpp.237: NBCONF_LIB: /usr/openv/lib/libVnbconf.so
10:10:27.565 [13648] <2> vauth_get_user_name: vauth_comm.c.726: cached_user_name: root
10:10:27.565 [13648] <2> vauth_get_user_name: vauth_comm.c.728: cached_user_name: :root
10:10:27.565 [13648] <2> vauth_get_user_name: vauth_comm.c.734: user_name: :root
10:10:27.565 [13648] <2> bpcr_connect: bpcr.c.341: connect_opts = 0x01000100 connect_opts2 = 0x01000100
10:10:27.565 [13648] <2> bpcr_connect: bpcr.c.348: connect_opts = 0x01000100
10:10:27.565 [13648] <2> bpcr_connect: bpcr.c.387: daemon_port_type = 0
10:10:27.567 [13648] <2> vnet_async_connect: vnet_vnetd.c.3970: connect in progress: 0 0x00000000
10:10:37.567 [13648] <2> vnet_async_connect: vnet_vnetd.c.3970: connect in progress: 1 0x00000001
10:13:36.567 [13648] <2> vnet_async_connect: vnet_vnetd.c.4161: getsockopt SO_ERROR returned: 110 0x0000006e
10:13:46.567 [13648] <2> vnet_async_connect: vnet_vnetd.c.4161: getsockopt SO_ERROR returned: 110 0x0000006e
10:13:47.570 [13648] <2> vnet_async_connect: vnet_vnetd.c.3970: connect in progress: 0 0x00000000
10:13:57.570 [13648] <2> vnet_async_connect: vnet_vnetd.c.3970: connect in progress: 1 0x00000001
10:15:29.572 [13648] <2> vnet_async_connect: vnet_vnetd.c.3750: ran out of time before connect: 302 0x0000012e
10:15:29.572 [13648] <2> vnet_async_connect: vnet_vnetd.c.3750: ran out of time before connect: 302 0x0000012e
10:15:29.572 [13648] <2> vnet_connect_to_service_or_vnetd: vnet_vnetd.c.3520: vnet_async_connect failed: 18 0x00000012
10:15:29.572 [13648] <2> nb_connect_to_vnetd_or_legacy: comm.c.2052: vnet_connect_to_vnetd_or_service failed: 18
10:15:29.572 [13648] <2> bpcr_connect: nb_connect_to_vnetd_service(dmzvm1) failed: 25
10:15:29.572 [13648] <2> bpcr_connect: Can't connect to client dmzvm1
10:15:29.572 [13648] <2> ConnectToBPCD: bpcd_connect_and_verify(dmzvm1, dmzvm1) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(dmzvm1) failed: 25
10:15:29.572 [13648] <16> bptestbpcd main: Function ConnectToBPCD(dmzvm1) failed: 25
<2>bptestbpcd: cannot connect on socket
10:15:29.572 [13648] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
10:15:29.572 [13648] <2> bptestbpcd: EXIT status = 25
cannot connect on socket

I am able to ping this client from the master server (ICMP is only filtered from client to master, not the other way around).

Update:
This particular one was some iptables rules.... will see if I can make progress with the others using the command that checks it from the master.

[Andy_Welburn]

Were these configured to allow NB communication for specific master IP address?

[kbrandt]

Andy, yes the rules were configured to only allow the IP.

I have gone through all the client now and fixed all but one of them.  Most of them I either had to update the Server list (for hte windows machines) or add a iptables rule.

However, I still have one Linux box that has no iptables rules, resolving seems to work fine, and I get Socket Error 25 when using ./bptestbpcd command.  Any ideas?

[Andy_Welburn]

netstat -a | egrep "bpcd|vnetd"

No SELinux "conflict"?

But then it was the Master you changed & not the client!

[kbrandt]

netstat -a | egrep "bpcd|vnetd"
tcp        0      0 *:bpcd                      *:*                         LISTEN      
tcp        0      0 *:vnetd                     *:*                         LISTEN

and no SELinux.

[Andy_Welburn]

services running.
name resolution ok
hosts files updated
bp.conf correct
iptables correct

No stupid typos anywhere?

Masters fully qualified & short name both in hosts file on client? I'm sure I read somewhere about the order these were in the hosts file (probably something totally unrelated to your issue) so maybe worth swapping them (just for a laugh!) e.g. IP....SHORT....FQHN or IP.....FQHN....SHORT.

Sorry, just clutching at straws now.

[kbrandt · Accepted solution]

The one remaining server had hosts.allow entries :\  Thanks everyone for the help!

[Andy_Welburn]

Thanks for the update - one to add to the list of things to check.