Odd connection to client issue

[rjrumfelt]

Master: 6.5.4 on Solaris 10 Client: 6.5.4 on Solaris 10 No media servers I can telnet to 13724 from the master to the client. I can telnet to 13724 from the client to the master yet every connection that bptestbpcd tries to make fails: master_server:root> ./bptestbpcd -client NBU_client -verbose -debug 11:45:54.005 [28151] <2> bptestbpcd: VERBOSE = 0 11:46:54.020 [28151] <2> vnet_sock_ready: vnet.c.506: Function failed: 11 0x0000000b 11:46:54.020 [28151] <2> vnet_pop_byte: vnet.c.171: Function failed: 11 0x0000000b 11:46:54.020 [28151] <2> vnet_pop_string: vnet.c.266: Function failed: 11 0x0000000b 11:46:54.020 [28151] <2> vnet_pop_signed: vnet.c.310: Function failed: 11 0x0000000b 11:46:54.020 [28151] <2> version_connect: vnet_vnetd.c.1815: Function failed: 11 0x0000000b 11:46:54.020 [28151] <2> do_vnetd_service: vnet_vnetd.c.4407: version_connect failed: 11 0x0000000b 11:46:54.020 [28151] <2> vnet_async_connect: vnet_vnetd.c.4207: do_vnetd_service failed: 11 0x0000000b 11:50:38.780 [28151] <2> vnet_async_connect: vnet_vnetd.c.4231: getsockopt SO_ERROR returned: 145 0x00000091 11:50:59.780 [28151] <2> vnet_async_connect: vnet_vnetd.c.3806: ran out of time before connect: 305 0x00000131 11:50:59.780 [28151] <2> vnet_async_connect: vnet_vnetd.c.3806: ran out of time before connect: 305 0x00000131 11:50:59.781 [28151] <2> vnet_connect_to_service_or_vnetd: vnet_vnetd.c.3565: vnet_async_connect failed: 18 0x00000012 11:50:59.781 [28151] <2> nb_connect_to_vnetd_or_legacy: comm.c.2030: vnet_connect_to_vnetd_or_service failed: 18 11:50:59.781 [28151] <2> local_bpcr_connect: nb_connect_to_vnetd_service(NBU_client) failed: 25 11:50:59.781 [28151] <2> local_bpcr_connect: Can't connect to client NBU_client 11:50:59.781 [28151] <2> ConnectToBPCD: bpcd_connect_and_verify(NBU_client, NBU_client) failed: 25 <16>bptestbpcd main: Function ConnectToBPCD(NBU_client) failed: 25 11:50:59.782 [28151] <16> bptestbpcd main: Function ConnectToBPCD(NBU_client) failed: 25 <2>bptestbpcd: cannot connect on socket 11:50:59.782 [28151] <2> bptestbpcd: cannot connect on socket <2>bptestbpcd: EXIT status = 25 11:50:59.782 [28151] <2> bptestbpcd: EXIT status = 25 cannot connect on socket From the client: bpclntcmd -self returns the following: yp_get_default_domain failed: (6) internal yp server or client error NIS does not seem to be running: (1_ args to yp functions are bad gethostname() returned: NBU_client host NBU_client: NBU_client.ourdomain.local at NBU_IP (hex_version) aliases: bpclntcmd -ip and bpclntcmd -hn return expected output bpclntcmd -pn just hangs for about 5 minutes and then finally returns the expected output. From the master: bpclntcmd -ip and bpclntcmd -hn return expected output What I'm thinking is that this host is just being slow to respond. My question is that if I adjust the client timeout values, do you think that would help me in this situation? The thing that casts doubt on that for me is the output of bptestbpcd. It was pretty quick in returning all of those errors. Thanks

[rjrumfelt]

that I have VERBOSE = 5 in my bp.conf file on the client, and bpcd, bprd, and bpbkar directories created in /usr/openv/netbackup/logs.  Yet for some reason no logs are being generated.

[Will_Restore]

check for the presence of the files /etc/hosts.allow and /etc/hosts.deny.

http://seer.entsupport.symantec.com/docs/301214.htm

[Marianne]

Mmmm - I was about to ask for bpcd log I guess you knew!
What does netstat -a output say - bpcd and vnetd both LISTEN'ing?

[Will_Restore]

In-depth Troubleshooting Guide for Exit Status Code 25 in Veritas NetBackup (tm)

ftp://exftpp.symantec.com/pub/support/products/NetBackup_Enterprise_Server/278569.pdf

[rjrumfelt]

THe hosts.allow and hosts.deny files have been checked and they are configured as they should be.


****Edit**** I also forgot to mention that the overall job failes exit status 58, while bptestbpcd fails 25

[rjrumfelt]

both vnetd and bpcd are listening.

Everything in my head tells me that the backup should be working, but it isnt.

And the fact that I'm not logging is also troubling.

[rjrumfelt]

When I execute bpcd from the client, I get a log generated.  Backups do not generate any logs.

[rjrumfelt]

The customer modified the /etc/nsswitch.conf file to read dns first rather than the host file, and in this environment we aren't using DNS.

I'll update if I find that reverting the /etc/nsswitch.conf entry fixes the problem.

[Will_Restore]

( apologies to w.c. fields )

[Andy_Welburn]

Might've been able to save you a bit of time.

We literally just had the same sort of issue with the same symptoms as yourself - everything seems ok initially but 58 failures & 25's on bptestbpcd, no logs unless manually running bpcd etc etc etc- in our instance there were inconsistencies in DNS - one IP address two hostnames almost as if it could connect out but not connect back. It'd've at least got you looking in the "name resolution" direction!

[rjrumfelt]

We're still waiting on a response as to whether or not we can switch back the nsswitch.conf file back to pre switched switching

[Andy_Welburn]

how come nsswitch.conf was changed anyway to put DNS first when DNS isn't used? Seems a bit, well .....strange.....

Did they populate a resolv.conf too?

[rjrumfelt]

around to be a mail relay server, and for some reason they needed to resolve DNS first.  You got me. 

I know host files are a pain, but for me they are far more reliable than DNS can ever be.  resolv.conf looked to be ok.

[Andy_Welburn]

we had a mixed bag when we started to implement DNS throughout out *NIX environment in keeping with our Wintel kit.

Luckily (?) I've got access to both aspects (DNS & hosts) so can make sure they're correct. Must admit, it gives me less to administer now than when it wasn't implemented!

Now maybe worth considering implementing DNS in toto & get everything reading from the same 'hymn book' (again!)?

I will note tho' that all our nsswitch.conf still read 'files' then 'DNS'

[Christoph_Linde]

Pls open a support call.

In addition to that:

You do not connect correctly to vnetd Port.

11:46:54.020 [28151] <2> do_vnetd_service: vnet_vnetd.c.4407: version_connect failed: 11 0x0000000b
11:46:54.020 [28151] <2> vnet_async_connect: vnet_vnetd.c.4207: do_vnetd_service failed: 11 0x0000000b
11:50:38.780 [28151] <2> vnet_async_connect: vnet_vnetd.c.4231: getsockopt SO_ERROR returned: 145 0x00000091

I takes 3:34 minutes befor bptestbpcd gives up. Something is wrong with your network between and or vnetd.

enable logging of vnetd on the client.

create a log directory for vnetd:

mkdir -p  /usr/openv/netbackup/logs/vnetd

I hope we will finde evidence in it. I think I remember from my mind, that it exits on clients, even it is not mentioned in the troubleshooting guide.

[rjrumfelt]

Unfortunatley, changing back the nsswitch.conf file did not work.

I guess I will take Christoph's recommendation and log a case.  I mean

It just seems like it should be working.  I'm wondering if we reinstall the client if it would fix things.

[rjrumfelt]

have some logs in possesion, I don't guess there is any way to request that the case be assigned to you is there?

[Christoph_Linde]

I am not from support, so I cannot take any cases. But support will help you.

But if you can paste the vnetd log from client for a bptestbpcd try. (at best with the bptestbpcd output) I can take a look. But faster and more reliable is support. I am looking in here every now and then - not really on a regular base.

[Andy_Welburn]

Have you also tried using nslookup (by ip & hostname) bothways & twice (incase of duplications) ?