Backup issues at a new location, new media server

[JHeaton]

Master server - Win2k8R2

Media server - Win2k8R2

client - Win2k8R2

NBU - 7.1

I have been installing new media servers the past few days.  All of them have been running just fine.  Last night I setup another one, Eurbkup1.  I followed the same exact steps as all the others, but when I went to do a test job, it failed with a Status 37.  After doing some troubleshooting, I ended up adding hosts records for the media server and client in the master server; the master server and the client in the media server; and the master server and media server in the client.  So now, bpclntcmd -ip works from all sites for all other sites.  However, now my backup jobs are failing with a Status 58, can't connect.  All my past experience has been to check the host records when this happens, but as I've said, all that works.  I'm going to reboot all boxes in question again in a minute, but I wanted to post this to see if anyone had any other ideas for me to pursue.

As always, any help is greatly appreciated.

Thanks,

Joe

[Nicolai]

Has Eurbkup1  more that one active network infterface - ?

Is both inbout and outbound Windows firewall opned for 1556/tcp ?

[JHeaton]

No, Eurbkup1 only have one NIC.

I'm thinking I resolved the SC 37 when I added all the HOSTS records.

Firewall - I have the following rules set up, as the Windows Firewall is a very frustrating thing to work with.

On the Media server:

1)  Any traffic coming from the master server's IP is allowed.

2)  TCP ports 13782 and 13720 are allowed from any IP.

On the client computer:

1)  Any traffic coming from either the master or media server is allowed.

I've never had to explicitly allow port 1556 on any of my machines.  I have 11 media servers now, and this is the only one I'm having issues with.  The only reason I did the #1 rules above was because I was having problems getting the master to be able to find the attached libraries with just the #2 rules.  All my other boxes only have the #2 rule.

[Nicolai]

Port 1556 is default port in NBU 7.x. You should use this port all they way.

A 3*1 second delay is introduced whenever Netbackup timeout a connection on 1556 and does fallback to the old style 13782/13724. 

[revarooo]

100% agree, open up 1556 too.

IF it still fails after that, turn off the firewall temporarily, does it work then?

[JHeaton]

Open 1556 between what?  I already have a rule allowing any traffic from the master server to the media server, and from master/media to the client.

[RonCaplinger]

You stated that "bpclntcmd -ip" is working between the servers.  What about "bpclntcmd -hn"?  Do the names you entered in the "hosts" files exactly match (FQDN versus short name) the names in the "Client Name" fields on all of the servers and clients?  And do all of the clients have the correct NBU master/media servers defined to the "Servers" property?

I've had issues with forward/reverse DNS names for my NBU servers not match (one used FQDN, the other used short names).  I've also forgotten to add the correct master server name on one occaision. 

[revarooo]

On the client, allow inbound connectivity to PBX (port 1556)

[JHeaton]

Ok, but as I have already stated, I have a rule on the client allowing traffic on any port from the master and the media servers.  So, I'm already allowing that port.

[JHeaton]

Yes, the -hn does work also.  I have had many issues in this arena.  Most of the time, as soon as I create/verify HOST records, the issues are resolved, but not in this case.  It's being really weird.  Especially since the box was built exactly the same way as my other 10 media servers.

[revarooo]

Try turning it off.

Then re-run the bptestbpcd -client <clientname> from the media server. Does it still fail?

If so, then run vxlogview command on the client for PBX (see below), do you see the Media servers IP coming in? Is an inbound connection being made?

vxlogview -p 50936 -o 103 -d all -t 00:05:00 > pbx.txt

[JHeaton]

We have NAP in place, so I can't turn the firewall off on the servers.

bptestbpcd -client fails with the following:

<16>bptestbpcd main: Function ConnectToBPCD(eurfile1.domain) failed: 25 cannot connect on socket

vxlogview gave this:

V-1-45 No log files found.

[revarooo]

vxlogview is confirming that nothing is being logged, no rejections from the NetBackup software, therefore something (firewall/mis-configured network) is blocking the connection from being established.

[JHeaton]

If there were misconfigured network, wouldn't the bpclntcmd -ip/-hn fail?  I can't see how it can be the firewall, when I'm allowing all traffic from the master/media to the client.

Thank you for taking all this time to help me, it's very much appreciated.  I think I'm going to have to fall back to my last ditch effort in troubleshooting, and actually open a support ticket.  I was just hoping to get this resolved faster than the support route.  Hopefully I can get someone to call me back sooner than 4 hours...

[JHeaton · Accepted solution]

Turns out I put an incorrect IP address in the firewall rule on the client.  I was allowing all traffic from the master, but the IP for the media server was wrong, so the job wouldn't run.

Thanks again to everyone's help.  It is always very much appreciated.

[revarooo]

bpclntcmd -ip / -hn only tells you what that host thinks is the IP / Hostname. It doesn't communicate to the remote server.

I agree, a support call is probably a good idea at this point. You may want to run: netbackup\bin\support\nbsu -c -t on the media / client so support have a configuration view of your affected servers. 

[revarooo]

Well done finding it, as I mentioned earlier:

 

vxlogview is confirming that nothing is being logged, no rejections from the NetBackup software, therefore something (firewall/mis-configured network) is blocking the connection from being established.