Netbackup client can't connect on backup interface

[mikebounds]

We have a Solaris NBU master, and Solaris and Linux clients, each of which have a NIC on the public network and a NIC on the  backup network,

The Solaris and linux clients are all configured with a bp.conf of:

SERVER = NBU master backup NIC

CLIENT = client backup NIC

and policy has been setup (one for each O/S) with the client using the backup NIC and Solaris clients work, but Linux does not as the host will not connect

If we change bp.conf to use public NIC for SERVER  and CLIENT  and change client in policy to use public NIC, then Linux will connect.

I have check all the obvious stuff between NBU master and Linux client:

• Both hosts can ping each other on backup interface using hostname
• Both hosts can telnet to each other on ports 1556 and 13782

Any ideas what else to check?

The NBU master is 7.5.0.6 on Solaris 10

The Solaris client is 7.5 on Solaris 10

The Linux client is 7.0.1 on Redhat 6.4

Mike

[mikebounds]

I thought I had found issue, but it turns out not:

I read the "-i" flag of traceroute which says

 -i interface
              Specifies the interface  through  which  traceroute  should  send  packets.  By
              default, the interface is selected according to the routing table.

to mean that

traceroute -i igb0 sol_client_bck

would send packets to sol_client_bck through the igb0 interface, but this is not what it does - it still uses the bck interface (igbe1), but marks the packet as being sent from the igbe0 interface - i.e if you snoop ibge0 you will see nothing being sent to sol_client_bck, but if you snoop the igbe1 bck interface you see packet being sent out as:

nbu_mast_vip -> sol_client_bck

as oppose to normally (without -i flag) you would see the packet sent out, originated from bck interface:

nbu_mast_vip_bck -> sol_client_bck

So I do not have a link on the network (a route on a switch in the network) between public and backup network for Solaris and so if I down bck interface on my Solaris client, then the client cannot reach the bck network from the public network.  Below is what happens with "traceroute -i igb0 sol_client_bck" ran from NBU master:

Packet is sent from bck interface on master as orignating from public interface and is received by the client on bck interface

As the packet is sent as orignating from public interface, it is sent back on the public interface and this works where the client is Solaris, but not for Linux.

The NBU master seems to do the same thing - i.e a snoop on the NBU master when I initiate a connection from Netbackup GUI to host client shows that packet is sent out of bck interface but as originating from the public interface, exactly as above for traceroute, and so both Solaris and Linux clients receive this packet on their bck interface, but only Solaris sends it back (via the public interface).  So Linux clients do not even attempt to send the packet out, as oppose to trying to send a packet out which does not get to its destination (back to the NBU master).

I think Marianne is closest mentioning /etc/notrouter as the situation I am seeing is like IP forwarding in that the Solaris hosts appear to be forwarding the IP from the bck interface to the public interface, except it is actually a "reply" rather than a forward via a different interface.  IP forwarding is NOT enabled on the Solaris hosts and I tried enabling on Linux as a test and this didn't work either.

So I am still at a loss at what to do - any ideas?

Mike

[Genericus]

I had this problem on my old EMC EDL.

Turn off MDNS.

modify /etc/host.conf as follows:

#
# /etc/host.conf - resolver configuration file
#
# Please read the manual page host.conf(5) for more information.
#
#
# The following option is only used by binaries linked against
# libc4 or libc5. This line should be in sync with the "hosts"
# option in /etc/nsswitch.conf.
#
order bind, hosts
#
# The following options are used by the resolver library:
#
multi on

# Add line below for multiple interfaces!

mdns off

[mikebounds]

For Linux we are using RHEL 6.4 and mdns is not valid in /etc/host.conf.  Looks like mdns is now configured using avahi-daemon, but we do not have the avahi package installed so the avahi-daemon service is not available so I assume this means MDNS is off.

Mike

[Marianne]

I am battling to understand why this is not working for you.

Backup network has been working for our customers over many years by simply adding /etc/hosts entries and using the backup NIC hostnames in NBU config...

For example:

# Public network
192.168.10.1  master
192.168.10.2  media1
192.168.10.3  client1
192.168.10.4  client2

# Backup network
10.10.10.1 master-bk
10.10.10.2 media-bk
10.10.10.3 client1-bk
10.10.10.4 client2-bk

Update hosts entries on all servers and clients.
Add bp.conf to add SERVER entries for -bk servers on all server and clients. 
If master server was installed with Public network address, you need to keep that name as 1st entry.

Change CLIENT_NAME in bp.conf on all clients to -bk name.

Change all policies to -bk client names.

WAIT... Just remembered something:

Unfortunately the TN has been removed, but I found a forum post where I posted this extract:

https://www-secure.symantec.com/connect/forums/after-upgrading-71-media-hosts-becomes-offline#commen...

Probably worth checking nsswitch.conf.

[mikebounds]

I tried adding to nsswitch.conf:

ipdnoes      files dns

But this made no difference.

Up until now, I have only being looking at the client configuration as I figured NBU master was ok as we have 100+ Solaris clients working fine on bck network, and I had originally thought (as I am sure I read it somewhere) that NBU only used first SERVER entry and ignores others, but I have looked in bp.conf on master and it has no entry for bck.

The master server was installed with the public address, so as suggested by Marianne above I will try adding

SERVER = nbu_master_bck

to bp.conf on NBU master tomorrow as the second entry (as we have a planned outage for something else in which we are restarting NBU master).

I am hoping this mean that the network packet that is sent from the NBU master will have a orginating address as nbu_master_bck, rather than nbu_master, so that the linux clients will reply back on the bck interface, but still not sure if NBU master will send out from 1st SERVER entry in it's bp.conf. 

Fingers crossed for tomorrow.

Mike

[RiaanBadenhorst]

Bit late but here are my two cents. Marianne's suggestion above would probably resolve it. My thoughts on the topic in general is that I don't like backup networks, but if you're going to implement one, then make sure the master (and media servers) is only part of the backup network. Its role after all, is to do backups, so why have it in another network as well? The backup network is created to segragate the traffic due to load/function.

If its windows, have the client create a dns zone just for backup (backup.company.com), then add all the client's 2nd IP's to that. If its NIX, stick it in the host files.

If you do that there is no confusion about which route to pick when communicating with the master, there is only one route.

[mikebounds]

The change to the NBU master bp,conf made no difference, so now the first 2 lines of the bp.conf on the master are:

SERVER = nbu_master_vip
SERVER = nbu_master_vip_bck 

But when I connect to lnx_client_bck in the Java GUI, then NBU still uses nbu_master_vip as the source IP in the connection and NOT nbu_master_vip_bck, or nbu_master_bck.

So I know what the issue is, as if ANY network connection uses a source IP which is not in the same network as the interface the network packet leaves from, then it will not work in our Linux environment.
So I have 2 plans of attack:

Plan 1: Configure NBU to use the "normal" source IP.

For network tools I have tested - ping/telnet/ftp/ssh/traceroute (with standard flags), then if I ping/telnet/ftp/ssh/traceroute to lnx_client_bck from the NBU master, then the source IP is always nbu_master_bck as nbu_master_bck is the first IP on the backup (igb1) interface and this is the interface that is used to send the packets, and if I ping/telnet/ftp/ssh/traceroute to lnx_client (non bck), then the source IP is always nbu_master (non bck) as nbu_master is the first IP on the public (igb0) interface and this is the interlace that is used to send the packets.  

NBU uses a fixed source IP - nbu_master_vip, so even when the packet is sent out of the interface that does NOT have nbu_master_vip on it, it still uses nbu_master_vip as the source IP.
So can NBU be configured to use standard networking so that it chooses the correct source IP?

Plan 2: Configure Linux to be able to send network packets back from a different interface from which they were received.  

This works on Solaris 10, but I am not sure if this is possible on Linux and if it possible, I don't know if there is something we have configured as non-default on Solaris which makes it work, or configured as non-default on Linux which makes it not work.  Could people test a traceroute from one server to another using the following:

traceroute -i  interface_on_network_1  IP_on_network2

example:

traceroute -i  eth0  host2_eth1

and report results and whether testing on Linux or Solaris (and which O/S version)

On my systems, the above "traceroute -i" works from Solaris 10 to Solaris 10, but does not work from RHEL 6.4 to RHEL 6.4 (also does not work from Solaris to RHEL or RHEL to Solaris - i.e if RHEL is involved then it doesn't work).

Thanks

Mike