How to configure LLT over UDP when you have more than 1 node at each site

[mikebounds]

The VCS install guide stays to set the broadcast address correctly for direct-attached (non-routed) links and for the links that cross an IP router, disable broadcast feature.  This doesn't make any sense for "good" clustering when you have more than 1 node at each site and all examples in install guide are for 2-node clusters.  Where you have more than 1 node at each site, then for every heartbeat link, nodes in same site (same subnet) will be directly attached, so install guide says set Broadcast, but nodes across sites (in different subnets) cross an IP router, so install guide says disable broadcasts.  I am guessing you have to disable broadcasts for all links - can someone confirm?

Example suppose you have nodes A1 & A2 at site A and node B1 at site B with heartbeats:

A1 HB1: 10.8.5.1
A1 HB2: 10.8.6.1

A2 HB1: 10.8.5.2
A2 HB2: 10.8.6.2

B1 HB1: 10.9.5.1
B1 HB2: 10.9.6.1

So for node A1, it has a direct HB to A2 (in same subnet) and a routed heartbeat (in different subnet) to B1, so I think you need to disable Broadcasts.

I also don't understand statement on VCS install guide which says:

If the LLT private links are not on different physical networks, then make sure
that the links are on separate subnets

I thought LLT links MUST be on separate physical links, otherwise a failure of a physical link could cause 2 LLT links to go down which could cause split brain.  Therefore all my LLT links ARE on different physical networks so this statement implies I do NOT need to make sure that the links are on separate subnets, but in my example, if I had used the same subnet for HB1 and HB2, then how would I ensure the host chose the right interface (I could specify interface on route between site A and B, but there is not route between A1 and A2 and therefore A1 host might use HB2 interface to reach HB1 on A2 (which would fail as HB1 and HB2 are physically separate)

Lastly, I assume you cannot mix LLT over UDP with LLT over Ethernet, so in my example, I presume you have to use exclusively LLT over UDP.

Thanks

Mike

[Adhiraj_Joshi · Accepted solution]

1. Yes, we will have to disable broadcast for all the links.

2. For LLT over UDP, we can use same physical network for different links but we recommend to use different physical networks to avoid a single point of failure.

According to me, the links must be on different subnets too, no matter the links are using same physical network or different. I will get back to you on this part in the VCS install guide.

3. Yes, we do not support using mixing of LLT over UDP and LLT over Ethernet.  In your setup, wou will have to use LLT over UDP only.

[mikebounds]

Thanks for quick reply.  I think regarding subnets, that VCS install guide should recommend using different subnets as this makes life easier, but whether you can use the same subnet for heartbeats is a networking question really, as for example I have a customer who is using the same subnet for each heartbeat in a 2-node cluster as each node is in a different subnet, so you can specify the interface when adding the route between 2 nodes, so for example you can add routes like

route add 10.8.5.1 10.9.5.254 -ifp interface_for_HB1

route add 10.8.5.3 10.9.5.254 -ifp interface_for_HB2

to force traffic for IPs on the same subnet down specific interfaces, but this only works if nodes are in different subnets.  Of course this is not ideal as the gateway becomes a single point of failure.

Mike