We have started to receive the following warnings and messages about every 30 minutes or so.
Sep 29 10:45:27 node1 kernel: LLT INFO V-14-1-10205 link 0 (eth1) node 1 in trouble
Sep 29 10:45:27 node1 kernel: LLT INFO V-14-1-10205 link 1 (eth2) node 1 in trouble
Sep 29 10:45:29 node1 kernel: LLT INFO V-14-1-10205 link 2 (eth0) node 1 in trouble
Sep 29 10:45:30 node1 kernel: LLT INFO V-14-1-10499 recvarpreq link 1 for node 1 addr change from 00:50:56:A9:30:0E to 00:50:56:A9:30:0F
Sep 29 10:45:30 node1 kernel: LLT WARNING V-14-1-10498 recvarpreq cross links? links 0 and 1 saw the same peer link number 1 for node 1
Sep 29 10:45:30 node1 kernel: LLT INFO V-14-1-10499 recvarpreq link 0 for node 1 addr change from 00:50:56:A9:30:0F to 00:50:56:A9:30:0E
Sep 29 10:45:30 node1 kernel: LLT INFO V-14-1-10024 link 1 (eth2) node 1 active
Sep 29 10:45:30 node1 kernel: LLT INFO V-14-1-10024 link 0 (eth1) node 1 active
Sep 29 10:45:31 node1 kernel: LLT INFO V-14-1-10024 link 2 (eth0) node 1 active
Description of environment:
Four nodes running on an ESX each
Two heartbeats on eth1 and eth2
Low-Pri Heartbeat on eth0
Each heartbeat is connected to a distributed vSwitch and is then routed through the management net of the VMware environment.
[root@node1 ~]# cat /etc/llttab
set-node node1
set-cluster 1
link eth1 eth-00:50:56:A9:30:0E - ether - -
link eth2 eth-00:50:56:A9:30:0F - ether - -
link-lowpri eth0 eth-00:50:56:A9:30:0D - ether - -
[root@node1 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:A9:30:0D
inet addr:xx.xx.xx.xx Bcast:xx.xx.xx.xx Mask:255.255.255.128
inet6 addr: fe80::250:56ff:fea9:300d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11414328 errors:0 dropped:0 overruns:0 frame:0
TX packets:14069031 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3814234814 (3.5 GiB) TX bytes:11927457378 (11.1 GiB)
eth0:0 Link encap:Ethernet HWaddr 00:50:56:A9:30:0D
inet addr:xx.xx.xx.xx Bcast:xx.xx.xx.xx Mask:255.255.255.128
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 00:50:56:A9:30:0E
inet6 addr: fe80::250:56ff:fea9:300e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:52761246 errors:0 dropped:0 overruns:0 frame:0
TX packets:22113328 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4128460055 (3.8 GiB) TX bytes:1701202284 (1.5 GiB)
eth2 Link encap:Ethernet HWaddr 00:50:56:A9:30:0F
inet6 addr: fe80::250:56ff:fea9:300f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:62670760 errors:0 dropped:0 overruns:0 frame:0
TX packets:22112578 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4936739054 (4.5 GiB) TX bytes:1411498489 (1.3 GiB)
The environment has been productive for four days. These messages have started today. During the test phase we did not encounter these warnings. The setup was not changed in the last few days, so it is a little puzzling why this should turn up all of a sudden.
I suppose because the heartbeats are routed through the level 3 network, the interfaces an now see each other and receive all of the heartbeats, thus causing these messages. Is there a configuration in VCS to handle this or do I have to get the VMware guys to change their setup?
