Veritas Cluster Server Heartbeat link down, jeapordy state..

Plumb test IP addresses on c4 on both nodes and see if you ping from both machines.

Also check /var/adm/messages on both nodes for possible NIC errors.

When I plumped up the interface and assigned it a temporary IP I got the following message. node1 had[5167]: [ID 702911 daemon.notice] VCS ERROR V-16-1-6525 (node2) MultiNICB:MNIC:monitor:Interface configuration is not valid. Check IP addresses, subnet masks, flags, groupnames on all interfaces. Make sure that all the interfaces of only one IP subnet are included under the resource. See Bundled Agents Reference Guide. That did not look good, and I immediately unplumbed it. #netstat -I interface shows traffic input ce4 output input (Total) output packets errs packets errs colls packets errs packets errs colls 37525 0 145600 0 0 3682466 0 4771776 0 0 1 0 0 0 0 12 0 0 0 0 0 0 0 0 0 9 0 0 0 0 1 0 0 0 0 10 0 0 0 0 0 0 0 0 0 9 0 0 0 0 1 0 0 0 0 10 0 0 0 0

Can you try snooping the interfaces to see if there's any traffic?

Also - testing with dlpiping may help:

http://www.symantec.com/business/support/index?page=content&id=TECH19998

Hi Brian,

 

The error was probably seen because you online the interface with the same subnet as the MultiNICB resource uses.

Excerpt from the agent guide:

As one of the requirements of MultiNICB agent is that the MultiNICB resource must include ALL of the interfaces that belong to the same IP subnet.  If some interfaces on the same IP subnet are outside of MultiNICB control,

this can lead to complications in the event of device failures.

If the link is up on both sides you can check llt communication using lltping:

http://sfdoccentral.symantec.com/sf/5.0/solaris/manpages/vcs/lltping_1m.html

Further you can check using lltdump for any incoming LLT packages

http://sfdoccentral.symantec.com/sf/5.0/solaris/manpages/vcs/lltdump_1m.html

 

Both commands reside in /opt/VRTSllt/bin.

In case the link is up on both nodes, but you can't get a connection using lltping or by configuring IP's on the link, you can be certain that the problem is within your network.

 

Thanks,
Dan

 

# kstat -p |grep link_up showed me that the links are up
However when I tried the lltping there was no response.

#lltstat -n
LLT node information:
    Node                 State    Links
   * 0    node1          OPEN        1
     1    node2          OPEN        2

From node 1:
bash-2.05$ ./lltping -c 1 -v
lltping: opening LLT dev: /dev/llt port: 30
lltping: mynodeid: 0
lltping: sending a msg to node 1
lltping: waiting for a msg from node 1
lltping: alarmhandler, exit on alarm
lltping: no response from server on node 1

From node 2:
bash-2.05$ ./lltping -c 0 -v
lltping: opening LLT dev: /dev/llt port: 30
lltping: mynodeid: 1
lltping: sending a msg to node 0
lltping: waiting for a msg from node 0
lltping: alarmhandler, exit on alarm
lltping: no response from server on node 0

I checked on the switches and the connections are on the same VLAN and it is active.
ce6 heartbeats are in VLAN 1 and ce4 heartbeats are in VLAN 6. How can I tell which on is the primary heartbeat and which is the secondary heart beat. I'm clueless as to what to do next.

Try using temporary IPs using a different subnet than existing IPs - I usually use:

1.1.1.1 on node1 and 1.1.1.2 on node 2 using mask 255.255.255.0.

and then ping 1.1.1.1 from node 2 and ping 1.1.1.2 from node 1.

All the tools you have used so far just tell you the NIC is up, so there is an issue with the connection of the 2 NICs.

Mike

 

Seems that way, I plumb the interfaces and tried pinging them but there was no respond. netstat shows that there are some traffic going through in that interface and I have no idea what it is and how to look for it.

netstat shows that there are some traffic going through in that interface and I have no idea what it is and how to look for it.

per earlier suggestion - have you tried looking at the output from snoop? dlpiping?

https://www-secure.symantec.com/connect/forums/veritas-cluster-server-heartbeat-link-down-jeapordy-state#comment-9082981

If ping is not working, then either there is an issue with the connection between the interfaces, or the switches may be blocking ping, so assuming that both ce4 and ce6 both have the same firewall configurations, unplumb the IPs from ce4 and plumb them onto ce6 - if ping works using ce6 then this proves the issue is with the network (this proof is usually needed if you have a separate network team as I have alway found the network team say the issue is the O/S until you can prove otherwise).

The otherthing you can do is to do a traceroute (do this first while IPs are on ce4 and then repeat when IPs are on ce6).  Also if you want IPs on both interfaces at the same time so you don't have to keep unplumbing to compare, then use 1.1.2.1 and 1.1.2.2 (with mask 255.255.255.0) on the second set of interfaces.

Mike

I did. But the moment I started the piping server I got this message on the engineA_log. As this is on the production side I'm very afraid to take chances. I don't want any failover happening or unavailability of resources while doing this. Dev is down for the time it will be some time next month before it gets back up. VCS NOTICE V-16-1-10438 Group group1 has been probed on system node2 VCS NOTICE V-16-1-10442 Initiating auto-start online of group group1 on system node1 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group group1 on all nodes VCS ERROR V-16-10001-6525 (node2) MultiNICB:MNIC:monitor:Interface configuration is not valid. Check IP addresses, subnet masks, flags, groupnames on all interfaces. Make sure that all the interfaces of only one IP subnet are included under the resource. See Bundled Agents Reference Guide