cancel
Showing results for 
Search instead for 
Did you mean: 

IPMP Issue with MultiNICB Agent on VCS 5.0 MP1

supermario
Level 3
Hi, Need help. Had posted an earlier problem with MultiNICB at <> That got settled by adding in a few IP addresses to verify againt. However, recently we have observed a lot of NIC failures on the servers. The current configuration for the server is: MultiNICB MultiNICB_pis ( MpathdCommand = "/usr/lib/inet/in.mpathd -a" Device = { e1000g0 = 0, nxge0 = 1 } NetworkHosts = { "172.21.52.1", "172.21.52.2", "172.21.52.11", "172.21.52.12", "172.21.150.22", "172.21.150.23" } LinkTestRatio = 3 IgnoreLinkStatus = 0 NetworkTimeout = 300 NoBroadcast = 1 DefaultRouter = "172.21.150.22" Failback = 1 ) however, we do detect a lot of failures on the e1000g0 interface (that is the primary interface). The network card by itself has no issue. The current state is: bash-3.00# ifconfig e1000g0 e1000g0: flags=19040843 mtu 1500 index 3 inet 172.21.33.41 netmask ffff0000 broadcast 172.21.255.255 groupname MultiNICB_pis ether 0:15:17:75:51:56 Also, funny to notice that there seems to be no process for mpath running. Any ideas on this would be great. Thanks,
1 ACCEPTED SOLUTION

Accepted Solutions

Gaurav_S
Moderator
Moderator
   VIP    Certified

From your configuration, it looks like you are NOT using solaris IPMP .... however your ifconfig output says that you indeed tried to configure an IPMP group..

groupname MultiNICB_pis   <<<<<<<<<<

So you have to decide whether you want to use IPMP or NO ... If Yes, then you need to update the same in your configuration by putting an entry in multinicb resource like this:

UseMpathd = 1

or else if you don't intend to use mpathd, then no need to configure IPMP groups.... check VCS Bundled agents guide for reference configuration....  you can find guide on:

https://vos.symantec.com/documents


Gaurav


View solution in original post

15 REPLIES 15

supermario
Level 3
One more thing. A simmilar configuration (same settings for MultiNICB) on another cluster have no issues at all - and in.mpathd is running fine. I did try to start the daemon manually - but MultiNICB_A.log reports that the MultiNICB agent kills it.

How do I proceed further?

Regards,

Marianne
Level 6
Partner    VIP    Accredited Certified

Link-based or probe-based IPMP? If link-based, see this TechNote:

http://seer.entsupport.symantec.com/docs/317562.htm

Set ConfigCheck attribute to 0 (default is 1) .

Marianne
Level 6
Partner    VIP    Accredited Certified
One more thing - multipathing daemon is only relevant if you are using Solaris IPMP.
Please confirm whether you are using IPMP or Base mode to manage multiNIC. If IPMP, your resource should look similar to this this:

    MultiNICB mnicb (

              Critical = 0
              UseMpathd = 1
              MpathdCommand = "/usr/lib/inet/in.mpathd"
              Device = { fjgi0, fjgi1 }
              ConfigCheck = 0
  GroupName = ipmp0
              )



Please post contents of /etc/hostname.*


supermario
Level 3

Hostname.* file contents:

/etc/hostname.e1000g0:TSHM4PIS
/etc/hostname.nxge0:TSHM4PIS-P2

/etc/hosts related to the ipmp:
172.21.35.21    TSHM4PIS
172.21.35.24    TSHM4PIS-VIP
172.21.35.22    TSHM4PIS-P2


Regards.


Gaurav_S
Moderator
Moderator
   VIP    Certified

From your configuration, it looks like you are NOT using solaris IPMP .... however your ifconfig output says that you indeed tried to configure an IPMP group..

groupname MultiNICB_pis   <<<<<<<<<<

So you have to decide whether you want to use IPMP or NO ... If Yes, then you need to update the same in your configuration by putting an entry in multinicb resource like this:

UseMpathd = 1

or else if you don't intend to use mpathd, then no need to configure IPMP groups.... check VCS Bundled agents guide for reference configuration....  you can find guide on:

https://vos.symantec.com/documents


Gaurav


g_lee
Level 6
The hostname files indicate the system is not configured to use IPMP

MultiNICB operates in one of two modes: base mode or multipathing (mpathd) mode.

If you are using Base mode (UseMpathd = 0 -- this is the default), you will not see an in.mpathd process running as it does not use Solaris mpathd to control the interfaces.

If you are using Multipathing mode (UseMpathd = 1), the in.mpathd daemon/process does need to be running, and you need to set the MpathdCommand attribute to reflect the correct path/location.

As Marianne and Gaurav both pointed out, if you want to use Solaris in.mpathd, then you will need to set UseMpathd=1

Regardless of which mode you choose, you also need to fix the hostname.* file configuration so the interfaces with MultiNICB - refer to the section "Checklist to ensure the proper operation of MultiNICB" in the Bundled Agents Guide
link to VCS 5.0MP3 Solaris version here: http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vcs_bundled_agents/ch_sol_network_agents57.html
particularly:
• Solaris: Test IP addresses have "nofailover" and "deprecated" flags set at boot time.
• Solaris: /etc/default/mpathd/ has TRACK_INTERFACES_ONLY_WITH_GROUPS=yes.

(see https://vos.symantec.com/documents from Gaurav's post above if you want more background / additional information / full set of requirements to setup IPMultiNICB/MultiNICB)

supermario
Level 3
Hi,
Thanks for all your help. I still dont get it. Kindly have a look at the following:
mpathd file (/etc/default/mpathd)
bash-3.00# cat mpathd
#
#pragma ident   "@(#)mpathd.dfl 1.2     00/07/17 SMI"
#
# Time taken by mpathd to detect a NIC failure in ms. The minimum time
# that can be specified is 100 ms.
#
FAILURE_DETECTION_TIME=10000
#
# Failback is enabled by default. To disable failback turn off this option
#
FAILBACK=yes
#
# By default only interfaces configured as part of multipathing groups
# are tracked. Turn off this option to track all network interfaces
# on the system
#
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes

ifconfig output: (modified so that the whole shows have removed the greater and less then signage)
nxge0: flags=9040843 UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER  mtu 1500 index 6
        inet 172.21.35.22 netmask ffff0000 broadcast 172.21.255.255
        groupname MultiNICB_pis
        ether 0:21:28:6:3f:2e
nxge0:1: flags=1000843 UP,BROADCAST,RUNNING,MULTICAST,IPv4 mtu 1500 index 6
        inet 172.21.35.24 netmask ffff0000 broadcast 172.21.255.255
nxge3: flags=1000803 UP,BROADCAST,MULTICAST,IPv4 mtu 1500 index 7
        inet 192.168.1.14 netmask ffffff00 broadcast 192.168.1.255
        ether 0:21:28:6:3f:31
e1000g0: flags=19040843 UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED mtu 1500 index 8
        inet 172.21.35.21 netmask ffff0000 broadcast 172.21.255.255
        groupname MultiNICB_pis
        ether 0:15:17:74:86:a0


Other machine:
e1000g0: flags=19040843 UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED mtu 1500 index 3
        inet 172.21.33.41 netmask ffff0000 broadcast 172.21.255.255
        groupname MultiNICB_pis
nxge0: flags=9040843 UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER mtu 1500 index 4
        inet 172.21.33.42 netmask ffff0000 broadcast 172.21.255.255
        groupname MultiNICB_pis
nxge0:1: flags=1000843 UP,BROADCAST,RUNNING,MULTICAST,IPv4  mtu 1500 index 4
        inet 172.21.33.43 netmask ffff0000 broadcast 172.21.255.255
nxge0:2: flags=1000843 UP,BROADCAST,RUNNING,MULTICAST,IPv4  mtu 1500 index 4
        inet 172.21.36.88 netmask ffff0000 broadcast 172.21.255.255
nxge0:3: flags=1000843 UP,BROADCAST,RUNNING,MULTICAST,IPv4  mtu 1500 index 4
        inet 172.21.36.86 netmask ffff0000 broadcast 172.21.255.255

As you can see, somehow the interfaces are in an IPMP group MultiNICB_pis all right. The settings I have already pasted earlier.

The Question I have is:
a) How the hell is this working in the current place?
b) Why is another cluster that I have (same VCS and patch level as well as configuration) working fine without any issues? (there in.mpathd is also working somehow)
c) The system error logs (/var/adm/messages) show the following errors on NIC failover:
Jul 21 22:01:43 TSHM4PIS llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (e1000g0) node 0 in trouble
Jul 21 22:01:43 TSHM4PIS llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (e1000g0) node 0 active
Jul 25 12:23:01 TSHM4PIS llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (e1000g0) node 0 in trouble
Jul 25 12:23:02 TSHM4PIS llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (e1000g0) node 0 active
Jul 25 12:23:10 TSHM4PIS llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (e1000g0) node 0 in trouble
Jul 25 12:23:11 TSHM4PIS llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (e1000g0) node 0 active


All your help is appreciated.

The problem I have is now where to start in the first place....

Regards,

avsrini
Level 4
Employee Accredited Certified
Hi supermario,

From the configuration and error message it appears you are using LLT private link e1000g0 for host public
connection. It may be a low priority link, but send us /etc/llttab file to verify.


The error message you see is nothing wrong with the NIC, but the LLT protocol used for VCS communication
is timing out. This may very well due to the other IP network on the NIC. Its a general rule LLT private links are
not to be used for other traffic.

will commend further after seeing the /etc/llttab file.

Srini

supermario
Level 3
The LLT Tab :
# cat /etc/llttab
set-node TSHM4PIS
set-cluster 21
link e1000g1 /dev/e1000g:1 - ether - -
link nxge1 /dev/nxge:1 - ether - -
link-lowpri e1000g0 /dev/e1000g:0 - ether - -

It seems to use e1000g0 as a low priority link..

Gaurav_S
Moderator
Moderator
   VIP    Certified

To answer your questions:

a) How the hell is this working in the current place?

-- well there has been some configuration issues, but at this point, as all pointed, IPMP doesn't seems to be configured & you are using MultiNICB in base mode.... so MultiNICB & IPMultiNICB are managing the show for you...

b) why another cluster is working ....

-- Not sure on this without seeing any evidence....

c) As Srini pointed above, we need to see /etc/llttab file..... we need to know what network interfaces you have configured for LLT .... Both the high priority links should NOT be using the plumbed public interfaces...

LLT messages you are seeing might be because of you have configured low pri link... however lets see your /etc/llttab file & then comment..


Gaurav

Gaurav_S
Moderator
Moderator
   VIP    Certified

yep, so its using e1000g0 as low pri link .... Now certainly when any failover on this interface would be attempted, you would obviously see those LLT messages coming up....

Till the time your 2 high pri links are active, these messages shouldn't be of much worry...

Gaurav

supermario
Level 3
Hi Gaurav,
Have a look at this:
        MultiNICB MultiNICB_erp (
                MpathdCommand = "/usr/lib/inet/in.mpathd -a"
                Device = { e1000g0 = 0, nxge0 = 1 }
                NetworkHosts = { "172.21.52.1", "172.21.52.2", "172.21.52.11",
                         "172.21.52.12",
                         "172.21.150.22",
                         "172.21.150.23" }
                LinkTestRatio = 3
                IgnoreLinkStatus = 0
                NetworkTimeout = 300
                NoBroadcast = 1
                DefaultRouter = "172.21.150.22"
                Failback = 1
                )

This is from another cluster (same setup differenent applications).  I have no issues here..

LLT config from that server:
set-node TSHM4ERP
set-cluster 20
link e1000g1 /dev/e1000g:1 - ether - -
link nxge1 /dev/nxge:1 - ether - -
link-lowpri e1000g0 /dev/e1000g:0 - ether - -

Any ideas?

Anyhow, how do I proceed further.
Regards,

Gaurav_S
Moderator
Moderator
   VIP    Certified

thing I am wondering is,

/etc/hostname.e1000g0:TSHM4PIS
/etc/hostname.nxge0:TSHM4PIS-P2

If above are the hostname file details.... how come system is picking "groupname MultiNICB_pis" ? Does it show the same way on other cluster which is working OK ? If yes, what are the hostname.* file details there ?

what solaris version you are using 9 or 10 ? also can u paste:

# which in.mpathd
# uptime
# ps -ef |egrep 'mpath|inetd'

Gaurav

Marianne
Level 6
Partner    VIP    Accredited Certified
The LLT errors that you see are telling you that heartbeats on public NIC is a bad idea - there is so much traffic on e1000g0 that heartbeats are missed.

Maybe there's less traffic on the other cluster's public network, therefore no missed heartbeats.

Maybe go back to basics are re-look your design and consider Symantec's recommendation with regards to Private Network configuration in the Install Guide:

"The private network requires two communication channels to guard against network partitions."

"On each system, use two independent network cards to provide redundancy."

"Do not use the network interface card that is used for the public network"

"In addition to the built-in public Ethernet controller, VCS requires at least one more Ethernet interface per system. Symantec recommends two additional interfaces." (for heartbeats/private network)

Although your setup is supported, it is certainly not recommended and far from 'best practice'.
Without I/O fencing in place, you ideally need four NIC's:
2 for MultiNIC
2 for Private Network

I personally do not believe that Low-Pri links is a good idea when MultiNIC is configured.
If I have only three NIC's in each cluster node and no I/O fencing, my config will be as follows:
Choose two independent NIC's for private network (heartbeats)
Config Public network on remaining NIC (no MultiNIC)

IMHO, reliable hearbeat links are the most important aspect of the cluster - you only have to deal with Split Brain once in your life to understand the importance of independent heartbeat links...

Please also read this section in the VCS User Guide :
Chapter 10: About communications, membership, and data protection in the cluster
About cluster communications
About data protection
About cluster membership and data protection without I/O fencing
Examples of VCS operation without I/O fencing


g_lee
Level 6
supermario,

What do the /etc/hostname.* files look like on the working cluster, compared to the non-working one?

regarding how to fix this - as a start it would help to try the suggestions in previous posts about the requirements to setup MultiNICB ie: you need to fix the /etc/hostname.* files to set the correct properties (deprecated, -failover), and to set up the interface groups correctly.

regarding where the group name is being set, normally VCS sets this when you configure the resource. In this case there are other resource config issues that are causing problems, but it is probably still able to set the initial group name, etc

regarding the e1000g0 (lowpri link) messages - are you using 5.0MP3? see the following from the Late Breaking News:
sparc: http://seer.entsupport.symantec.com/docs/281987.htm
x64/x86: http://seer.entsupport.symantec.com/docs/286955.htm
(below is from sparc TN 281987, the workaround is the same for both platforms though)

Additions to Veritas Cluster Server 5.0 MP3 Release Notes:


Starting with 5.0MP3, Symantec has added the feature of one-way link detection in LLT (Etrack Incident# 1031514)  

Prior to this, (i.e. until 5.0MP1), LLT used broadcast heartbeats by default. Beginning with 5.0MP3, instead of using broadcast heartbeats, LLT uses unicast heartbeats.

LLT considers a link to be in trouble for a peer node when it finds that the link has not been up for that node for 2 seconds (peertrouble 200). On lo-pri links, LLT sends heartbeats just once every second; on hi-pri links, it is twice every second. With the above described change in LLT, it is possible that for lo-pri links, the troubled condition is hit more often. And hence the corresponding LLT messages will be printed more often in the system log. While this message is only informational, frequent appearance in the system logs may alarm the customer.

Therefore, it is recommended that the LLT peertrouble tunable should be increased to 400, so that link inactivity for up to 4 seconds is ignored by LLT before printing out the message in the system log.
            
As noted before, the trouble message is only informational and changing the trouble time to 4 seconds from 2 seconds is harmless.

If the following messages are seen frequently for an LLT lo-pri link, then change the LLT tunable named peertrouble to 400. Its default value is 200.

LLT INFO V-14-1-10205 link 2 (eri0) node 1 in trouble LLT INFO V-14-1-10024 link 2 (eri0) node 1 active

The peertrouble tunable can be changed with the following command on all the nodes in the cluster:

      lltconfig -T peertrouble:400

It is recommended that the following line be added to /etc/llttab on all the nodes in the cluster in order to ensure that this value is used across server reboots. On each node, the change will take effect only after restarting LLT.

      set-timer      peertrouble:400

Example:

# lltconfig -T query

Current LLT timer values (.01 sec units):
 . . .
 peertrouble = 200    <--------- before
 peerinact   = 1600
 . . .

# lltconfig -T peertrouble:400 <--- cmd

Use the following command to ensure that the values are indeed changed:

# lltconfig -T query

Current LLT timer values (.01 sec units):
 . . .
 peertrouble = 400    <--------- after
 peerinact   = 1600
 . . .