VCS ERROR V-16-3-18205

[Zahid_Haseeb]

Environment

OS = 6.3

SFHA = 6.1

2014/03/11 18:59:54 VCS INFO V-16-1-51058 Received connection request from wide area connector
2014/03/11 18:59:54 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted
2014/03/11 18:59:54 VCS INFO V-16-1-50403 VCS engine on system SECNODE has established connection with wide area connector
2014/03/11 18:59:54 VCS INFO V-16-10031-509 (SECNODE) Application:wac:online:Executed </opt/VRTSvcs/bin/wacstart> as user <root>. The program exited with return code <0>.
2014/03/11 18:59:54 VCS ERROR V-16-3-18205 (SECNODE) File /opt/VRTSvcs/bin/hb/em4/em4Agent not found
2014/03/11 18:59:54 VCS ERROR V-16-3-18205 (SECNODE) File /opt/VRTSvcs/bin/hb/GlobalHB/GlobalHBAgent not found
2014/03/11 18:59:54 VCS ERROR V-16-3-18205 (SECNODE) File /opt/VRTSvcs/bin/hb/GCO-HB/GCO-HBAgent not found
2014/03/11 18:59:57 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on SECNODE (VCS initiated)
2014/03/11 18:59:57 VCS NOTICE V-16-1-10447 Group ClusterService is online on system SECNODE
2014/03/11 19:00:53 VCS INFO V-16-1-50135 User admin fired command: haconf -dump from ::ffff:172.16.203.2
 

[mikebounds]

/opt/VRTSvcs/bin/hb contains the global heartbeat agents.  What is em4 and GlobalHB - have you tried to create your own global heartbeats.

In early version of VCS I think you could create your own global heartbeats, but I don't think you can anymore so your global heartbeat should be called Icmp.

Mike

[Zahid_Haseeb]

- Why cant we create extra global heartbeats. There are option availabvle at JAVA CONSOLE > EDIT > CONFIGURE HEARTBEATS.

- I removed the em4 , its a global heartbeat i configured as above.

[mikebounds · Accepted solution]

See extract from VCS admin:

The wide-area heartbeat agent manages the inter-cluster heartbeat. Heartbeats

are used to monitor the health of remote clusters. VCS wide-area hearbeat agents

include Icmp and IcmpS. While other VCS resource agents report their status to

VCS engine, heartbeat agents report their status directly to the WAC process. The

heartbeat name must be the same as the heartbeat type name. You can add only

one heartbeat of a specific heartbeat type.

See “Sample configuration for the wide-area heartbeat agent” on page 543.

You can create custom wide-area heartbeat agents. For example, the VCS

replication agent for SRDF includes a custom heartbeat agent for Symmetrix arrays.

You can add heartbeats using the hahb -add heartbeatname command and

change the default values of the heartbeat agents using the hahb -modify

command.

So actually you can create your own heartbeat, but it's name must be the same as the heartbeat type name (where as in GCM the predecessor to GCO, you could have multiple heartbeats of the same type), but you will need to create a new agent.  The "edit heartbeat" should only be used to change the attributes of the heartbeat.

Now you have removed em4, is the wac resource ok?

Mike

[Zahid_Haseeb]

Yap. Things are going good. But this is unfortunate that we only depend on one global heartbeat.

Mike, does not it fair. The Internet connection might down sometimes so we have an option to add one more heartbeat from a different internet connectivity

[Wally_Heim]

Hi Zahid,

I see this on Windows sometimes.  With the create new heartbeat option that you select in the Java GUI, you need to create the heartbeat binaries for the connection that you are trying to create.  This is for a new type of heartbeat and not another heartbeat.

VCS comes bundled with ICMP and ICMPS heartbeats.  You can configure either one of these as needed.  They both work off the TCP/IP protocol ICMP or ping.  This means that they go over your standard TCP/IP transport paths to get to the "ClusterAddress" of the remote cluster.   The main difference between ICMP and ICMPS is that ICMPS is serialized so that it will talk to 1 remote cluster at a time until it has cycled through all remote clusters.  ICMP will talk to all remote clusters at the same time.  ICMPS is used for testing more than actual production configurations.

The main point is that ICMP/ICMPS uses the OS TCP/IP stack so if you make backup routes either heartbeat will start using them in a weighted fashion as needed.  Additional global heartbeats would not be needed in such cases.

Also global heartbeats are slightly less important than local cluster heartbeats when using default settings.  By default global clusters do not automatically failover the service groups between sites.  The cluster failover policy is set to "Manual" by default. 

If you need to automate site failover and you want to minimize accidental site failover, you can set the cluster failover policy to "Connected".  This will only failover the global application if the GCO heartbeat is connected.  If the GCO heartbeat is down then global failover reverts back to manual.

I do not recommend setting the cluster failover policy to "Automatic".  This just invites a split brain situation when you loose a GCO heartbeat connection.

Thank you,

Wally

[Zahid_Haseeb]

Hmm So we cannot establish two Global heartbeats with default icmp. Else we need to write an agent. This is what I understand finally.