SFRAC - 5.0 configuration problem

[Saran_a]

Hi All.

I am trying to setup the VCS 5.0 across 2 HP-UX 11.23 machines, I am getting error messages in VCSweb and cvm_clus.
Please have a look on the following cluster command outputs. Cluster setup is not getting UP between the two systems { invicta and charger }.
And gabconfig -a output ensures Port w - vxconfigd port is not getting enabled in both the systems.

# hastatus -summary

-- SYSTEM STATE
-- System State Frozen

A charger RUNNING 0
A invicta RUNNING 0

-- GROUP STATE
-- Group System Probed AutoDisabled State

B ClusterService charger Y N OFFLINE
B ClusterService invicta Y N STARTING|PART
B cvm charger Y N OFFLINE|FAULT
B cvm invicta Y N ONLINE

-- RESOURCES FAILED
-- Group Type Resource System

C ClusterService VRTSWebApp VCSweb invicta
C cvm CVMCluster cvm_clus charger
#


grep -i "fault" /var/VRTSvcs/log/engine_A.log

2006/10/18 11:28:24 VCS ERROR V-16-1-10303 Resource VCSweb (Owner: unknown, Group: ClusterService) is FAULTED (timed out) on sys invicta
2006/10/18 11:28:24 VCS INFO V-16-6-15004 (invicta) hatrigger:Failed to send trigger for resfault; script doesn't exist
2006/10/18 11:45:16 VCS ERROR V-16-1-10303 Resource cvm_clus (Owner: unknown, Group: cvm) is FAULTED (timed out) on sys invicta
2006/10/18 11:45:16 VCS ERROR V-16-1-10205 Group cvm is faulted on system invicta
2006/10/18 11:45:17 VCS INFO V-16-6-15004 (invicta) hatrigger:Failed to send trigger for resfault; script doesn't exist
2006/10/18 14:45:53 VCS ERROR V-16-1-10303 Resource VCSweb (Owner: unknown, Group: ClusterService) is FAULTED (timed out) on sys invicta
2006/10/18 14:45:53 VCS INFO V-16-6-15004 (invicta) hatrigger:Failed to send trigger for resfault; script doesn't exist
2006/10/18 14:49:06 VCS ERROR V-16-1-10303 Resource VCSweb (Owner: unknown, Group: ClusterService) is FAULTED (timed out) on sys charger
2006/10/18 14:49:06 VCS INFO V-16-6-15004 (charger) hatrigger:Failed to send trigger for resfault; script doesn't exist
2006/10/18 14:57:34 VCS ERROR V-16-1-10303 Resource VCSweb (Owner: unknown, Group: ClusterService) is FAULTED (timed out) on sys invicta
2006/10/18 14:57:34 VCS INFO V-16-6-15004 (invicta) hatrigger:Failed to send trigger for resfault; script doesn't exist
2006/10/18 15:14:46 VCS ERROR V-16-1-10303 Resource cvm_clus (Owner: unknown, Group: cvm) is FAULTED (timed out) on sys charger
2006/10/18 15:14:46 VCS ERROR V-16-1-10205 Group cvm is faulted on system charger
2006/10/18 15:14:46 VCS INFO V-16-6-15004 (charger) hatrigger:Failed to send trigger for resfault; script doesn't exist


In Node A

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen 5eb501 membership 01
Port b gen 5eb503 membership 01
Port d gen 5eb505 membership 01
Port f gen 5eb512 membership 01
Port h gen 5eb524 membership 01
Port o gen 5eb505 membership 01
Port v gen 5eb526 membership 01
Port w gen 5eb527 membership 0
Port w gen 5eb527 visible ;1
#

In Node B

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen 5eb501 membership 01
Port b gen 5eb503 membership 01
Port d gen 5eb505 membership 01
Port f gen 5eb512 membership 01
Port h gen 5eb524 membership 01
Port o gen 5eb505 membership 01
Port v gen 5eb526 membership 01
#

Please pass me hints on how to debug this problem, possibly with which binaries should be started with which conf files..etc

Thanks
Saravanan

[Hywel_Mallett]

I've only used VCS on Windows before, but the basic concepts should be similar.Starting at the lowest level, I noticed that the gab port membership was different on the two systems. On Windows, if I remember correctly there are only two ports (a and h), but you need both to operate, and it should have the same port membership on each node.
HP-UX may be different, but I'd start looking at that first.
Can you paste the output of lltstat? LLT runs at a lower level even than GAB.

I'm sure Gene will be along soon with the answer!

[Gene_Henriksen]

Sorry Hywel, I am a bit off schedule this week being in Sweden rather than the US. Port W is used by vxconfigd in CVM. port W transfers changes in the configuration of the VM objects.

I am not sure why it is not starting. A call to support may help.

Port a gen 4a1c0001 membership 01 (GAB)
Port d gen 40100001 membership 01 (ODM)
Port f gen f1990002 membership 01 (CFS)
Port h gen d8850002 membership 01 (HAD)
Port o gen 243f0002 membership 01 (VCSMM)
Port q gen 28d10002 membership 01 (qlog)
Port v gen 1fc60002 membership 01 (VxVM)
Port w gen 15ba0002 membership 01 (VxCONFIGD)

This list is for 4.0, so may differ in 5.0.

[Saran_a]

Thanks for replying. Please find the lltstat output below.

lltstat output In node A ( charger )

# lltstat
LLT statistics:
119105 Snd data packets
47 Snd retransmit data
2134480 Snd connect packets
35127 Snd independent ACKs
56328 Snd piggyback ACKs
0 Snd independent NACKs
0 Snd piggyback NACKs
6445 Snd loopback packets
142356 Rcv data packets
0 Rcv out of window
0 Rcv duplicates
0 Rcv datagrams dropped
0 Rcv multiblock data
0 Rcv misaligned data
1 Snd chained header
LLT errors:
7 Rcv not connected
0 Rcv unconfigured
0 Rcv bad dest address
0 Rcv bad source address
0 Rcv bad generation
0 Rcv no buffer
0 Rcv malformed packet
0 Rcv bad SAP
0 Rcv bad STREAM primitive
0 Rcv bad DLPI primitive
0 Rcv DLPI error
360 Snd not connected
0 Snd no buffer
0 Snd stream flow drops
12 Snd no links up
0 Rcv bad checksum
0 Rcv bad udp/ether source address
0 Rcv DLPI link-down error
#

lltstat output In node B ( invicta )

# lltstat
LLT statistics:
142480 Snd data packets
0 Snd retransmit data
2136154 Snd connect packets
47252 Snd independent ACKs
56756 Snd piggyback ACKs
0 Snd independent NACKs
0 Snd piggyback NACKs
234927 Snd loopback packets
119209 Rcv data packets
47 Rcv out of window
0 Rcv duplicates
0 Rcv datagrams dropped
0 Rcv multiblock data
0 Rcv misaligned data
0 Snd chained header
LLT errors:
6 Rcv not connected
0 Rcv unconfigured
0 Rcv bad dest address
0 Rcv bad source address
0 Rcv bad generation
0 Rcv no buffer
0 Rcv malformed packet
0 Rcv bad SAP
0 Rcv bad STREAM primitive
0 Rcv bad DLPI primitive
0 Rcv DLPI error
393 Snd not connected
0 Snd no buffer
0 Snd stream flow drops
69 Snd no links up
0 Rcv bad checksum
0 Rcv bad udp/ether source address
0 Rcv DLPI link-down error
#

Port w is Vxconfigd port. And verified the daemon is running in both the hosts.

# ps -ef | grep vxconfigd
root 284 1 0 Oct 17 ? 0:17 /usr/sbin/vxconfigd -k -m enable
root 9780 9240 1 11:35:23 pts/0 0:00 grep vxconfigd
#

From the logs, it seems the problem is with the cvm_clus and VCSweb.. Any assistance/hints on solving this problem ?

Thanks
Saravanan

[Gene_Henriksen]

Again, I recommend you contact support.