How to delete non-existant system from the Cluster "-i"

I agree with Marianne, you still need to do hastop -force as in essence "-i" for a host is an invalid string which is rejected by "check of main.cf" when VCS started and "check of command hasys", but there is a side case where if VCS node name (which is normally taken from /etc/VRTSvcs/conf/sysname) is not in main.cf when VCS start then it adds it to VCS and it looks as though the "checks" don't take place here.  Once VCS is started if you change /etc/VRTSvcs/conf/sysname (or if this is not there, then I think VCS uses hostname) then this will not affect VCS as this is only determined at VCS start up.  So as "hasys" rejects "-i" for host you need to run "hastop -all -force", and then restart VCS which will only start on first node if main.cf does NOT contain the invalid "-i" hostname (all other systems will build main.cf from the first node)

Mike

As per Mike's previous post, 'hastop -all -force' will stop VCS on all nodes, but leave apps running.

So, we want node1 with correct main.cf to start 'had' and load correct config into memory.
Run hastatus in one window to view continuous progess. (Will firstly say 'cannot connect...' when had is down on all nodes.)
After stopping VCS (had) on all nodes, run 'hastart' on node 1.

Wait for 'hastatus' to show node 1 in 'LOCAL BUILD'. Wait for RUNNING state.
Run hastart on remaining nodes. They should all do a 'Remote Build' from node 1.

All nodes should now share the correct main.cf with no '-i' entry.

I can see currently Node3 is master node, this is for your information.

bash-3.00# /etc/vx/bin/vxclustadm nidmap
Name                             CVM Nid    CM Nid     State              
MHOGW01                          0          0          Joined: Slave      
MHOGW02                          1          1          Joined: Slave      
MHOGW03                          2          2          Joined: Master     
MHOGW04                          3          3          Joined: Slave

 

Hi Mike,

I checked /etc/VRTSvcs/conf/sysname in all the node but there is no such entry in the sysname in all 4 nodes.

I think you are aware that main.cf on node 1 is correct but main.cf in other 3 nodes having entry of "-i"

kindly let me know "hastop -all force" will not affect running services in cluster as all the nodes having some running services.

bash-3.00# hastatus -summ | grep ONLINE
B  DG1     MHOGW01              Y          N               ONLINE        
B  DG1     MHOGW02              Y          N               ONLINE        
B  DG1     MHOGW03              Y          N               ONLINE        
B  DG1     MHOGW04              Y          N               ONLINE        
B  Network         MHOGW01              Y          N               ONLINE        
B  Network         MHOGW02              Y          N               ONLINE        
B  Network         MHOGW03              Y          N               ONLINE        
B  Network         MHOGW04              Y          N               ONLINE        
B  ALARM     MHOGW01              Y          N               ONLINE        
B  MANAGER   MHOGW01              Y          N               ONLINE        
B  SERVER    MHOGW01              Y          N               ONLINE        
B  SERVER10  MHOGW04              Y          N               ONLINE        
B  SERVER2   MHOGW03              Y          N               ONLINE        
B  SERVER3   MHOGW03              Y          N               ONLINE        
B  SERVER4   MHOGW01              Y          N               ONLINE        
B  SERVER5   MHOGW04              Y          N               ONLINE        
B  SERVER6   MHOGW02              Y          N               ONLINE        
B  SERVER7   MHOGW04              Y          N               ONLINE        
B  SERVER8   MHOGW01              Y          N               ONLINE        
B  SERVER9   MHOGW04              Y          N               ONLINE        
B  TRACER    MHOGW01              Y          N               ONLINE        
B  Sentinel        MHOGW01              Y          N               ONLINE        
B  cvm             MHOGW01              Y          N               ONLINE        
B  cvm             MHOGW02              Y          N               ONLINE        
B  cvm             MHOGW03              Y          N               ONLINE        
B  cvm             MHOGW04              Y          N               ONLINE

Node2,3,4

bash-3.00# hastatus -summ

-- SYSTEM STATE
-- System               State                Frozen            

A  -i                   FAULTED              0                   
A  MHOGW01              RUNNING              0                   
A  MHOGW02              RUNNING              0                   
A  MHOGW03              RUNNING              0                   
A  MHOGW04              RUNNING              0

Node 1:

bash-3.00# hastatus -summ | more

-- SYSTEM STATE
-- System               State                Frozen            

A  MHOGW01              RUNNING              0                   
A  MHOGW02              RUNNING              0                   
A  MHOGW03              RUNNING              0                   
A  MHOGW04              RUNNING              0

 

I managed to replicate what you have by putting "-i" in /etc/VRTSvcs/conf/sysname and when you do this and start vcs it adds host "-i" to main.cf.

So to fix I would do:

Stop VCS on all node, but leave apps running

hastop -all force

Correct /etc/VRTSvcs/conf/sysname

Start VCS on node with correct main.cf (node 1)

Start VCS on other nodes (this will build main.cf from node1 so no need to edit main.cf on other nodes)

Mike

Dear Mike,

This problem occured as hostname of one of the node was changed. because of someone has run the command 'hostname -i'

IF you have test system, can u please check by changing hostname of the server by "hostname -i"

share me you observation.

 

Thanks in advance.

 

Regards,

Amit Mane

Hi,

its Solaris 10 and VRTSvcs 5.1

Regards,

AMit MAne

I did not notice if you provided the relevent release levels, but I see you are on solaris -- FYI that my testing was done on "VRTSvcs      6.0.300.000" and "Red Hat Enterprise Linux Server release 6.4 (Santiago)"

 

Well that is very strange indeed.

I agree with Mike; I also could not in any way insert a node name of "-i" into the cluster, either by CLI or direct editing of the main.cf file. 

I also agree with Marianne that if a node had a bad configuration and it was the first to boot, it would not form a cluster; but when one of the other nodes with a good main.cf file did start VCS, it would local build, and only then would node 1 (with the bad main.cf) would do a remote build, and then overwrite the main.cf file.

The fact that node 1 now has an incorrect main.cf means that somebody with root authority has directly modified the file after the node had joined the currently runing cluster.

On the nodes that show you the incorrect system name, you can attempt the following command, though I doubt it will work:

  -$ hasys -delete %-i

Probably, the best thing to do is to fix the bad file on node 1 simply by opening the cluster configuration and then closing it again -- this will overwrite every main.cf on every node in the cluster with a good copy.  Here's the CLI to do that:

  -$ haconf -makerw

  -$ haconf -dump -makero

Once you have done that, if the three nodes that "reveal" that bad nodename persists I guess then you could stop the cluster and restart it. 

Something tells me that there is more to this story as the situation you describe should not really be possible -- I directly modified the main.cf file and put in "system "-i" ( ) " but I could not re-create your result of having the bad system name revealed via the "hasys -list" command -- neither from running that command on the host with the good main.cf (as I believe you describe), nor from running it on the host with the bad main.cf.