Listener resource remain faulted
Hello,
we are doing some failure tests for a customer. We have VCS 6.2 running on solaris 10. We have an Oracle database and of course the listener associated with it.
We try to simulate different kind of failures. One of them is to kill the listener. In this situation the cluster observes that the listener has died, and it fails over the service to the other node. BUT the listener resource will remain in FAULTED state on the original node, and the group to which belongs will be in OFFLINE FAULTED state. In this situation if something goes wrong on the second node the service will not fail back to the original one until we manually run hagrp -clear.
Is there anything we can do to fix this? (to have the clear done automatically)
Here are some lines from the log:
2015/03/30 17:26:10 VCS ERROR V-16-2-13067 (node2p) Agent is calling clean for resource(ora_listener-res) because the resource became OFFLINE unexpectedly, on its own.
2015/03/30 17:26:11 VCS INFO V-16-2-13068 (node2p) Resource(ora_listener-res) - clean completed successfully.
2015/03/30 17:26:11 VCS INFO V-16-1-10307 Resource ora_listener-res (Owner: Unspecified, Group: oracle_rg) is offline on node2p (Not initiated by VCS)
in these it says that clean for the resource has completed successfully, but the resource is still faulted.
but if I run hares -clear manually, the the fault goes away.
20150330-173628:root@node1p:~# hares -state ora_listener-res
#Resource Attribute System Value
ora_listener-res State node1p ONLINE
ora_listener-res State node2p FAULTED
20150330-173636:root@node1p:~# hares -clear ora_listener-res
20150330-173653:root@node1p:~# hares -state ora_listener-res
#Resource Attribute System Value
ora_listener-res State node1p ONLINE
ora_listener-res State node2p OFFLINE
20150330-173655:root@node1p:~#
- HI, Seems vcs can't do this automatically. It's by design. In real production system, listener fault by some reason, it failover to other nodes. System admin need find that, and clear the fault. So listener can failover back later. If vcs automatically clear the fault, listener may still fault if error condition still there. So by design, vcs not clear the fault automatically. Regards
As starflyfly says, this is by design to prevent "ping -ponging" - i.e if there is an issue on node1 which causes listener to fail and then group fails over to node 2 and listener fails on node2 also, then if node1 was not in faulted state, then it would fail back to node 1 and then continue to ping-pong between the servers. You could use a Preonline script to run "hagrp -clear", so that when group comes up on node 2 it clears all faults (on node 1), but I would not recommend this as then you can get "ping-ponging", so you should manually clear faults which indicates you have fixed the issue that cause the resource to fault and so system is now valid to fail back to.
For the listener resource I would recommend setting the RestartLimit so that the process restarts locally first so if the process just happens to die and there is nothing wrong with system, then VCS will restart the listener without having to fail the whole group over. You can set RestartLimit to 1 or more to restart 1 or more times - example:
hatype -modify Netlsnr RestartLimit 2
Mike
Hi,
The above message indicates that "clean" script completed successfully which means what ever was defined as clean action was executed & returned a completed status code without errors ..
Whether the above action has cleared the fault or not, will be determined by "monitor" script again .. so once clean is executed, post that a monitor will execute to determine the status of the resource
G
