Red exclamation mark make the cluster hung / Failover stuck

Question

Environment
OS = RHEL 7.4
InfoScale HighAvailability = 7.4
SAN = ISCSI SAN(targetcli) connected with a separate ethernet card
Cluster nodes = 2
Resources = LVM-VolumeGroup &gt; LVM-LogicalVolume &gt; Mount &gt; Redis(custom agent) &gt; Application(custom agent) &gt; VIP
Incident
I hard boot active node and expected a successful failover. However a resource (redis server which is a custom agent) got stuck with a red exclamation mark. I suspect that this make the failover hung in middle. Its my assumption that redis resource was not able to be stopped due to its child resource become already unavailable/crashed/faulted and this seems a real reason for a unsuccessful failover.(image is attached for reference). Why red exclamation sign hung the failover?
&nbsp;
&nbsp;
&nbsp;
&nbsp;

riaanbadenhorst · Answer

What do you mean by hard boot?

zahid_haseeb · Answer

Power unplugged

riaanbadenhorst · Answer

Share the engine logs as well as the main.cf

zahid_haseeb · Answer

STANDBY NODE2018/09/10 09:51:38 VCS INFO V-16-6-15002 (COM2-XXXXX) hatrigger:hatrigger executed /opt/VRTSvcs/bin/internal_triggers/dump_tunables COM2-XXXXX 1 successfully2018/09/10 09:51:41 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system COM2-XXXXX2018/09/10 09:51:41 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System COM2-XXXXX as the system is not a part of AutoStartList attribute of the group.2018/09/10 09:52:49 VCS NOTICE V-16-1-10322 System COM1-XXXXX (Node '0') changed state from RUNNING to LEAVING2018/09/10 09:52:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource VIP (Owner: Unspecified, Group: SG) on System COM1-XXXXX2018/09/10 09:52:50 VCS INFO V-16-1-10305 Resource VIP (Owner: Unspecified, Group: SG) is offline on COM1-XXXXX (VCS initiated)2018/09/10 09:52:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource xxxx (Owner: Unspecified, Group: SG) on System COM1-XXXXX2018/09/10 09:52:50 VCS ERROR V-16-2-13067 (COM1-XXXXX) Agent is calling clean for resource(mount_lv) because the resource became OFFLINE unexpectedly, on its own.2018/09/10 09:52:50 VCS INFO V-16-2-13068 (COM1-XXXXX) Resource(mount_lv) - clean completed successfully.2018/09/10 09:52:50 VCS INFO V-16-1-10307 Resource mount_lv (Owner: Unspecified, Group: SG) is offline on COM1-XXXXX (Not initiated by VCS)2018/09/10 09:52:50 VCS INFO V-16-6-15015 (COM1-XXXXX) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed2018/09/10 09:52:50 VCS ERROR V-16-2-13067 (COM1-XXXXX) Agent is calling clean for resource(xxxx) because the resource became OFFLINE unexpectedly, on its own.2018/09/10 09:52:52 VCS INFO V-16-2-13068 (COM1-XXXXX) Resource(xxxx) - clean completed successfully.2018/09/10 09:52:53 VCS INFO V-16-1-10305 Resource xxxx (Owner: Unspecified, Group: SG) is offline on COM1-XXXXX (VCS initiated)2018/09/10 09:52:53 VCS NOTICE V-16-1-10300 Initiating Offline of Resource REDIS (Owner: Unspecified, Group: SG) on System COM1-XXXXX2018/09/10 09:52:53 VCS INFO V-16-1-55029 Resource xxxx in offline state received recurring offline message on system COM1-XXXXX2018/09/10 09:52:56 VCS ERROR V-16-2-13064 (COM1-XXXXX) Agent is calling clean for resource(REDIS) because the resource is up even after offline completed.2018/09/10 09:52:57 VCS INFO V-16-2-13068 (COM1-XXXXX) Resource(REDIS) - clean completed successfully.2018/09/10 09:52:59 VCS ERROR V-16-2-13077 (COM1-XXXXX) Agent is unable to offline resource(REDIS). Administrative intervention may be required.2018/09/10 09:53:16 VCS ERROR V-16-2-13067 (COM1-XXXXX) Agent is calling clean for resource(VG) because the resource became OFFLINE unexpectedly, on its own.2018/09/10 09:53:17 VCS ERROR V-16-2-13069 (COM1-XXXXX) Resource(VG) - clean failed.2018/09/10 09:53:22 VCS ERROR V-16-2-13067 (COM1-XXXXX) Agent is calling clean for resource(LV) because the resource became OFFLINE unexpectedly, on its own.2018/09/10 09:53:22 VCS INFO V-16-2-13068 (COM1-XXXXX) Resource(LV) - clean completed successfully.2018/09/10 09:53:22 VCS INFO V-16-1-10307 Resource LV (Owner: Unspecified, Group: SG) is offline on COM1-XXXXX (Not initiated by VCS)2018/09/10 09:53:22 VCS INFO V-16-6-15015 (COM1-XXXXX) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executedMAIN.CF FILE&nbsp;cat /etc/VRTSvcs/conf/config/main.cfinclude "OracleASMTypes.cf"include "types.cf"include "Db2udbTypes.cf"include "OracleTypes.cf"include "SybaseTypes.cf"cluster CLUSTER-XXXXX (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UserNames = { admin = cpqIpkPmqLqqOyqKpn }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Administrators = { admin }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )system COM1-XXXXX (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )system COM2-XXXXX (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )group SG (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SystemList = { COM1-XXXXX = 0, COM2-XXXXX = 1 }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AutoStartList = { COM1-XXXXX, COM2-XXXXX }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Application xxxx (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Critical = 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; User = xxxx&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; StartProgram = "/home/xxxx/HA/xxxx.sh start"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; StopProgram = "/home/xxxx/HA/xxxx.sh stop"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MonitorProcesses = { "xxxx.exe COMMS" }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; UseSUDash = 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Application REDIS (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; StartProgram = "/home/xxxx/HA/redis-start.sh"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; StopProgram = "/home/xxxx/HA/redis-stop.sh"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MonitorProcesses = { "/usr/bin/redis-server 192.168.2.12:8999" }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; IP REDIS-VIPVIP (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Device = eno16780032&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Address = "192.168.2.12"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetMask = "255.255.248.0"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; IP VIP (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Device = eno16780032&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Address = "192.168.2.20"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetMask = "255.255.248.0"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LVMLogicalVolume LV (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LogicalVolume = xxxx-VG-VOL1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VolumeGroup = xxxx-VG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LVMVolumeGroup VG (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VolumeGroup = xxxx-VG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; StartVolumes = 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ScanDevices = 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Mount mount_lv (&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MountPoint = "/home/xxxx"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; BlockDevice = "/dev/mapper/xxxx--VG-xxxx--VG--VOL1"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FSType = ext4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FsckOpt = "-y"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; xxxx requires REDIS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LV requires VG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; REDIS requires REDIS-VIPVIP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; REDIS requires mount_lv&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VIP requires xxxx&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mount_lv requires LV&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // resource dependency tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; group SG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; IP VIP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Application xxxx&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Application REDIS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Mount mount_lv&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LVMLogicalVolume LV&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LVMVolumeGroup VG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; IP REDIS-VIPVIP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }

frankgfan · Answer

Forum Discussion

Red exclamation mark make the cluster hung / Failover stuck

Related Content

Netbackup 8.1 client and Windows failover cluster

Cluster service group failed to failover

Cluster Failover

Cluster Failover: Clearing VCS for Automatic Failover

Windows Failover cluster backup fails