Highlighted

hastatus show fault

I run the cluster to show the group faulted status.  I ran the hagrp -clear , but it failed to start up the service group

root@devuaedms11 # hastatus -sum

-- SYSTEM STATE
-- System State Frozen

A devuaedms11 RUNNING 0
A devuaedms12 RUNNING 0

-- GROUP STATE
-- Group System Probed AutoDisabled State

B Documentum devuaedms11 Y N PARTIAL
B Documentum devuaedms12 Y N PARTIAL|FAULTED
B Documentum_ENV_B devuaedms11 Y N OFFLINE|FAULTED
B Documentum_ENV_B devuaedms12 Y N OFFLINE|FAULTED
B MNICB_SITgroup devuaedms11 Y N ONLINE
B MNICB_SITgroup devuaedms12 Y N ONLINE
B cvm devuaedms11 Y N ONLINE
B cvm devuaedms12 Y N ONLINE
B documentum_bkup devuaedms11 Y N ONLINE
B documentum_bkup devuaedms12 Y N OFFLINE

-- RESOURCES FAILED
-- Group Type Resource System

C Documentum CVMVolDg diskgroup_cfs1 devuaedms12
C Documentum_ENV_B Proxy DocBroker_B devuaedms11
C Documentum_ENV_B Proxy DocBroker_B devuaedms12

 

4 Replies
Highlighted

Re: hastatus show fault

I think you need to have a look at this resource and why it failed - type CVMVolDg diskgroup_cfs1  on devuaedms12 and also there are proxy resources that are failed Documentum_ENV_B  on devuaedms11 and devuaedms12

 

Highlighted

Re: hastatus show fault

I can't run the command 

 

root@devuaedms12 # CVMVolDg diskgroup_cfs1
CVMVolDg: not found

Highlighted

Re: hastatus show fault

i check the log to show this message, I try to clear the fault but it still not starting.   

2017/12/27 12:41:08 VCS INFO V-16-1-10305 Resource diskgroup_cfs2 (Owner: unknown, Group: Documentum) is offline on devuaedms11 (VCS initiated)
2017/12/27 12:41:08 VCS ERROR V-16-1-10205 Group Documentum is faulted on system devuaedms11
2017/12/27 12:41:08 VCS NOTICE V-16-1-10446 Group Documentum is offline on system devuaedms11
2017/12/27 12:41:08 VCS INFO V-16-1-10493 Evaluating devuaedms11 as potential target node for group Documentum
2017/12/27 12:41:08 VCS INFO V-16-1-50010 Group Documentum is online or faulted on system devuaedms11
2017/12/27 12:41:08 VCS INFO V-16-1-10493 Evaluating devuaedms12 as potential target node for group Documentum
2017/12/27 12:41:08 VCS WARNING V-16-1-10158 Group Documentum autodisabled on system devuaedms12. This can happen if VCS is not running on the system or group is not probed on the system
2017/12/27 12:41:08 VCS INFO V-16-6-15004 (devuaedms11) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2017/12/27 12:41:08 VCS INFO V-16-6-15004 (devuaedms11) hatrigger:Failed to send trigger for nofailover; script doesn't exist
2017/12/27 12:41:08 VCS INFO V-16-2-13075 (devuaedms11) Resource(ContentServer_B) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(3).
2017/12/27 12:41:57 VCS INFO V-16-1-10307 Resource DocBroker_B (Owner: unknown, Group: Documentum_ENV_B) is offline on devuaedms11 (Not initiated by VCS)
2017/12/27 12:41:57 VCS NOTICE V-16-1-10300 Initiating Offline of Resource JavaMethodServer_B (Owner: unknown, Group: Documentum_ENV_B) on System devuaedms11
2017/12/27 12:41:57 VCS INFO V-16-6-15004 (devuaedms11) hatrigger:Failed to send trigger for resfault; script doesn't exist
2017/12/27 12:41:59 VCS INFO V-16-2-13001 (devuaedms11) Resource(JavaMethodServer_B): Output of the completed operation (offline)

Highlighted

Re: hastatus show fault

I got the message from the vcs log , 

2017/12/27 15:01:50 VCS WARNING V-16-10001-1074 (devuaedms12) CVMVolDg:diskgroup_cfs1Smiley Surprisednline:setup_vxnotify: old vxnotify of pid 4863 will be killed. my pid is 7548
2017/12/27 15:01:52 VCS ERROR V-16-10001-1009 (devuaedms12) CVMVolDg:diskgroup_cfs1Smiley Surprisednline:could not find diskgroup cfs1_datadg imported. If it was previously deported, it will have to be manually imported
2017/12/27 15:01:52 VCS ERROR V-16-10001-1044 (devuaedms12) CVMVolDg:diskgroup_cfs1Smiley Surprisednline:Error saving vxprint file
2017/12/27 15:01:52 VCS INFO V-16-10001-1046 (devuaedms12) CVMVolDg:diskgroup_cfs2Smiley Surprisednline:resource diskgroup_cfs2 is online
2017/12/27 15:01:53 VCS INFO V-16-2-13001 (devuaedms12) Resource(diskgroup_cfs1): Output of the completed operation (online)
VxVM vxprint ERROR V-5-1-582 Disk group cfs1_datadg: No such disk group
2017/12/27 15:02:01 VCS ERROR V-16-2-13066 (devuaedms11) Agent is calling clean for resource(diskgroup_cfs1) because the resource is not up even after online completed.
2017/12/27 15:02:02 VCS INFO V-16-2-13001 (devuaedms11) Resource(diskgroup_cfs1): Output of the completed operation (clean)
/var/VRTSvcs/lock/diskgroup_cfs1_cfs1_datadg_stat: No such file or directory
2017/12/27 15:02:02 VCS INFO V-16-2-13068 (devuaedms11) Resource(diskgroup_cfs1) - clean completed successfully.
2017/12/27 15:02:02 VCS INFO V-16-2-13071 (devuaedms11) Resource(diskgroup_cfs1): reached OnlineRetryLimit(0).
2017/12/27 15:02:03 VCS ERROR V-16-1-10303 Resource diskgroup_cfs1 (Owner: unknown, Group: Documentum) is FAULTED (timed out) on sys devuaedms11
2017/12/27 15:02:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource CFS_Mnt_cfs2_d01 (Owner: unknown, Group: Documentum) on System devuaedms11
2017/12/27 15:02:03 VCS INFO V-16-6-15004 (devuaedms11) hatrigger:Failed to send trigger for resfault; script doesn't exist
2017/12/27 15:02:05 VCS INFO V-16-1-10305 Resource CFS_Mnt_cfs2_d01 (Owner: unknown, Group: Documentum) is offline on devuaedms11 (VCS initiated)
2017/12/27 15:02:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource diskgroup_cfs2 (Owner: unknown, Group: Documentum) on System devuaedms11
2017/12/27 15:02:07 VCS INFO V-16-1-10305 Resource diskgroup_cfs2 (Owner: unknown, Group: Documentum) is offline on devuaedms11 (VCS initiated)
2017/12/27 15:02:07 VCS ERROR V-16-1-10205 Group Documentum is faulted on system devuaedms11
2017/12/27 15:02:07 VCS NOTICE V-16-1-10446 Group Documentum is offline on system devuaedms11
2017/12/27 15:02:07 VCS INFO V-16-1-10493 Evaluating devuaedms11 as potential target node for group Documentum
2017/12/27 15:02:07 VCS INFO V-16-1-50010 Group Documentum is online or faulted on system devuaedms11
2017/12/27 15:02:07 VCS INFO V-16-1-10493 Evaluating devuaedms12 as potential target node for group Documentum
2017/12/27 15:02:07 VCS INFO V-16-1-50010 Group Documentum is online or faulted on system devuaedms12
2017/12/27 15:02:07 VCS INFO V-16-6-15004 (devuaedms11) hatrigger:Failed to send trigger for nofailover; script doesn't exist
2017/12/27 15:02:07 VCS INFO V-16-6-15004 (devuaedms11) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2017/12/27 15:02:24 VCS INFO V-16-1-10298 Resource diskgroup_cfs2 (Owner: unknown, Group: Documentum) is online on devuaedms12 (VCS initiated)
2017/12/27 15:02:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource CFS_Mnt_cfs2_d01 (Owner: unknown, Group: Documentum) on System devuaedms12
2017/12/27 15:02:25 VCS INFO V-16-10001-5506 (devuaedms12) CFSMount:CFS_Mnt_cfs2_d01Smiley Surprisednline:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster, Block Device /dev/vx/dsk/cfs2_datadg/cfs2_d01_vol : MountPoint /dctm/dctm_storage_02
2017/12/27 15:02:27 VCS INFO V-16-1-10298 Resource CFS_Mnt_cfs2_d01 (Owner: unknown, Group: Documentum) is online on devuaedms12 (VCS initiated)
2017/12/27 15:03:54 VCS ERROR V-16-2-13066 (devuaedms12) Agent is calling clean for resource(diskgroup_cfs1) because the resource is not up even after online completed.
2017/12/27 15:03:55 VCS INFO V-16-2-13001 (devuaedms12) Resource(diskgroup_cfs1): Output of the completed operation (clean)
/var/VRTSvcs/lock/diskgroup_cfs1_cfs1_datadg_stat: No such file or directory
2017/12/27 15:03:55 VCS INFO V-16-2-13068 (devuaedms12) Resource(diskgroup_cfs1) - clean completed successfully.
2017/12/27 15:03:55 VCS INFO V-16-2-13071 (devuaedms12) Resource(diskgroup_cfs1): reached OnlineRetryLimit(0).
2017/12/27 15:03:56 VCS ERROR V-16-1-10303 Resource diskgroup_cfs1 (Owner: unknown, Group: Documentum) is FAULTED (timed out) on sys devuaedms12
2017/12/27 15:03:56 VCS NOTICE V-16-1-10300 Initiating Offline of Resource CFS_Mnt_cfs2_d01 (Owner: unknown, Group: Documentum) on System devuaedms12
2017/12/27 15:03:56 VCS INFO V-16-6-15004 (devuaedms12) hatrigger:Failed to send trigger for resfault; script doesn't exist
2017/12/27 15:03:58 VCS INFO V-16-1-10305 Resource CFS_Mnt_cfs2_d01 (Owner: unknown, Group: Documentum) is offline on devuaedms12 (VCS initiated)
2017/12/27 15:03:58 VCS NOTICE V-16-1-10300 Initiating Offline of Resource diskgroup_cfs2 (Owner: unknown, Group: Documentum) on System devuaedms12
2017/12/27 15:04:00 VCS INFO V-16-1-10305 Resource diskgroup_cfs2 (Owner: unknown, Group: Documentum) is offline on devuaedms12 (VCS initiated)
2017/12/27 15:04:00 VCS ERROR V-16-1-10205 Group Documentum is faulted on system devuaedms12
2017/12/27 15:04:00 VCS NOTICE V-16-1-10446 Group Documentum is offline on system devuaedms12
2017/12/27 15:04:00 VCS INFO V-16-1-10493 Evaluating devuaedms11 as potential target node for group Documentum
2017/12/27 15:04:00 VCS INFO V-16-1-50010 Group Documentum is online or faulted on system devuaedms11
2017/12/27 15:04:00 VCS INFO V-16-1-10493 Evaluating devuaedms12 as potential target node for group Documentum
2017/12/27 15:04:00 VCS INFO V-16-1-50010 Group Documentum is online or faulted on system devuaedms12
2017/12/27 15:04:01 VCS INFO V-16-6-15004 (devuaedms11) hatrigger:Failed to send trigger for nofailover; script doesn't exist
2017/12/27 15:04:01 VCS INFO V-16-6-15004 (devuaedms12) hatrigger:Failed to send trigger for postoffline; script doesn't exist