cancel
Showing results for 
Search instead for 
Did you mean: 

Need to know the root cause for filesystem failure in SFS NAS

symsonu
Level 6

detected an issue with SFS NAS cluster and found that filesystems a cannot be confirmed on df, and general users cannot login to the node.

 

 

SFS event log:->

 

84441) 2013 Jun 19 17:42:39 kyornas051_01 sfsfs_event.network.alert: Node kyornas051_02 went offline.
84442) 2013 Jun 19 17:45:01 kyornas051_01 sfsfs_event.storage.alert.master: Slave node(s) successfully fenced out from cluster.
84443) 2013 Jun 19 18:19:49 kyornas051_01 sfsfs_event.network.info: Node kyornas051_01 joined the cluster.
84444) 2013 Jun 19 18:19:49 192.168.13.159 sfsfs_event.network.info: Node kyornas051_01 joined the cluster.
84445) 2013 Jun 19 18:21:49 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84446) 2013 Jun 19 18:22:36 kyornas051_01 sfsfs_event.network.info.master: ip addr show successful
84447) 2013 Jun 19 18:23:40 kyornas051_01 sfsfs_event.network.info.master: services show successful
84448) 2013 Jun 19 18:24:20 kyornas051_01 sfsfs_event.network.alert: Interface bond0 is down on system kyornas051_02.
84449) 2013 Jun 19 18:25:09 kyornas051_01 sfsfs_event.network.info.master: IO Fencing Status: Enabled with SCSI3 Persistent Reservations
84450) 2013 Jun 19 18:25:15 kyornas051_01 sfsfs_event.network.info.master: Checked status of IO Fencing on the coordinator disks
84451) 2013 Jun 19 18:25:22 kyornas051_01 sfsfs_event.network.alert: Interface bond0 is up on system kyornas051_02.
84452) 2013 Jun 19 18:25:26 kyornas051_02 sfsfs_event.network.info: Node kyornas051_02 joined the cluster.
84453) 2013 Jun 19 18:27:01 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84454) 2013 Jun 19 18:27:21 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84455) 2013 Jun 19 18:27:56 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84456) 2013 Jun 19 18:35:05 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful

==================================================================

 

message file

 

 


message 51_01

2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 1 (priveth1) node 1 in trouble
2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 0 (priveth0) node 1 in trouble
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 8 sec (1698350566)
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 8 sec (1698356198)
2013 Jun 19 17:42:27 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 9 sec (1698350566)

then it took sometime for llt to expire

013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 1 (priveth1) node 1 expired
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 0 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port h gen  1132317 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port v gen  113231a membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port w gen  113231c membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port a gen  1132305 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port b gen  1132314 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port f gen  113231e membership 0
2013 Jun 19 17:42:38 kyornas051_01 Had[30829]: VCS INFO V-16-1-10077 Received new cluster membership
2013 Jun 19 17:42:38 kyornas051_01 kernel: VXFEN INFO V-11-1-68 Completed ejection of leaving node(s) from data disks.
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-7899 CVM_VOLD_CHANGE command received
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-13170 Preempting CM NID 1
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-0 Calling join complete
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-8062 master: not a cluster startup
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-10994 join completed for node 0
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-4123 cluster established successfully
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED
2013 Jun 19 17:42:39 kyornas051_01 sfsfs_event.network.alert: Node kyornas051_02 went offline.



2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 617 Phase 0 - /dev/vx/dsk/sfsdg/_nlm_ - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0
2013 Jun 19 17:42:41 kyornas051_01 kernel:
2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 618 Phase 0 - /dev/vx/dsk/sfsdg/oss2a-segment1 - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0
2013 Jun 19 17:42:41 kyornas051_01 kernel:
2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 619 Phase 0 - /dev/vx/dsk/sfsdg/oss2a-sgwcg - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0

2013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 695 - /dev/vx/dsk/sfsdg/_nlm_ - PNOLT bitmap in vrt: 0x3 0x0 0x0 0x0, Log replay for per node logs: 0x1 0x0 0x0 0x0
013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 724 Phase 2 - /dev/vx/dsk/sfsdg/_nlm_ - Buffer reads allowed
2013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 726 Phase 9 - /dev/vx/dsk/sfsdg/_nlm_ - Set Primary nodeid to 0
2013 Jun 19 17:42:43 kyornas051_01 kernel:
2013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 727 Phase 10 -/dev/vx/dsk/sfsdg/_nlm_ - Processing extended operations.
2013 Jun 19 17:42:43 kyornas051_01 kernel:

2013 Jun 19 17:43:06 kyornas051_01 kernel: nfsd: last server has exited
2013 Jun 19 17:43:06 kyornas051_01 kernel: nfsd: unexporting all filesystems
2013 Jun 19 17:43:06 kyornas051_01 rpc.mountd: Caught signal 15, un-registering and exiting.
2013 Jun 19 17:43:07 kyornas051_01 AgentFramework[30925]: VCS ERROR V-16-1-13067 Thread(4146072480) Agent is calling clean for resource(nasgw_nfs) because the resource became OFFLINE unexpectedly, on its own.
2013 Jun 19 17:43:07 kyornas051_01 Had[30829]: VCS ERROR V-16-1-13067 (kyornas051_01) Agent is calling clean for resource(nasgw_nfs) because the resource became OFFLINE unexpectedly, on its own.
2013 Jun 19 17:43:08 kyornas051_01 kernel: nfsd: failed to unregister export cache
2013 Jun 19 17:43:08 kyornas051_01 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
2013 Jun 19 17:43:08 kyornas051_01 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
2013 Jun 19 17:43:08 kyornas051_01 kernel: NFSD: starting 90-second grace period
2013 Jun 19 17:43:08 kyornas051_01 logger: event_notify.sh will insert a syslog marker now!
2013 Jun 19 17:43:08 kyornas051_01 sfsfs_event.marker.info: =============== Event Log Marker ===============
2013 Jun 19 17:43:14 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:43:15 kyornas051_01 logger: 2013 Jun 19 17:42:39 [kyornas051_01,alert,sfs] Node kyornas051_02 went offline.
2013 Jun 19 17:43:37 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:43:59 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:44:08 kyornas051_01 AgentFramework[30925]: VCS ERROR V-16-1-13006 Thread(4143971232) Resource(nasgw_nfs): clean procedure did not complete within the expected time.
2013 Jun 19 17:44:08 kyornas051_01 Had[30829]: VCS ERROR V-16-1-13006 (kyornas051_01) Resource(nasgw_nfs): clean procedure did not complete within the expected time.
2013 Jun 19 17:44:22 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:45:01 kyornas051_01 /usr/sbin/cron[25755]: (root) CMD (LOCKDISABLED=LOCKDISABLED /opt/VRTSnasgw/scripts/rrdtool.pl update >/dev/null 2>&1)
2013 Jun 19 17:45:01 kyornas051_01 /usr/sbin/cron[25773]: (root) CMD (/opt/VRTSnasgw/scripts/sav_event_notify.sh addevents >/dev/null 2>/dev/null)
2013 Jun 19 17:45:01 kyornas051_01 sfsfs_event.storage.alert.master: Slave node(s) successfully fenced out from cluster.
 

=============================================================================================

 

engine log

 

2013/06/19 17:42:33 VCS WARNING V-16-1-11155 LLT heartbeat link status changed. Previous status = 0x3; Current status = 0x1.
2013/06/19 17:42:38 VCS INFO V-16-1-10077 Received new cluster membership
2013/06/19 17:42:38 VCS NOTICE V-16-1-10112 System (kyornas051_01) - Membership: 0x1, DDNA: 0x0
2013/06/19 17:42:38 VCS NOTICE V-16-1-10034 RECONFIG received. VCS waiting for I/O fencing to be completed
2013/06/19 17:42:39 VCS NOTICE V-16-1-10036 I/O fencing completed
2013/06/19 17:42:39 VCS ERROR V-16-1-10079 System kyornas051_02 (Node '1') is in Down State - Membership: 0x1
2013/06/19 17:42:39 VCS ERROR V-16-1-10322 System kyornas051_02 (Node '1') changed state from RUNNING to FAULTED
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group NFS is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group Phantomgroup_priveth0 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup1 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup10 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup11 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup12 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup2 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup3 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount1 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount2 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount3 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount4 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount5 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount6 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount7 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount8 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount9 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount10 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount11 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount12 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount13 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount14 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount15 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount16 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount17 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount18 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount19 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount20 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount21 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount22 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount23 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount24 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount25 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount26 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount27 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount28 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount29 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group CanHostConsole is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group CanHostNLM is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group Phantomgroup_bond0 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group cvm is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group ReconfigGroup is offline on system kyornas051_02
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group NFS
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group NFS is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group NFS
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group Phantomgroup_priveth0
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group Phantomgroup_priveth0 is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group Phantomgroup_priveth0
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup1
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup1
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup10
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup10
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup11
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup11
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup12
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup12
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup2
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup2
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup3
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup3
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group CanHostConsole
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group CanHostConsole is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group CanHostConsole
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group CanHostNLM
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group CanHostNLM is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group CanHostNLM
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group Phantomgroup_bond0
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group Phantomgroup_bond0 is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group Phantomgroup_bond0
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group ReconfigGroup
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group ReconfigGroup is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group ReconfigGroup
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:40 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup10  from localhost
2013/06/19 17:42:40 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup11  from localhost
2013/06/19 17:42:41 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup3  from localhost
2013/06/19 17:42:42 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup1  from localhost
2013/06/19 17:42:42 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup12  from localhost
2013/06/19 17:42:42 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup2  from localhost
2013/06/19 17:42:44 VCS WARNING V-16-1-11155 LLT heartbeat link status changed. Previous status = 0x1; Current status = 0x0.
2013/06/19 17:42:44 VCS INFO V-16-1-50135 User root fired command: hagrp -online CanHostConsole  kyornas051_01  from localhost
2013/06/19 17:42:44 VCS INFO V-16-1-50135 User root fired command: hagrp -online CanHostNLM  kyornas051_01  from localhost
2013/06/19 17:42:44 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/sysoffline kyornas051_02 FAULTED   successfully
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup11  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup11 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP11 of group VIPgroup11 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup11 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup11 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP11 (Owner: unknown, Group: VIPgroup11) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup10  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup10 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP10 of group VIPgroup10 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup10 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup10 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP10 (Owner: unknown, Group: VIPgroup10) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup12  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup12 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP12 of group VIPgroup12 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup12 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup12 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP12 (Owner: unknown, Group: VIPgroup12) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup11 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup1  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup1 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP1 of group VIPgroup1 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup1 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup1 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP1 (Owner: unknown, Group: VIPgroup1) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup2  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup2 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP2 of group VIPgroup2 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup2 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup2 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP2 (Owner: unknown, Group: VIPgroup2) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup3  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup3 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP3 of group VIPgroup3 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup3 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup3 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP3 (Owner: unknown, Group: VIPgroup3) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup10 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup12 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup1 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup2 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup3 MANUAL  successfully
2013/06/19 17:42:59 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount28  Primary  kyornas051_01  from localhost
2013/06/19 17:43:01 VCS INFO V-16-1-10298 Resource VIP1 (Owner: unknown, Group: VIPgroup1) is online on kyornas051_01 (VCS initiated)
2013/06/19 17:43:01 VCS NOTICE V-16-1-10447 Group VIPgroup1 is online on system kyornas051_01
2013/06/19 17:43:05 VCS INFO V-16-1-10298 Resource VIP12 (Owner: unknown, Group: VIPgroup12) is online on kyornas051_01 (VCS initiated)
2013/06/19 17:43:05 VCS NOTICE V-16-1-10447 Group VIPgroup12 is online on system kyornas051_01
2013/06/19 17:43:06 VCS INFO V-16-1-50135 User root fired command: hares -clear VIP1  from localhost
2013/06/19 17:43:06 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE nasgw_nfs  kyornas051_01  from localhost
2013/06/19 17:43:06 VCS WARNING V-16-10031-7017 (kyornas051_01) NFS:nasgw_nfs:monitor:nfsd filesystem not mounted, returning offline
2013/06/19 17:43:07 VCS ERROR V-16-2-13067 (kyornas051_01) Agent is calling clean for resource(nasgw_nfs) because the resource became OFFLINE unexpectedly, on its own.
2013/06/19 17:43:10 VCS INFO V-16-1-50135 User root fired command: hares -clear VIP12  from localhost
2013/06/19 17:43:10 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postonline kyornas051_01 VIPgroup12   successfully
2013/06/19 17:43:12 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount22  Primary  kyornas051_01  from localhost
2013/06/19 17:43:18 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount29  Primary  kyornas051_01  from localhost
2013/06/19 17:43:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount23  Primary  kyornas051_01  from localhost
2013/06/19 17:43:37 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount24  Primary  kyornas051_01  from localhost
2013/06/19 17:43:50 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount25  Primary  kyornas051_01  from localhost
2013/06/19 17:44:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(NicMonitor_bond0): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:44:08 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(nasgw_nfs): clean procedure did not complete within the expected time.
2013/06/19 17:44:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postonline kyornas051_01 VIPgroup1   successfully
2013/06/19 17:45:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(NicMonitor_bond0): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:47:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(canHostConsole_consoleNIC): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP10): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP10) because online did not complete within the expected time.
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP11): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP2): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP2) because online did not complete within the expected time.
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP3): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP11) because online did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP3) because online did not complete within the expected time.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP10) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP11) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP3) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP2) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP10): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP11): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP2): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP3): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:48:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(canHostConsole_consoleNIC): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:49:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(canHostConsole_consoleNIC): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_priveth1) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(nlmmasterNIC) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(canHostConsole_consoleNIC) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_bond0) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth1) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth4) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(canHostNLM_nlmmaster_device) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth3) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth2) - monitor procedure did not complete within the expected time.
2013/06/19 17:51:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_priveth0) - monitor procedure did not complete within the expected time.
2013/06/19 17:51:08 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(nasgw_smweb) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(consoleNIC) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth0) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(phantomNIC_bond0) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth5) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP2): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP3): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP11): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP10): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP11) because online did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP10) because online did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP2) because online did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP3) because online did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP10): clean procedure did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP3): clean procedure did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP11): clean procedure did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP2): clean procedure did not complete within the expected time.
2013/06/19 17:57:08 VCS ERROR V-16-2-13210 (kyornas051_01) Agent is calling clean for resource(nasgw_smweb) because 4 successive invocations of the monitor procedure did not complete within the expected time.
2013/06/19 17:57:09 VCS INFO V-16-2-13068 (kyornas051_01) Resource(nasgw_smweb) - clean completed successfully.
2013/06/19 17:58:10 VCS ERROR V-16-2-13077 (kyornas051_01) Agent is unable to offline resource(nasgw_smweb). Administrative intervention may be required.
2013/06/19 18:18:42 VCS NOTICE V-16-1-11022 VCS engine (had) started
2013/06/19 18:18:42 VCS INFO V-16-1-10196 Cluster logger started
2013/06/19 18:18:42 VCS NOTICE V-16-1-11050 VCS engine version=5.0
2013/06/19 18:18:42 VCS NOTICE V-16-1-11051 VCS engine join version=5.0.40.0
2013/06/19 18:18:42 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.0MP4HF2-08/02/10-23:10:00
2013/06/19 18:18:42 VCS NOTICE V-16-1-10114 Opening GAB library
2013/06/19 18:18:42 VCS NOTICE V-16-1-10619 'HAD' starting on: kyornas051_01
2013/06/19 18:18:42 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms
2013/06/19 18:18:46 VCS INFO V-16-1-10077 Received new cluster membership
2013/06/19 18:18:46 VCS NOTICE V-16-1-10112 System (kyornas051_01) - Membership: 0x1, DDNA: 0x0
2013/06/19 18:18:46 VCS NOTICE V-16-1-10086 System kyornas051_01 (Node '0') is in Regular Membership - Membership: 0x1
2013/06/19 18:18:46 VCS NOTICE V-16-1-10322 System kyornas051_01 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2013/06/19 18:18:48 VCS NOTICE V-16-1-10032 VxFEN driver available. Local node id=0
2013/06/19 18:18:48 VCS NOTICE V-16-1-10322 System kyornas051_01 (Node '0') changed state from LOCAL_BUILD to RUNNING
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSMount/CFSMountAgent for resource type CFSMount successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSfsckd/CFSfsckdAgent for resource type CFSfsckd successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMCluster/CVMClusterAgent for resource type CVMCluster successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVolDg/CVMVolDgAgent for resource type CVMVolDg successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVxconfigd/CVMVxconfigdAgent for resource type CVMVxconfigd successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NFS/NFSAgent for resource type NFS successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NFSRestart/NFSRestartAgent for resource type NFSRestart successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NetBios/NetBiosAgent for resource type NetBios successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Phantom/PhantomAgent for resource type Phantom successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Proxy/ProxyAgent for resource type Proxy successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/SambaServer/SambaServerAgent for resource type SambaServer successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Share/ShareAgent for resource type Share successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed Jun 19 18:18:48 2013

 

==================================================

 

I can see that one of the node 51_02 got fenced out due to issue with hearbeat.

But I could not understand , why there was an issue with filesystems. why they were affected by this as this is a parallel cluster

 

 

 

 

1 REPLY 1

Kimberley
Level 6
Partner

Moved to VCS forum.