#
# Log Name:     engine
# System:       cms-app-49-52.workgroup
# SysInfo:      Linux:cms-app-49-52.workgroup,#1 SMP Thu Jul 5 21:08:58 EDT 2012,2.6.32-279.2.1.el6.x86_64,x86_64
# Created:      2012/12/05 17:13:22 
#

2012/12/05 17:13:22 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/05 17:13:22 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/05 17:13:22 VCS INFO V-16-1-10196 Cluster logger started
2012/12/05 17:13:22 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/05 17:13:22 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/05 17:13:22 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/05 17:13:22 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-52
2012/12/05 17:13:22 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/05 17:13:22 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/05 17:13:23 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/05 17:13:23 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/05 17:13:23 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/05 17:13:27 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/05 17:13:27 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x1
2012/12/05 17:13:27 VCS NOTICE V-16-1-10086 System cms-app-49-52 (Node '1') is in Regular Membership - Membership: 0x2
2012/12/05 17:13:27 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2012/12/05 17:13:28 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/05 17:13:28 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from LOCAL_BUILD to RUNNING
2012/12/05 17:13:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed Dec  5 17:13:28 2012

2012/12/05 17:13:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Wed Dec  5 17:13:28 2012

2012/12/05 17:13:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed Dec  5 17:13:28 2012

2012/12/05 17:13:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Wed Dec  5 17:13:28 2012

2012/12/05 17:13:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed Dec  5 17:13:28 2012

2012/12/05 17:13:28 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/05 17:13:28 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/05 17:13:28 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
PWD=/var/VRTSvcs/diag/had
SHLVL=4
TERM=xterm
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/05 17:13:28 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-52
2012/12/05 17:13:28 VCS NOTICE V-16-1-10439 Group (CMSApp_Notifier) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system cms-app-49-52 after probe
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:28 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/05 17:13:29 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-52
2012/12/05 17:13:29 VCS NOTICE V-16-1-10439 Group (CMSApp_Cluster) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system cms-app-49-52 after probe
2012/12/05 17:13:35 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/05 17:13:35 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/05 17:13:35 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/05 17:13:35 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-52
2012/12/05 17:13:35 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-52 as the system is not a part of AutoStartList attribute of the group.
2012/12/05 17:13:35 VCS NOTICE V-16-1-10453 Node: 0 changed name from: 'cms-app-49-51' to: 'cms-app-49-51'
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from UNKNOWN to INITING
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/05 17:13:35 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0x1
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/12/05 17:13:35 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
PWD=/var/VRTSvcs/diag/had
SHLVL=4
TERM=xterm
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/05 17:13:35 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/05 17:13:35 VCS INFO V-16-1-50007 Initiating auto-start online of group CMSApp_Notifier
2012/12/05 17:13:35 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/05 17:13:35 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Notifier on all nodes
2012/12/05 17:13:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-51
2012/12/05 17:13:36 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:36 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-51
2012/12/05 17:13:36 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/05 17:13:36 VCS INFO V-16-1-50007 Initiating auto-start online of group CMSApp_Cluster
2012/12/05 17:13:36 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/05 17:13:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/05 17:13:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:37 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/05 17:13:37 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/05 17:13:44 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:45 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 17:13:45 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/05 17:13:45 VCS INFO V-16-10031-504 (cms-app-49-51) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/05 17:13:46 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/05 17:13:46 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/05 17:13:46 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [  OK  ]
==============================================

2012/12/05 17:13:46 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:46 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:46 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:50 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/05 17:13:51 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/05 17:13:51 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:53 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/05 17:13:54 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-51 to cms-app-49-52 in 5 seconds intervals
==============================================

2012/12/05 17:13:54 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:56 VCS INFO V-16-10031-504 (cms-app-49-51) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/05 17:13:56 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  12215) is running...
settings server (pid  12224) is running...
alerts (pid  12620) is running...
==============================================

2012/12/05 17:13:56 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:58 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/05 17:13:59 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/05 17:13:59 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:59 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/05 17:31:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:31:37 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:offline:Executed /etc/init.d/tomcat as user root
2012/12/05 17:31:37 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (offline) 
==============================================
Stopping Tomcat
==============================================

2012/12/05 17:31:37 VCS INFO V-16-1-10305 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/05 17:31:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/05 17:31:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:31:41 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 17:31:42 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/05 17:31:42 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:31:42 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/05 17:39:13 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/05 17:39:13 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x0
2012/12/05 17:39:13 VCS ERROR V-16-1-10079 System cms-app-49-51 (Node '0') is in Down State - Membership: 0x2
2012/12/05 17:39:13 VCS ERROR V-16-1-10322 System cms-app-49-51 (Node '0') changed state from RUNNING to FAULTED
2012/12/05 17:39:13 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-51
2012/12/05 17:39:13 VCS NOTICE V-16-1-10446 Group CMSApp_Notifier is offline on system cms-app-49-51
2012/12/05 17:39:13 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/05 17:39:13 VCS INFO V-16-1-10494 System cms-app-49-51 not in RUNNING state
2012/12/05 17:39:13 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/05 17:39:13 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/05 17:39:13 VCS INFO V-16-1-10494 System cms-app-49-51 not in RUNNING state
2012/12/05 17:39:13 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Notifier
2012/12/05 17:39:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-52
2012/12/05 17:39:14 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:14 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-52
2012/12/05 17:39:19 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 UP bond1 UP; Current status = bond0 DOWN bond1 DOWN.
2012/12/05 17:39:22 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:22 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/05 17:39:22 VCS INFO V-16-10031-504 (cms-app-49-52) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/05 17:39:22 VCS INFO V-16-10031-504 (cms-app-49-52) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 17:39:23 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/05 17:39:23 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/05 17:39:23 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [  OK  ]
==============================================

2012/12/05 17:39:23 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:23 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:23 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:27 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/05 17:39:28 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/05 17:39:28 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:30 VCS INFO V-16-10031-504 (cms-app-49-52) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/05 17:39:30 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  19268) is running...
settings server (pid  19318) is running...
alerts (pid  19641) is running...
==============================================

2012/12/05 17:39:30 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/05 17:39:30 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 17:39:31 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-52 to cms-app-49-51 in 5 seconds intervals
==============================================

2012/12/05 17:39:31 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:32 VCS INFO V-16-10031-504 (cms-app-49-52) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/05 17:39:33 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/05 17:39:33 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 17:39:33 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-52
2012/12/05 17:50:52 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 DOWN bond1 DOWN; Current status = bond0 UP bond1 UP.
2012/12/05 17:51:00 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/05 17:51:00 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/05 17:51:00 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/05 17:51:00 VCS NOTICE V-16-1-10449 Group CMSApp_Cluster autodisabled on node cms-app-49-51 until it is probed
2012/12/05 17:51:00 VCS NOTICE V-16-1-10449 Group CMSApp_Notifier autodisabled on node cms-app-49-51 until it is probed
2012/12/05 17:51:00 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on node cms-app-49-51 until it is probed
2012/12/05 17:51:00 VCS NOTICE V-16-1-10453 Node: 0 changed name from: 'cms-app-49-51' to: 'cms-app-49-51'
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from FAULTED to INITING
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/05 17:51:00 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0x1
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/05 17:51:01 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/05 17:51:01 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:02 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/05 17:51:03 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/05 17:51:03 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/05 18:34:23 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(RCSSchedulerService) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:23 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(xportmanager) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:23 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(tomcat) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:23 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:tomcat:clean:PidFile </etc/init.d/tomcat.pid> does not exist, process will not be killed
2012/12/05 18:34:23 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:xportmanager:clean:PidFile </etc/init.d/xportmanager.pid> does not exist, process will not be killed
2012/12/05 18:34:23 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:RCSSchedulerService:clean:PidFile </usr/local/tstv/RCSScheduler/config/rcsScheduler.pid> does not exist, process will not be killed
2012/12/05 18:34:24 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(RCSSchedulerService) - clean completed successfully.
2012/12/05 18:34:24 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(tomcat) - clean completed successfully.
2012/12/05 18:34:24 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(xportmanager) - clean completed successfully.
2012/12/05 18:34:24 VCS INFO V-16-1-10307 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS INFO V-16-1-10305 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:24 VCS INFO V-16-1-10305 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:24 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/05 18:34:24 VCS INFO V-16-1-10306 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/05 18:34:27 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/05 18:34:28 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(autorsync): Output of the completed operation (offline) 
==============================================
Stopping autorsync
==============================================

2012/12/05 18:34:28 VCS INFO V-16-1-10305 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:30 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xporttomcat:offline:Executed /etc/init.d/xporttomcat as user root
2012/12/05 18:34:30 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(alerts) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:30 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:alerts:clean:PidFile </etc/init.d/alerts.pid> does not exist, process will not be killed
2012/12/05 18:34:30 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:alerts:clean:PidFile </etc/init.d/alerts-nameservice.pid> does not exist, process will not be killed
2012/12/05 18:34:30 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:alerts:clean:PidFile </etc/init.d/alerts-settingserver.pid> does not exist, process will not be killed
2012/12/05 18:34:30 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xporttomcat): Output of the completed operation (offline) 
==============================================
Stopping XportTomcat
==============================================

2012/12/05 18:34:30 VCS INFO V-16-1-10305 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:31 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(alerts) - clean completed successfully.
2012/12/05 18:34:31 VCS INFO V-16-1-10307 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/05 18:34:33 VCS INFO V-16-10031-504 (cms-app-49-52) Application:cms:offline:Executed /etc/init.d/cms as user root
2012/12/05 18:34:33 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(cms): Output of the completed operation (offline) 
==============================================
Stopping Content Management System
Stopping ADI NameServer
==============================================

2012/12/05 18:34:33 VCS INFO V-16-1-10305 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:35 VCS INFO V-16-1-10305 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:35 VCS ERROR V-16-1-10205 Group CMSApp_Cluster is faulted on system cms-app-49-52
2012/12/05 18:34:35 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-52
2012/12/05 18:34:35 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/05 18:34:35 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/05 18:34:35 VCS INFO V-16-1-50010 Group CMSApp_Cluster is online or faulted on system cms-app-49-52
2012/12/05 18:34:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 18:34:43 VCS INFO V-16-10031-504 (cms-app-49-51) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/05 18:34:43 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/05 18:34:44 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/05 18:34:44 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/05 18:34:44 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [60G[[0;32m  OK  [0;39m]
==============================================

2012/12/05 18:34:44 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:44 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:44 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:48 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/05 18:34:49 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/05 18:34:49 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:51 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/05 18:34:52 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-51 to cms-app-49-52 in 5 seconds intervals
==============================================

2012/12/05 18:34:52 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:55 VCS INFO V-16-10031-504 (cms-app-49-51) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/05 18:34:55 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  18488) is running...
settings server (pid  18524) is running...
alerts (pid  18913) is running...
==============================================

2012/12/05 18:34:55 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:58 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/05 18:34:58 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/05 18:34:58 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:58 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/05 18:34:58 VCS NOTICE V-16-1-10448 Group CMSApp_Cluster failed over to system cms-app-49-51
2012/12/05 18:41:46 VCS INFO V-16-1-50135 User root fired command: hares -offline xportmanager  cms-app-49-52  from localhost
2012/12/05 18:42:26 VCS INFO V-16-1-50135 User root fired command: hares -clear xportmanager  cms-app-49-52  from localhost
2012/12/05 18:42:26 VCS INFO V-16-1-10307 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/05 18:42:36 VCS INFO V-16-1-50135 User root fired command: hares -clear alerts  cms-app-49-52  from localhost
2012/12/05 18:42:36 VCS INFO V-16-1-10307 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/05 19:45:05 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 UP bond1 UP; Current status = bond0 DOWN bond1 DOWN.
2012/12/05 19:45:08 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/05 19:45:08 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x0
2012/12/05 19:45:08 VCS ERROR V-16-1-10079 System cms-app-49-51 (Node '0') is in Down State - Membership: 0x2
2012/12/05 19:45:08 VCS ERROR V-16-1-10322 System cms-app-49-51 (Node '0') changed state from RUNNING to FAULTED
2012/12/05 19:45:08 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-51
2012/12/05 19:45:08 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/05 19:45:08 VCS INFO V-16-1-10494 System cms-app-49-51 not in RUNNING state
2012/12/05 19:45:08 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/05 19:45:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:17 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:17 VCS INFO V-16-10031-504 (cms-app-49-52) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 19:45:17 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/05 19:45:17 VCS INFO V-16-10031-504 (cms-app-49-52) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/05 19:45:17 VCS INFO V-16-10031-504 (cms-app-49-52) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/05 19:45:18 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xportmanager): Output of the completed operation (online) 
==============================================
XportManager is already running...
==============================================

2012/12/05 19:45:18 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(tomcat): Output of the completed operation (online) 
==============================================
Tomcat is already running...
==============================================

2012/12/05 19:45:18 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(alerts): Output of the completed operation (online) 
==============================================
alerts is already running...
==============================================

2012/12/05 19:45:18 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [  OK  ]
==============================================

2012/12/05 19:45:18 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:18 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:18 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:18 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 19:45:20 VCS INFO V-16-10031-504 (cms-app-49-52) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/05 19:45:21 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/05 19:45:21 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:22 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/05 19:45:23 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/05 19:45:23 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:25 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/05 19:45:26 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-52 to cms-app-49-51 in 5 seconds intervals
==============================================

2012/12/05 19:45:26 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/05 19:45:26 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-52
2012/12/05 20:08:34 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/12/05 20:47:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 65%
2012/12/05 20:56:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/05 21:03:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 61%
2012/12/05 21:09:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 66%
2012/12/05 21:10:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 66%
2012/12/05 23:06:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/05 23:35:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 61%
2012/12/06 00:08:35 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/12/06 00:13:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 63%
2012/12/06 00:38:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 71%
2012/12/06 00:40:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 63%
2012/12/06 00:42:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 63%
2012/12/06 00:43:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 73%
2012/12/06 00:45:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 71%
2012/12/06 00:46:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 79%
2012/12/06 00:48:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 77%
2012/12/06 00:50:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 64%
2012/12/06 00:55:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 64%
2012/12/06 00:58:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 70%
2012/12/06 01:00:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 64%
2012/12/06 01:01:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 61%
2012/12/06 01:06:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 61%
2012/12/06 01:11:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 64%
2012/12/06 01:16:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 67%
2012/12/06 01:19:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 63%
2012/12/06 01:20:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 64%
2012/12/06 01:22:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 66%
2012/12/06 03:02:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 03:05:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 71%
2012/12/06 03:35:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 66%
2012/12/06 04:02:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 63%
2012/12/06 04:05:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 04:08:36 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/12/06 04:34:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 61%
2012/12/06 05:08:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 05:18:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 63%
2012/12/06 05:19:04 VCS NOTICE V-16-1-50086 CPU usage on cms-app-49-52 is 79%
2012/12/06 05:20:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 64%
2012/12/06 05:22:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 66%
2012/12/06 05:46:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 62%
2012/12/06 06:02:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 07:36:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 62%
2012/12/06 08:05:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 08:08:37 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/12/06 08:35:34 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 09:02:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 09:34:04 VCS INFO V-16-1-50086 CPU usage on cms-app-49-52 is 60%
2012/12/06 10:57:41 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 10:57:41 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x1
2012/12/06 10:57:41 VCS ERROR V-16-1-10113 System cms-app-49-51 (Node '0') is in DDNA Membership - Membership: 0x2, Visible: 0x0
2012/12/06 10:57:41 VCS NOTICE V-16-1-10449 Group CMSApp_Cluster autodisabled on node cms-app-49-51 until it is probed
2012/12/06 10:57:41 VCS NOTICE V-16-1-10449 Group CMSApp_Notifier autodisabled on node cms-app-49-51 until it is probed
2012/12/06 10:57:41 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on node cms-app-49-51 until it is probed
2012/12/06 10:57:43 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 DOWN bond1 DOWN; Current status = bond0 UP bond1 UP.
2012/12/06 10:57:48 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 10:57:48 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/06 10:57:48 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 10:57:48 VCS NOTICE V-16-1-10453 Node: 0 changed name from: 'cms-app-49-51' to: 'cms-app-49-51'
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from FAULTED to INITING
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 10:57:48 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0x1
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/12/06 10:57:48 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:48 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 10:57:48 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/06 10:57:48 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:48 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/06 10:57:50 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/06 10:57:50 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 11:37:14 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/06 11:37:14 VCS INFO V-16-1-10196 Cluster logger started
2012/12/06 11:37:14 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/06 11:37:14 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/06 11:37:14 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/06 11:37:14 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/06 11:37:14 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-52
2012/12/06 11:37:14 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/06 11:37:14 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/06 11:37:14 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/06 11:37:14 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/06 11:37:14 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 11:37:19 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 11:37:19 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/06 11:37:19 VCS NOTICE V-16-1-10086 System  (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 11:37:19 VCS NOTICE V-16-1-10086 System cms-app-49-52 (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 11:37:19 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 11:37:19 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 11:37:19 VCS NOTICE V-16-1-10453 Node: 0 changed name from: '' to: 'cms-app-49-51'
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to RUNNING
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 11:37:19 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 0
2012/12/06 11:37:19 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 11:37:19 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/06 11:37:19 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/06 11:37:19 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/06 11:37:19 VCS INFO V-16-1-10466 End of snapshot received from node: 0.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 11:37:19 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/06 11:37:19 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 11:37:19 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu Dec  6 11:37:19 2012

2012/12/06 11:37:19 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Thu Dec  6 11:37:19 2012

2012/12/06 11:37:19 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu Dec  6 11:37:19 2012

2012/12/06 11:37:19 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Thu Dec  6 11:37:19 2012

2012/12/06 11:37:19 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Dec  6 11:37:19 2012

2012/12/06 11:37:19 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/06 11:37:19 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-52
2012/12/06 11:37:20 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 11:37:20 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10297 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group CMSApp_Cluster
2012/12/06 11:37:20 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS WARNING V-16-6-15034 (cms-app-49-52) violation:Offlining group CMSApp_Cluster on system cms-app-49-52
2012/12/06 11:37:20 VCS INFO V-16-1-50135 User root fired command: hagrp -offline CMSApp_Cluster  cms-app-49-52  from localhost
2012/12/06 11:37:20 VCS NOTICE V-16-1-10167 Initiating manual offline of group CMSApp_Cluster on system cms-app-49-52
2012/12/06 11:37:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 11:37:20 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/violation cms-app-49-52 CMSApp_Cluster   successfully
2012/12/06 11:37:20 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-52
2012/12/06 11:37:22 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-52
2012/12/06 11:37:22 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-52 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 11:37:23 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/06 11:37:24 VCS ERROR V-16-2-13064 (cms-app-49-52) Agent is calling clean for resource(autorsync) because the resource is up even after offline completed.
2012/12/06 11:37:24 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:37:27 VCS ERROR V-16-2-13069 (cms-app-49-52) Resource(autorsync) - clean failed.
2012/12/06 11:38:28 VCS ERROR V-16-2-13077 (cms-app-49-52) Agent is unable to offline resource(autorsync). Administrative intervention may be required.
2012/12/06 11:38:28 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:39:31 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:40:42 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 11:40:42 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x0
2012/12/06 11:40:42 VCS ERROR V-16-1-10079 System cms-app-49-51 (Node '0') is in Down State - Membership: 0x2
2012/12/06 11:40:42 VCS ERROR V-16-1-10322 System cms-app-49-51 (Node '0') changed state from RUNNING to FAULTED
2012/12/06 11:40:42 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-51
2012/12/06 11:40:42 VCS NOTICE V-16-1-10446 Group CMSApp_Notifier is offline on system cms-app-49-51
2012/12/06 11:40:42 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/06 11:40:42 VCS INFO V-16-1-10494 System cms-app-49-51 not in RUNNING state
2012/12/06 11:40:42 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Notifier
2012/12/06 11:40:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-52
2012/12/06 11:40:42 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:40:42 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-52 (VCS initiated)
2012/12/06 11:40:42 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-52
2012/12/06 11:40:48 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 UP bond1 UP; Current status = bond0 DOWN bond1 DOWN.
2012/12/06 11:41:45 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:42:16 VCS INFO V-16-1-50135 User root fired command: hares -online cms  cms-app-49-52  from localhost
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource RCSSchedulerService of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource alerts of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource autorsync of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource cms of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource tomcat of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource xportmanager of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource xporttomcat of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10460 Clearing start attribute for resource Virtual_IP of group CMSApp_Cluster on node cms-app-49-51
2012/12/06 11:42:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 11:42:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 11:42:24 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 11:42:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 11:42:34 VCS INFO V-16-10031-504 (cms-app-49-52) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/06 11:42:35 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  4792) is running...
settings server (pid  4796) is running...
alerts (pid  5209) is running...
==============================================

2012/12/06 11:42:35 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 11:42:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 11:42:37 VCS INFO V-16-10031-504 (cms-app-49-52) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/06 11:42:38 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/06 11:42:38 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 11:42:48 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:43:51 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:44:54 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:45:57 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:47:01 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:47:04 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 10 invocations of the clean procedure have failed.
2012/12/06 11:48:04 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:49:08 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:50:11 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:51:14 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:52:17 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:53:20 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:54:23 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:55:26 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:56:29 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:57:32 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:57:35 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 20 invocations of the clean procedure have failed.
2012/12/06 11:58:36 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:59:39 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:00:42 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:01:45 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:02:48 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:03:51 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:04:54 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:05:15 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/12/06 12:05:15 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from RUNNING to LEAVING
2012/12/06 12:05:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-52
2012/12/06 12:05:16 VCS INFO V-16-1-10305 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 12:05:16 VCS NOTICE V-16-1-10446 Group CMSApp_Notifier is offline on system cms-app-49-52
2012/12/06 12:05:16 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/06 12:05:16 VCS INFO V-16-1-10494 System cms-app-49-51 not in RUNNING state
2012/12/06 12:05:16 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Notifier
2012/12/06 12:05:16 VCS INFO V-16-1-10494 System cms-app-49-52 not in RUNNING state
2012/12/06 12:05:16 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=1) for group CMSApp_Notifier for node cms-app-49-52
2012/12/06 12:05:57 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:06:27 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/12/06 12:07:00 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:07:05 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:07:05 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x1
2012/12/06 12:07:05 VCS ERROR V-16-1-10113 System cms-app-49-51 (Node '0') is in DDNA Membership - Membership: 0x2, Visible: 0x0
2012/12/06 12:07:05 VCS NOTICE V-16-1-10449 Group CMSApp_Cluster autodisabled on node cms-app-49-51 until it is probed
2012/12/06 12:07:05 VCS NOTICE V-16-1-10449 Group CMSApp_Notifier autodisabled on node cms-app-49-51 until it is probed
2012/12/06 12:07:05 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on node cms-app-49-51 until it is probed
2012/12/06 12:07:13 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:07:13 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/06 12:07:13 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/06 12:07:13 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 12:07:13 VCS NOTICE V-16-1-10453 Node: 0 changed name from: 'cms-app-49-51' to: 'cms-app-49-51'
2012/12/06 12:07:13 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from FAULTED to INITING
2012/12/06 12:07:13 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/06 12:07:13 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to ADMIN_WAIT
2012/12/06 12:08:04 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:08:07 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 30 invocations of the clean procedure have failed.
2012/12/06 12:09:08 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:10:11 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:11:01 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from ADMIN_WAIT to EXITED
2012/12/06 12:11:06 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:11:06 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x1
2012/12/06 12:11:06 VCS ERROR V-16-1-10113 System cms-app-49-51 (Node '0') is in DDNA Membership - Membership: 0x2, Visible: 0x0
2012/12/06 12:11:14 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:12:17 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:13:20 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:14:23 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:15:26 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:16:29 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:16:45 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:16:45 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/06 12:16:45 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/06 12:16:45 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 12:16:45 VCS NOTICE V-16-1-10453 Node: 0 changed name from: 'cms-app-49-51' to: 'cms-app-49-51'
2012/12/06 12:16:45 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from EXITED to INITING
2012/12/06 12:16:45 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/06 12:16:45 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to ADMIN_WAIT
2012/12/06 12:17:33 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:18:36 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/12/06 12:18:37 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:18:40 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 40 invocations of the clean procedure have failed.
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from ADMIN_WAIT to STALE_PEER_WAIT
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from LEAVING to STALE_PEER_WAIT
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from STALE_PEER_WAIT to LOCAL_BUILD
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from LOCAL_BUILD to RUNNING
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from STALE_PEER_WAIT to REMOTE_BUILD
2012/12/06 12:19:36 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 0
2012/12/06 12:19:36 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10416 Attribute already exists 'IntentionalOffline' (Apache)
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-51199 Cannot modify attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-52010 Cannot modify ContainerOpts after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10411 Cannot modify Operations attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-51199 Cannot modify attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-52010 Cannot modify ContainerOpts after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10411 Cannot modify Operations attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10416 Attribute already exists 'RegList' (Mount)
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-51199 Cannot modify attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-52010 Cannot modify ContainerOpts after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10411 Cannot modify Operations attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-51199 Cannot modify attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-52010 Cannot modify ContainerOpts after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10411 Cannot modify Operations attribute after resources of this type have been instantiated
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:36 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RegList' (RVGSnapshot)
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IntentionalOffline' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10407 Type already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-51199 Cannot modify attribute after resources of this type have been instantiated
2012/12/06 12:19:37 VCS WARNING V-16-1-52010 Cannot modify ContainerOpts after resources of this type have been instantiated
2012/12/06 12:19:37 VCS WARNING V-16-1-10411 Cannot modify Operations attribute after resources of this type have been instantiated
2012/12/06 12:19:37 VCS WARNING V-16-1-10129 Attempting to add existing group
2012/12/06 12:19:37 VCS WARNING V-16-1-10173 Cannot modify Parallel attribute after resources have been added to group
2012/12/06 12:19:37 VCS WARNING V-16-1-10173 Cannot modify Parallel attribute after resources have been added to group
2012/12/06 12:19:37 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/06 12:19:37 VCS WARNING V-16-1-10129 Attempting to add existing group
2012/12/06 12:19:37 VCS WARNING V-16-1-10173 Cannot modify Parallel attribute after resources have been added to group
2012/12/06 12:19:37 VCS WARNING V-16-1-10173 Cannot modify Parallel attribute after resources have been added to group
2012/12/06 12:19:37 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/06 12:19:37 VCS WARNING V-16-1-10129 Attempting to add existing group
2012/12/06 12:19:37 VCS WARNING V-16-1-10173 Cannot modify Parallel attribute after resources have been added to group
2012/12/06 12:19:37 VCS WARNING V-16-1-10173 Cannot modify Parallel attribute after resources have been added to group
2012/12/06 12:19:37 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ResLogLevel' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'httpdDir' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SharedObjDir' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'EnvFile' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PidFile' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'HostName' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Port' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'User' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SecondLevelMonitor' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SecondLevelTimeout' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ConfigFile' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'EnableSSL' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DirectiveAfter' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DirectiveBefore' (Apache)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'User' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StartProgram' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StopProgram' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'CleanProgram' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MonitorProgram' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PidFiles' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MonitorProcesses' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'EnvFile' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'UseSUDash' (Application)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'FaultTolerance' (CoordPoint)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Domain' (DNS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'TTL' (DNS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'TSIGKeyFile' (DNS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StealthMasters' (DNS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ResRecord' (DNS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'CreatePTR' (DNS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'OffDelRR' (DNS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DiskGroup' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StartVolumes' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StopVolumes' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MonitorReservation' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PanicSystemOnDGLoss' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'tempUseFence' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DiskGroupType' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'UmountVolumes' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Reservation' (DiskGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'TargetResName' (DiskGroupSnap)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'FDSiteName' (DiskGroupSnap)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Disks' (DiskReservation)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'FailFast' (DiskReservation)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Percentage' (DiskReservation)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ProbeInterval' (DiskReservation)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PathName' (ElifNone)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PathName' (FileNone)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PathName' (FileOnOff)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PathName' (FileOnOnly)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Device' (IP)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Address' (IP)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NetMask' (IP)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PrefixLen' (IP)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Options' (IP)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IPOptions' (IP)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IPRouteOptions' (IP)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Address' (IPMultiNIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MultiNICAResName' (IPMultiNIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NetMask' (IPMultiNIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PrefixLen' (IPMultiNIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Options' (IPMultiNIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IPOptions' (IPMultiNIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'LogicalVolume' (LVMLogicalVolume)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'VolumeGroup' (LVMLogicalVolume)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'VolumeGroup' (LVMVolumeGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StartVolumes' (LVMVolumeGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'EnableLVMTagging' (LVMVolumeGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MountPoint' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'BlockDevice' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'FSType' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MountOpt' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'FsckOpt' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SnapUmount' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'CkptUmount' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SecondLevelMonitor' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SecondLevelTimeout' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'OptCheck' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'CreateMntPt' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ReuseMntPt' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MntPtPermission' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MntPtOwner' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MntPtGroup' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'AccessPermissionChk' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RecursiveMnt' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'VxFSMountLock' (Mount)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Device' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DualDevice' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NetMask' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PrefixLen' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Options' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RouteOptions' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'LinkOptions' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IPv4AddrOptions' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IPv6AddrOptions' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IPv4RouteOptions' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IPv6RouteOptions' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PingOptimize' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NetworkHosts' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Failback' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Mii' (MultiNICA)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Nproc' (NFS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'GracePeriod' (NFS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NFSSecurity' (NFS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NFSv4Support' (NFS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'LockFileTimeout' (NFS)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NFSRes' (NFSRestart)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Lower' (NFSRestart)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'LocksPathName' (NFSRestart)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NFSLockFailover' (NFSRestart)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Device' (NIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PingOptimize' (NIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Mii' (NIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NetworkHosts' (NIC)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SambaServerRes' (NetBios)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NetBiosName' (NetBios)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NetBiosAliases' (NetBios)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Interfaces' (NetBios)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'WinsSupport' (NetBios)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DomainMaster' (NetBios)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PidFile' (NetBios)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'EngineListeningPort' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'MessagesQueue' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NotifierListeningPort' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NotifierSourceIP' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SnmpdTrapPort' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SnmpCommunity' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SnmpConsoles' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SmtpServer' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SmtpServerVrfyOff' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SmtpServerTimeout' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SmtpReturnPath' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SmtpFromPath' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SmtpRecipients' (NotifierMngr)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PathName' (Process)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Arguments' (Process)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'UserName' (Process)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Priority' (Process)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PidFile' (Process)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PathName' (ProcessOnOnly)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Arguments' (ProcessOnOnly)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'UserName' (ProcessOnOnly)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Priority' (ProcessOnOnly)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PidFile' (ProcessOnOnly)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IgnoreArgs' (ProcessOnOnly)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'TargetResName' (Proxy)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'TargetSysName' (Proxy)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RVG' (RVG)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DiskGroup' (RVG)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StorageRVG' (RVG)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StorageDG' (RVG)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StorageHostIds' (RVG)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RVG' (RVGLogowner)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DiskGroup' (RVGLogowner)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StorageRVG' (RVGLogowner)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StorageDG' (RVGLogowner)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'StorageHostIds' (RVGLogowner)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RvgResourceName' (RVGPrimary)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'AutoTakeover' (RVGPrimary)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'AutoResync' (RVGPrimary)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'BunkerSyncTimeOut' (RVGPrimary)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'BunkerSyncElapsedTime' (RVGPrimary)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RVG' (RVGShared)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DiskGroup' (RVGShared)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RvgResourceName' (RVGSharedPri)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'AutoTakeover' (RVGSharedPri)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'AutoResync' (RVGSharedPri)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'VCSResLock' (RVGSharedPri)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'RvgResourceName' (RVGSnapshot)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'CacheObj' (RVGSnapshot)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Prefix' (RVGSnapshot)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DestroyOnOffline' (RVGSnapshot)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'FDFile' (RVGSnapshot)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'VCSResLock' (RVGSnapshot)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IpAddress' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Port' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Username' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Password' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'GroupName' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'VCSSysName' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ControlMode' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'OfflineWaitTime' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DomainType' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'BrokerIp' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ReturnIntOffline' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'OfflineMonitoringNode' (RemoteGroup)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ConfFile' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'LockDir' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Ports' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'IndepthMonitorCyclePeriod' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ResponseTimeout' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SambaTopDir' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PidFile' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SocketAddress' (SambaServer)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'SambaServerRes' (SambaShare)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ShareName' (SambaShare)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'ShareOptions' (SambaShare)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'PathName' (Share)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Client' (Share)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'OtherClients' (Share)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Options' (Share)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'NFSRes' (Share)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DiskGroup' (Volume)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'Volume' (Volume)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'DiskGroup' (VolumeSet)
2012/12/06 12:19:37 VCS WARNING V-16-1-10416 Attribute already exists 'VolumeSet' (VolumeSet)
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10238 Attempting to add existing resource
2012/12/06 12:19:37 VCS WARNING V-16-1-10905 Dependency between 'RCSSchedulerService' and 'Virtual_IP' already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10905 Dependency between 'alerts' and 'Virtual_IP' already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10905 Dependency between 'autorsync' and 'Virtual_IP' already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10905 Dependency between 'cms' and 'alerts' already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10905 Dependency between 'tomcat' and 'Virtual_IP' already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10905 Dependency between 'xportmanager' and 'Virtual_IP' already exists
2012/12/06 12:19:37 VCS WARNING V-16-1-10905 Dependency between 'xporttomcat' and 'Virtual_IP' already exists
2012/12/06 12:19:37 VCS INFO V-16-1-10466 End of snapshot received from node: 0.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 12:19:37 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/06 12:19:37 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 12:19:37 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
PWD=/var/VRTSvcs/diag/had
SHLVL=4
TERM=xterm
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 12:19:37 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/06 12:19:37 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 12:19:37 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/06 12:19:37 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 DOWN bond1 DOWN; Current status = bond0 UP bond1 UP.
2012/12/06 12:19:37 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 12:19:37 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/06 12:19:37 VCS NOTICE V-16-1-10439 Group (CMSApp_Notifier) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system cms-app-49-51 after probe
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:38 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/06 12:19:38 VCS NOTICE V-16-1-10439 Group (CMSApp_Cluster) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system cms-app-49-51 after probe
2012/12/06 12:19:39 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/06 12:19:39 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 12:20:14 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/12/06 12:20:14 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from RUNNING to LEAVING
2012/12/06 12:20:14 VCS NOTICE V-16-1-10010 Stopping all agents
2012/12/06 12:20:14 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from LEAVING to EXITING
2012/12/06 12:20:14 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from EXITING to EXITED
2012/12/06 12:20:15 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 12:20:15 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 0 ms
2012/12/06 12:20:19 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/06 12:20:19 VCS INFO V-16-1-10196 Cluster logger started
2012/12/06 12:20:19 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/06 12:20:19 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/06 12:20:19 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/06 12:20:19 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/06 12:20:19 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-52
2012/12/06 12:20:19 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/06 12:20:19 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/06 12:20:19 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/06 12:20:19 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/06 12:20:19 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 12:20:34 VCS CRITICAL V-16-1-11306 Did not receive cluster membership, manual intervention may be needed for seeding
2012/12/06 12:20:44 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:20:44 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/06 12:20:44 VCS NOTICE V-16-1-10086 System  (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 12:20:44 VCS NOTICE V-16-1-10086 System cms-app-49-52 (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 12:20:44 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 12:20:44 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 12:20:44 VCS NOTICE V-16-1-10453 Node: 0 changed name from: '' to: 'cms-app-49-51'
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to RUNNING
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 12:20:44 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 0
2012/12/06 12:20:44 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 12:20:44 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/06 12:20:44 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/06 12:20:44 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/06 12:20:44 VCS INFO V-16-1-10466 End of snapshot received from node: 0.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 12:20:44 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/06 12:20:44 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 12:20:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu Dec  6 12:20:44 2012

2012/12/06 12:20:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Thu Dec  6 12:20:44 2012

2012/12/06 12:20:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu Dec  6 12:20:44 2012

2012/12/06 12:20:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Thu Dec  6 12:20:44 2012

2012/12/06 12:20:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Dec  6 12:20:44 2012

2012/12/06 12:20:44 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/06 12:20:44 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/06 12:20:44 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
PWD=/var/VRTSvcs/diag/had
SHLVL=4
TERM=xterm
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 12:20:44 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 12:20:44 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:44 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-52
2012/12/06 12:20:44 VCS INFO V-16-1-50007 Initiating auto-start online of group CMSApp_Notifier
2012/12/06 12:20:44 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/06 12:20:44 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Notifier on all nodes
2012/12/06 12:20:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-51
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-51
2012/12/06 12:20:45 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10442 Initiating auto-start online of group CMSApp_Cluster on system cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:45 VCS INFO V-16-10031-504 (cms-app-49-52) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/06 12:20:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/06 12:20:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/06 12:20:46 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [  OK  ]
==============================================

2012/12/06 12:20:46 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/06 12:20:46 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/06 12:20:46 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-52
2012/12/06 12:20:46 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-52 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 12:20:46 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:46 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:46 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:51 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/06 12:20:51 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/06 12:20:51 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:51 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-52
2012/12/06 13:02:08 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/06 13:02:08 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/06 13:02:08 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/06 13:02:08 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/06 13:02:08 VCS INFO V-16-1-10196 Cluster logger started
2012/12/06 13:02:08 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/06 13:02:08 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-52
2012/12/06 13:02:08 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/06 13:02:08 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/06 13:02:08 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/06 13:02:08 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/06 13:02:08 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 13:02:13 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 13:02:13 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x3, DDNA: 0x0
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System  (Node '0') changed state from UNKNOWN to INITING
2012/12/06 13:02:13 VCS NOTICE V-16-1-10086 System  (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 13:02:13 VCS NOTICE V-16-1-10086 System cms-app-49-52 (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 13:02:13 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 13:02:13 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 13:02:13 VCS WARNING V-16-1-50129 Operation 'hasys -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 13:02:13 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 13:02:13 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 13:02:13 VCS NOTICE V-16-1-10453 Node: 0 changed name from: '' to: 'cms-app-49-51'
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from INITING to RUNNING
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 13:02:13 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 0
2012/12/06 13:02:13 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 13:02:13 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/06 13:02:13 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/06 13:02:13 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/06 13:02:13 VCS INFO V-16-1-10466 End of snapshot received from node: 0.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 13:02:13 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/06 13:02:13 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 13:02:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu Dec  6 13:02:13 2012

2012/12/06 13:02:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Thu Dec  6 13:02:13 2012

2012/12/06 13:02:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu Dec  6 13:02:13 2012

2012/12/06 13:02:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Thu Dec  6 13:02:13 2012

2012/12/06 13:02:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Dec  6 13:02:13 2012

2012/12/06 13:02:13 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/06 13:02:13 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 13:02:14 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-52
2012/12/06 13:02:14 VCS INFO V-16-1-10297 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group CMSApp_Cluster
2012/12/06 13:02:14 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS WARNING V-16-6-15034 (cms-app-49-52) violation:Offlining group CMSApp_Cluster on system cms-app-49-52
2012/12/06 13:02:14 VCS INFO V-16-1-50135 User root fired command: hagrp -offline CMSApp_Cluster  cms-app-49-52  from localhost
2012/12/06 13:02:14 VCS NOTICE V-16-1-10167 Initiating manual offline of group CMSApp_Cluster on system cms-app-49-52
2012/12/06 13:02:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 13:02:14 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/violation cms-app-49-52 CMSApp_Cluster   successfully
2012/12/06 13:02:15 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-52
2012/12/06 13:02:16 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-52
2012/12/06 13:02:16 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-52 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 13:02:17 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/06 13:02:18 VCS ERROR V-16-2-13064 (cms-app-49-52) Agent is calling clean for resource(autorsync) because the resource is up even after offline completed.
2012/12/06 13:02:18 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:02:21 VCS ERROR V-16-2-13069 (cms-app-49-52) Resource(autorsync) - clean failed.
2012/12/06 13:03:21 VCS ERROR V-16-2-13077 (cms-app-49-52) Agent is unable to offline resource(autorsync). Administrative intervention may be required.
2012/12/06 13:03:21 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:04:24 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:05:27 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:06:30 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:07:33 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:08:37 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:09:41 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:10:44 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:11:47 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:11:50 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 10 invocations of the clean procedure have failed.
2012/12/06 13:12:50 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:13:53 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:14:56 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:15:59 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:17:02 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:18:06 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:19:10 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:20:13 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:21:16 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:22:19 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:22:22 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 20 invocations of the clean procedure have failed.
2012/12/06 13:23:22 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:24:25 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:25:28 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:26:31 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:27:34 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:28:37 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:29:41 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:30:45 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:31:48 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:32:51 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:32:54 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 30 invocations of the clean procedure have failed.
2012/12/06 13:33:54 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:34:57 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:36:00 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:37:04 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:38:08 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:39:11 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:40:14 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:41:17 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:42:20 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:43:23 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:43:26 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 40 invocations of the clean procedure have failed.
2012/12/06 13:44:26 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:45:29 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:46:32 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:47:36 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:48:40 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:49:43 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:50:46 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:51:49 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:52:52 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:53:55 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:53:58 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 50 invocations of the clean procedure have failed.
2012/12/06 13:54:58 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:56:01 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:57:04 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:58:08 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:59:12 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:00:16 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:01:19 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:02:22 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:03:25 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:04:28 VCS INFO V-16-1-10305 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 14:04:28 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-52
2012/12/06 14:58:38 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 UP bond1 UP; Current status = bond0 DOWN bond1 DOWN.
2012/12/06 14:58:39 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 14:58:39 VCS NOTICE V-16-1-10112 System (cms-app-49-52) - Membership: 0x2, DDNA: 0x0
2012/12/06 14:58:39 VCS ERROR V-16-1-10079 System cms-app-49-51 (Node '0') is in Down State - Membership: 0x2
2012/12/06 14:58:39 VCS ERROR V-16-1-10322 System cms-app-49-51 (Node '0') changed state from RUNNING to FAULTED
2012/12/06 14:58:39 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-51
2012/12/06 14:58:39 VCS NOTICE V-16-1-10446 Group CMSApp_Notifier is offline on system cms-app-49-51
2012/12/06 14:58:39 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/06 14:58:39 VCS INFO V-16-1-10494 System cms-app-49-51 not in RUNNING state
2012/12/06 14:58:39 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Notifier
2012/12/06 14:58:39 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-52
2012/12/06 14:58:39 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-52 (VCS initiated)
2012/12/06 14:58:39 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-52
