#
# Log Name:     engine
# System:       cms-app-49-51.workgroup
# SysInfo:      Linux:cms-app-49-51.workgroup,#1 SMP Thu Jul 5 21:08:58 EDT 2012,2.6.32-279.2.1.el6.x86_64,x86_64
# Created:      2012/12/05 17:13:30 
#

2012/12/05 17:13:30 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/05 17:13:30 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/05 17:13:30 VCS INFO V-16-1-10196 Cluster logger started
2012/12/05 17:13:30 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/05 17:13:30 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/05 17:13:30 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/05 17:13:30 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-51
2012/12/05 17:13:30 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/05 17:13:30 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/05 17:13:30 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/05 17:13:30 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/05 17:13:30 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/05 17:13:35 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/05 17:13:35 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/05 17:13:35 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/05 17:13:35 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/12/05 17:13:35 VCS WARNING V-16-1-50129 Operation 'hasys -modify ... -update' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/05 17:13:35 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/05 17:13:35 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/05 17:13:35 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/05 17:13:35 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/05 17:13:35 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'cms-app-49-52'
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to RUNNING
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/05 17:13:35 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 1
2012/12/05 17:13:35 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/05 17:13:35 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/05 17:13:35 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/05 17:13:35 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/05 17:13:35 VCS INFO V-16-1-10466 End of snapshot received from node: 1.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/05 17:13:35 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/05 17:13:35 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/05 17:13:35 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/12/05 17:13:35 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed Dec  5 17:13:35 2012

2012/12/05 17:13:35 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Wed Dec  5 17:13:35 2012

2012/12/05 17:13:35 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed Dec  5 17:13:35 2012

2012/12/05 17:13:35 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Wed Dec  5 17:13:35 2012

2012/12/05 17:13:35 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed Dec  5 17:13:35 2012

2012/12/05 17:13:35 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/05 17:13:35 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/05 17:13:35 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
PWD=/var/VRTSvcs/diag/had
SHLVL=4
TERM=xterm
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/05 17:13:35 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/05 17:13:35 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/05 17:13:35 VCS INFO V-16-1-50007 Initiating auto-start online of group CMSApp_Notifier
2012/12/05 17:13:35 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/05 17:13:35 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Notifier on all nodes
2012/12/05 17:13:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-51
2012/12/05 17:13:36 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:36 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-51
2012/12/05 17:13:36 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/05 17:13:36 VCS INFO V-16-1-50007 Initiating auto-start online of group CMSApp_Cluster
2012/12/05 17:13:36 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/05 17:13:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/05 17:13:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:37 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/05 17:13:37 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/05 17:13:44 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:45 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 17:13:45 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/05 17:13:45 VCS INFO V-16-10031-504 (cms-app-49-51) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/05 17:13:46 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/05 17:13:46 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/05 17:13:46 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [  OK  ]
==============================================

2012/12/05 17:13:46 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:46 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:46 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:50 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/05 17:13:51 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/05 17:13:51 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:53 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/05 17:13:54 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-51 to cms-app-49-52 in 5 seconds intervals
==============================================

2012/12/05 17:13:54 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:56 VCS INFO V-16-10031-504 (cms-app-49-51) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/05 17:13:56 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  12215) is running...
settings server (pid  12224) is running...
alerts (pid  12620) is running...
==============================================

2012/12/05 17:13:56 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:13:58 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/05 17:13:59 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/05 17:13:59 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:13:59 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/05 17:31:36 VCS INFO V-16-1-50135 User root fired command: hares -offline tomcat  cms-app-49-51  1  from localhost
2012/12/05 17:31:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:31:37 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:offline:Executed /etc/init.d/tomcat as user root
2012/12/05 17:31:37 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (offline) 
==============================================
Stopping Tomcat
==============================================

2012/12/05 17:31:37 VCS INFO V-16-1-10305 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/05 17:31:41 VCS INFO V-16-1-50135 User root fired command: hares -online tomcat  cms-app-49-51  from localhost
2012/12/05 17:31:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/05 17:31:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 17:31:41 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 17:31:42 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/05 17:31:42 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 17:31:42 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/05 17:50:52 VCS INFO V-16-1-10196 Cluster logger started
2012/12/05 17:50:52 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/05 17:50:52 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/05 17:50:52 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/05 17:50:52 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/05 17:50:52 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/05 17:50:52 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-51
2012/12/05 17:50:52 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/05 17:50:52 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/05 17:50:52 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/05 17:50:52 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/05 17:50:52 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/05 17:51:00 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/05 17:51:00 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/05 17:51:00 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/05 17:51:00 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/12/05 17:51:00 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/05 17:51:00 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/05 17:51:00 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'cms-app-49-52'
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to RUNNING
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/05 17:51:00 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 1
2012/12/05 17:51:00 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/05 17:51:00 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/05 17:51:00 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/05 17:51:00 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/05 17:51:00 VCS INFO V-16-1-10466 End of snapshot received from node: 1.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/05 17:51:00 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/05 17:51:00 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/05 17:51:00 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/12/05 17:51:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed Dec  5 17:51:00 2012

2012/12/05 17:51:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Wed Dec  5 17:51:00 2012

2012/12/05 17:51:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed Dec  5 17:51:00 2012

2012/12/05 17:51:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Wed Dec  5 17:51:00 2012

2012/12/05 17:51:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed Dec  5 17:51:00 2012

2012/12/05 17:51:00 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/05 17:51:00 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/05 17:51:01 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/05 17:51:01 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:01 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/05 17:51:02 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/05 17:51:03 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/05 17:51:03 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/05 18:34:23 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(RCSSchedulerService) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:23 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(xportmanager) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:23 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(tomcat) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:23 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:tomcat:clean:PidFile </etc/init.d/tomcat.pid> does not exist, process will not be killed
2012/12/05 18:34:23 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:xportmanager:clean:PidFile </etc/init.d/xportmanager.pid> does not exist, process will not be killed
2012/12/05 18:34:23 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:RCSSchedulerService:clean:PidFile </usr/local/tstv/RCSScheduler/config/rcsScheduler.pid> does not exist, process will not be killed
2012/12/05 18:34:24 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(RCSSchedulerService) - clean completed successfully.
2012/12/05 18:34:24 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(tomcat) - clean completed successfully.
2012/12/05 18:34:24 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(xportmanager) - clean completed successfully.
2012/12/05 18:34:24 VCS INFO V-16-1-10307 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:24 VCS INFO V-16-1-10305 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:24 VCS INFO V-16-1-10305 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:24 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/05 18:34:24 VCS INFO V-16-1-10306 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/05 18:34:27 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/05 18:34:28 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(autorsync): Output of the completed operation (offline) 
==============================================
Stopping autorsync
==============================================

2012/12/05 18:34:28 VCS INFO V-16-1-10305 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:30 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xporttomcat:offline:Executed /etc/init.d/xporttomcat as user root
2012/12/05 18:34:30 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(alerts) because the resource became OFFLINE unexpectedly, on its own.
2012/12/05 18:34:30 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:alerts:clean:PidFile </etc/init.d/alerts.pid> does not exist, process will not be killed
2012/12/05 18:34:30 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:alerts:clean:PidFile </etc/init.d/alerts-nameservice.pid> does not exist, process will not be killed
2012/12/05 18:34:30 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:alerts:clean:PidFile </etc/init.d/alerts-settingserver.pid> does not exist, process will not be killed
2012/12/05 18:34:30 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xporttomcat): Output of the completed operation (offline) 
==============================================
Stopping XportTomcat
==============================================

2012/12/05 18:34:30 VCS INFO V-16-1-10305 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:31 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(alerts) - clean completed successfully.
2012/12/05 18:34:31 VCS INFO V-16-1-10307 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/05 18:34:33 VCS INFO V-16-10031-504 (cms-app-49-52) Application:cms:offline:Executed /etc/init.d/cms as user root
2012/12/05 18:34:33 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(cms): Output of the completed operation (offline) 
==============================================
Stopping Content Management System
Stopping ADI NameServer
==============================================

2012/12/05 18:34:33 VCS INFO V-16-1-10305 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/05 18:34:35 VCS INFO V-16-1-10305 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/05 18:34:35 VCS ERROR V-16-1-10205 Group CMSApp_Cluster is faulted on system cms-app-49-52
2012/12/05 18:34:35 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-52
2012/12/05 18:34:35 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/05 18:34:35 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/05 18:34:35 VCS INFO V-16-1-50010 Group CMSApp_Cluster is online or faulted on system cms-app-49-52
2012/12/05 18:34:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:43 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/05 18:34:43 VCS INFO V-16-10031-504 (cms-app-49-51) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/05 18:34:43 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/05 18:34:44 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/05 18:34:44 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/05 18:34:44 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [60G[[0;32m  OK  [0;39m]
==============================================

2012/12/05 18:34:44 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:44 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:44 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:48 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/05 18:34:49 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/05 18:34:49 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:51 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/05 18:34:52 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-51 to cms-app-49-52 in 5 seconds intervals
==============================================

2012/12/05 18:34:52 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:55 VCS INFO V-16-10031-504 (cms-app-49-51) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/05 18:34:55 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  18488) is running...
settings server (pid  18524) is running...
alerts (pid  18913) is running...
==============================================

2012/12/05 18:34:55 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/05 18:34:58 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/05 18:34:58 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/05 18:34:58 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/05 18:34:58 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/05 18:34:58 VCS NOTICE V-16-1-10448 Group CMSApp_Cluster failed over to system cms-app-49-51
2012/12/05 18:42:26 VCS INFO V-16-1-10307 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/05 18:42:36 VCS INFO V-16-1-10307 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/06 10:57:40 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/06 10:57:40 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/06 10:57:40 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/06 10:57:40 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/06 10:57:40 VCS INFO V-16-1-10196 Cluster logger started
2012/12/06 10:57:40 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/06 10:57:40 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-51
2012/12/06 10:57:40 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/06 10:57:40 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/06 10:57:41 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/06 10:57:41 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/06 10:57:41 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 10:57:48 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 10:57:48 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/06 10:57:48 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 10:57:48 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 10:57:48 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 10:57:48 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 10:57:48 VCS WARNING V-16-1-50129 Operation 'hasys -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 10:57:48 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'cms-app-49-52'
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to RUNNING
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 10:57:48 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 1
2012/12/06 10:57:48 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 10:57:48 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/06 10:57:48 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/06 10:57:48 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/06 10:57:48 VCS INFO V-16-1-10466 End of snapshot received from node: 1.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 10:57:48 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/06 10:57:48 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 10:57:48 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/12/06 10:57:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu Dec  6 10:57:48 2012

2012/12/06 10:57:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Thu Dec  6 10:57:48 2012

2012/12/06 10:57:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu Dec  6 10:57:48 2012

2012/12/06 10:57:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Thu Dec  6 10:57:48 2012

2012/12/06 10:57:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Dec  6 10:57:48 2012

2012/12/06 10:57:48 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/06 10:57:48 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/06 10:57:48 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:48 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 10:57:48 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/06 10:57:48 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:48 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 10:57:49 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/06 10:57:50 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/06 10:57:50 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 11:28:47 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 UP bond1 UP; Current status = bond0 DOWN bond1 DOWN.
2012/12/06 11:28:48 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 11:28:48 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x1, DDNA: 0x0
2012/12/06 11:28:48 VCS ERROR V-16-1-10079 System cms-app-49-52 (Node '1') is in Down State - Membership: 0x1
2012/12/06 11:28:48 VCS ERROR V-16-1-10322 System cms-app-49-52 (Node '1') changed state from RUNNING to FAULTED
2012/12/06 11:28:48 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-52
2012/12/06 11:28:48 VCS NOTICE V-16-1-10446 Group CMSApp_Notifier is offline on system cms-app-49-52
2012/12/06 11:28:48 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/06 11:28:48 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/06 11:28:48 VCS INFO V-16-1-10494 System cms-app-49-52 not in RUNNING state
2012/12/06 11:28:48 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/06 11:28:48 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Notifier
2012/12/06 11:28:48 VCS INFO V-16-1-10494 System cms-app-49-52 not in RUNNING state
2012/12/06 11:28:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:28:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-51
2012/12/06 11:28:48 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:28:48 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-51
2012/12/06 11:28:56 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:28:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:28:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:28:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:28:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:28:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:28:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:28:56 VCS INFO V-16-10031-504 (cms-app-49-51) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/06 11:28:56 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/06 11:28:56 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/06 11:28:57 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/06 11:28:57 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/06 11:28:57 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [60G[[0;32m  OK  [0;39m]
==============================================

2012/12/06 11:28:57 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:28:57 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:28:57 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:29:01 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/06 11:29:02 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/06 11:29:02 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:29:04 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/06 11:29:05 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-51 to cms-app-49-52 in 5 seconds intervals
==============================================

2012/12/06 11:29:05 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:29:07 VCS INFO V-16-10031-504 (cms-app-49-51) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/06 11:29:08 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  14211) is running...
settings server (pid  14228) is running...
alerts (pid  14623) is running...
==============================================

2012/12/06 11:29:08 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:29:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 11:29:10 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/06 11:29:11 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/06 11:29:11 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 11:29:11 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/06 11:37:13 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 DOWN bond1 DOWN; Current status = bond0 UP bond1 UP.
2012/12/06 11:37:19 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 11:37:19 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/06 11:37:19 VCS NOTICE V-16-1-10086 System cms-app-49-52 (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 11:37:19 VCS NOTICE V-16-1-10449 Group CMSApp_Cluster autodisabled on node cms-app-49-52 until it is probed
2012/12/06 11:37:19 VCS NOTICE V-16-1-10449 Group CMSApp_Notifier autodisabled on node cms-app-49-52 until it is probed
2012/12/06 11:37:19 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on node cms-app-49-52 until it is probed
2012/12/06 11:37:19 VCS NOTICE V-16-1-10453 Node: 1 changed name from: 'cms-app-49-52' to: 'cms-app-49-52'
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from FAULTED to INITING
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 11:37:19 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0x2
2012/12/06 11:37:19 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-52
2012/12/06 11:37:20 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 11:37:20 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10297 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group CMSApp_Cluster
2012/12/06 11:37:20 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS INFO V-16-1-10306 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 11:37:20 VCS WARNING V-16-6-15034 (cms-app-49-52) violation:Offlining group CMSApp_Cluster on system cms-app-49-52
2012/12/06 11:37:20 VCS NOTICE V-16-1-10167 Initiating manual offline of group CMSApp_Cluster on system cms-app-49-52
2012/12/06 11:37:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 11:37:20 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/violation cms-app-49-52 CMSApp_Cluster   successfully
2012/12/06 11:37:20 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-52
2012/12/06 11:37:22 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-52
2012/12/06 11:37:22 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-52 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 11:37:23 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/06 11:37:24 VCS ERROR V-16-2-13064 (cms-app-49-52) Agent is calling clean for resource(autorsync) because the resource is up even after offline completed.
2012/12/06 11:37:24 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:37:27 VCS ERROR V-16-2-13069 (cms-app-49-52) Resource(autorsync) - clean failed.
2012/12/06 11:38:28 VCS ERROR V-16-2-13077 (cms-app-49-52) Agent is unable to offline resource(autorsync). Administrative intervention may be required.
2012/12/06 11:38:28 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 11:39:31 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:07:05 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/06 12:07:05 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/06 12:07:05 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/06 12:07:05 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/06 12:07:05 VCS INFO V-16-1-10196 Cluster logger started
2012/12/06 12:07:05 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/06 12:07:05 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-51
2012/12/06 12:07:05 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/06 12:07:05 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/06 12:07:06 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/06 12:07:06 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/06 12:07:06 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 12:07:13 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:07:13 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/06 12:07:13 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/06 12:07:13 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 12:07:13 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 12:07:13 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'cms-app-49-52'
2012/12/06 12:07:13 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to LEAVING
2012/12/06 12:07:13 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to ADMIN_WAIT
2012/12/06 12:07:13 VCS WARNING V-16-1-10071 Last RUNNING node has failed or build from local disk failed.  Unable to form a cluster.
2012/12/06 12:07:22 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:07:24 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:07:35 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:07:38 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:08:04 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:08:07 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 30 invocations of the clean procedure have failed.
2012/12/06 12:08:22 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:08:24 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:08:36 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:08:39 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:09:08 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:09:22 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:09:24 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:09:36 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:09:39 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:10:11 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:10:22 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:10:24 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:10:35 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:10:38 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:11:01 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/12/06 12:11:01 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from ADMIN_WAIT to EXITED
2012/12/06 12:11:02 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 12:11:02 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 0 ms
2012/12/06 12:16:40 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/06 12:16:40 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/06 12:16:40 VCS INFO V-16-1-10196 Cluster logger started
2012/12/06 12:16:40 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/06 12:16:40 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/06 12:16:40 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/06 12:16:40 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-51
2012/12/06 12:16:40 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/06 12:16:40 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/06 12:16:40 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/06 12:16:40 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/06 12:16:40 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 12:16:45 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:16:45 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/06 12:16:45 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/06 12:16:45 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 12:16:45 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 12:16:45 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'cms-app-49-52'
2012/12/06 12:16:45 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to LEAVING
2012/12/06 12:16:45 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to ADMIN_WAIT
2012/12/06 12:16:45 VCS WARNING V-16-1-10071 Last RUNNING node has failed or build from local disk failed.  Unable to form a cluster.
2012/12/06 12:17:22 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:17:24 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:17:33 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:17:36 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:17:39 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:18:22 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:18:24 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:18:36 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:18:37 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 12:18:38 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:18:40 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 40 invocations of the clean procedure have failed.
2012/12/06 12:19:22 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:19:24 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:19:35 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in ADMIN_WAIT state
2012/12/06 12:19:36 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_FORCE_LOCAL cms-app-49-51  from localhost
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from ADMIN_WAIT to STALE_PEER_WAIT
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from LEAVING to STALE_PEER_WAIT
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from STALE_PEER_WAIT to LOCAL_BUILD
2012/12/06 12:19:36 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from LOCAL_BUILD to RUNNING
2012/12/06 12:19:36 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from STALE_PEER_WAIT to REMOTE_BUILD
2012/12/06 12:19:36 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu Dec  6 12:19:36 2012

2012/12/06 12:19:36 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Thu Dec  6 12:19:36 2012

2012/12/06 12:19:36 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu Dec  6 12:19:36 2012

2012/12/06 12:19:36 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Thu Dec  6 12:19:36 2012

2012/12/06 12:19:36 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Dec  6 12:19:36 2012

2012/12/06 12:19:36 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/06 12:19:36 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0x2
2012/12/06 12:19:36 VCS CRITICAL V-16-1-51135 GlobalCounter not updated
2012/12/06 12:19:36 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/06 12:19:37 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
PWD=/var/VRTSvcs/diag/had
SHLVL=4
TERM=xterm
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 12:19:37 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/06 12:19:37 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 12:19:37 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 12:19:37 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/06 12:19:37 VCS NOTICE V-16-1-10439 Group (CMSApp_Notifier) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system cms-app-49-51 after probe
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:37 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 12:19:38 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/06 12:19:38 VCS NOTICE V-16-1-10439 Group (CMSApp_Cluster) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system cms-app-49-51 after probe
2012/12/06 12:19:39 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/06 12:19:39 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 12:20:14 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from RUNNING to LEAVING
2012/12/06 12:20:14 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from LEAVING to EXITING
2012/12/06 12:20:14 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from EXITING to EXITED
2012/12/06 12:20:30 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:20:30 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x1, DDNA: 0x2
2012/12/06 12:20:30 VCS ERROR V-16-1-10113 System cms-app-49-52 (Node '1') is in DDNA Membership - Membership: 0x1, Visible: 0x0
2012/12/06 12:20:44 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:20:44 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/06 12:20:44 VCS NOTICE V-16-1-10086 System cms-app-49-52 (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 12:20:44 VCS NOTICE V-16-1-10453 Node: 1 changed name from: 'cms-app-49-52' to: 'cms-app-49-52'
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from EXITED to INITING
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 12:20:44 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0x2
2012/12/06 12:20:44 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 12:20:44 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
PWD=/var/VRTSvcs/diag/had
SHLVL=4
TERM=xterm
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 12:20:44 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 12:20:44 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:44 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-52
2012/12/06 12:20:44 VCS INFO V-16-1-50007 Initiating auto-start online of group CMSApp_Notifier
2012/12/06 12:20:44 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Notifier
2012/12/06 12:20:44 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Notifier on all nodes
2012/12/06 12:20:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) on System cms-app-49-51
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10297 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS INFO V-16-1-10298 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:20:45 VCS NOTICE V-16-1-10447 Group CMSApp_Notifier is online on system cms-app-49-51
2012/12/06 12:20:45 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10442 Initiating auto-start online of group CMSApp_Cluster on system cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 12:20:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/06 12:20:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/06 12:20:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/06 12:20:46 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [  OK  ]
==============================================

2012/12/06 12:20:46 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/06 12:20:46 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/06 12:20:46 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-52
2012/12/06 12:20:46 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-52 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 12:20:46 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:46 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:46 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:51 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/06 12:20:51 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/06 12:20:51 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 12:20:51 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-52
2012/12/06 12:39:47 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 UP bond1 UP; Current status = bond0 DOWN bond1 DOWN.
2012/12/06 12:39:49 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 12:39:49 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x1, DDNA: 0x0
2012/12/06 12:39:49 VCS ERROR V-16-1-10079 System cms-app-49-52 (Node '1') is in Down State - Membership: 0x1
2012/12/06 12:39:49 VCS ERROR V-16-1-10322 System cms-app-49-52 (Node '1') changed state from RUNNING to FAULTED
2012/12/06 12:39:49 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-52
2012/12/06 12:39:49 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/06 12:39:49 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/06 12:39:49 VCS INFO V-16-1-10494 System cms-app-49-52 not in RUNNING state
2012/12/06 12:39:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:39:57 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:39:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:39:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:39:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:39:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:39:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:39:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:39:57 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/06 12:39:57 VCS INFO V-16-10031-504 (cms-app-49-51) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/06 12:39:57 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/06 12:39:58 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/06 12:39:58 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/06 12:39:58 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [  OK  ]
==============================================

2012/12/06 12:39:58 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:39:58 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:39:58 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:40:02 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/06 12:40:03 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/06 12:40:03 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:40:05 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/06 12:40:06 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-51 to cms-app-49-52 in 5 seconds intervals
==============================================

2012/12/06 12:40:06 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:40:09 VCS INFO V-16-10031-504 (cms-app-49-51) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/06 12:40:09 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  3767) is running...
settings server (pid  3785) is running...
alerts (pid  4192) is running...
==============================================

2012/12/06 12:40:09 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:40:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 12:40:12 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/06 12:40:12 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/06 12:40:12 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 12:40:12 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/06 13:02:09 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = bond0 DOWN bond1 DOWN; Current status = bond0 UP bond1 UP.
2012/12/06 13:02:13 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 13:02:13 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/06 13:02:13 VCS NOTICE V-16-1-10086 System cms-app-49-52 (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 13:02:13 VCS NOTICE V-16-1-10449 Group CMSApp_Cluster autodisabled on node cms-app-49-52 until it is probed
2012/12/06 13:02:13 VCS NOTICE V-16-1-10449 Group CMSApp_Notifier autodisabled on node cms-app-49-52 until it is probed
2012/12/06 13:02:13 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on node cms-app-49-52 until it is probed
2012/12/06 13:02:13 VCS NOTICE V-16-1-10453 Node: 1 changed name from: 'cms-app-49-52' to: 'cms-app-49-52'
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from FAULTED to INITING
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 13:02:13 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0x2
2012/12/06 13:02:13 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from REMOTE_BUILD to RUNNING
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-6-15023 (cms-app-49-52) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 13:02:14 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-52 1   successfully
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-52
2012/12/06 13:02:14 VCS INFO V-16-1-10297 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group CMSApp_Cluster
2012/12/06 13:02:14 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group CMSApp_Cluster on all nodes
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (First probe)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS INFO V-16-1-10306 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 13:02:14 VCS WARNING V-16-6-15034 (cms-app-49-52) violation:Offlining group CMSApp_Cluster on system cms-app-49-52
2012/12/06 13:02:14 VCS NOTICE V-16-1-10167 Initiating manual offline of group CMSApp_Cluster on system cms-app-49-52
2012/12/06 13:02:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 13:02:14 VCS INFO V-16-6-15002 (cms-app-49-52) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/violation cms-app-49-52 CMSApp_Cluster   successfully
2012/12/06 13:02:15 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-52
2012/12/06 13:02:16 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-52
2012/12/06 13:02:16 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-52 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 13:02:17 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/06 13:02:18 VCS ERROR V-16-2-13064 (cms-app-49-52) Agent is calling clean for resource(autorsync) because the resource is up even after offline completed.
2012/12/06 13:02:18 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:02:21 VCS ERROR V-16-2-13069 (cms-app-49-52) Resource(autorsync) - clean failed.
2012/12/06 13:03:21 VCS ERROR V-16-2-13077 (cms-app-49-52) Agent is unable to offline resource(autorsync). Administrative intervention may be required.
2012/12/06 13:03:21 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:04:24 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:05:27 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:06:30 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:07:33 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:08:37 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:09:41 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:10:44 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:11:47 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:11:50 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 10 invocations of the clean procedure have failed.
2012/12/06 13:12:50 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:13:53 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:14:56 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:15:59 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:17:02 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:18:06 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:19:10 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:20:13 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:21:16 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:22:19 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:22:22 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 20 invocations of the clean procedure have failed.
2012/12/06 13:23:22 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:24:25 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:25:28 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:26:31 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:27:34 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:28:37 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:29:41 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:30:45 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:31:48 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:32:51 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:32:54 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 30 invocations of the clean procedure have failed.
2012/12/06 13:33:54 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:34:57 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:36:00 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:37:04 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:38:08 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:39:11 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:40:14 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:41:17 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:42:20 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:43:23 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:43:26 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 40 invocations of the clean procedure have failed.
2012/12/06 13:44:26 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:45:29 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:46:32 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:47:36 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:48:40 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:49:43 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:50:46 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:51:49 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:52:52 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:53:55 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:53:58 VCS ERROR V-16-2-13079 (cms-app-49-52) Resource(autorsync): The last 50 invocations of the clean procedure have failed.
2012/12/06 13:54:58 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:56:01 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:57:04 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:58:08 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 13:59:12 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:00:16 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:01:19 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:02:22 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:03:25 VCS INFO V-16-10031-521 (cms-app-49-52) Application:autorsync:clean:Killing process: 1284
2012/12/06 14:04:28 VCS INFO V-16-1-10305 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 14:04:28 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-52
2012/12/06 15:26:59 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/12/06 15:26:59 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/12/06 15:26:59 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.12.0
2012/12/06 15:26:59 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.120.000-5.1SP1PR2-2011-01-05_02.04.08
2012/12/06 15:26:59 VCS INFO V-16-1-10196 Cluster logger started
2012/12/06 15:26:59 VCS NOTICE V-16-1-10114 Opening GAB library
2012/12/06 15:26:59 VCS NOTICE V-16-1-10619 'HAD' starting on: cms-app-49-51
2012/12/06 15:26:59 VCS INFO V-16-1-51138 Number of processors configured on this system are 24
2012/12/06 15:26:59 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/12/06 15:27:00 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/12/06 15:27:00 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
2012/12/06 15:27:00 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
2012/12/06 15:27:07 VCS INFO V-16-1-10077 Received new cluster membership
2012/12/06 15:27:07 VCS NOTICE V-16-1-10112 System (cms-app-49-51) - Membership: 0x3, DDNA: 0x0
2012/12/06 15:27:07 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/12/06 15:27:07 VCS NOTICE V-16-1-10086 System cms-app-49-51 (Node '0') is in Regular Membership - Membership: 0x3
2012/12/06 15:27:07 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/12/06 15:27:07 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 15:27:07 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 15:27:07 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 15:27:07 VCS WARNING V-16-1-50129 Operation 'hasys -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 15:27:07 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 15:27:07 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 15:27:07 VCS ERROR V-16-1-50129 Operation 'hares -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/12/06 15:27:07 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'cms-app-49-52'
2012/12/06 15:27:07 VCS NOTICE V-16-1-10322 System cms-app-49-52 (Node '1') changed state from INITING to RUNNING
2012/12/06 15:27:07 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/12/06 15:27:07 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 1
2012/12/06 15:27:07 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 15:27:07 VCS NOTICE V-16-1-10181 Group CMSApp_Cluster AutoRestart set to 1
2012/12/06 15:27:07 VCS NOTICE V-16-1-10181 Group CMSApp_Notifier AutoRestart set to 1
2012/12/06 15:27:07 VCS NOTICE V-16-1-10181 Group VCShmg AutoRestart set to 1
2012/12/06 15:27:07 VCS INFO V-16-1-10466 End of snapshot received from node: 1.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 15:27:07 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/12/06 15:27:07 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/12/06 15:27:07 VCS NOTICE V-16-1-10322 System cms-app-49-51 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/12/06 15:27:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu Dec  6 15:27:07 2012

2012/12/06 15:27:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Thu Dec  6 15:27:07 2012

2012/12/06 15:27:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu Dec  6 15:27:07 2012

2012/12/06 15:27:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Thu Dec  6 15:27:07 2012

2012/12/06 15:27:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Dec  6 15:27:07 2012

2012/12/06 15:27:07 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/12/06 15:27:07 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = UNKNOWN; Current status = bond0 UP bond1 UP.
2012/12/06 15:27:07 VCS INFO V-16-1-10304 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:07 VCS INFO V-16-6-15023 (cms-app-49-51) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLETYPE=vt
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
LANG=en_US.UTF-8
LANGSH_SOURCED=1
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=3
RUNLEVEL=3
SHLVL=5
TERM=linux
UPSTART_EVENTS=runlevel
UPSTART_INSTANCE=
UPSTART_JOB=rc
_=/usr/bin/env
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger

2012/12/06 15:27:07 VCS INFO V-16-6-15002 (cms-app-49-51) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables cms-app-49-51 1   successfully
2012/12/06 15:27:07 VCS INFO V-16-1-10304 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:07 VCS INFO V-16-1-10304 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:07 VCS INFO V-16-1-10304 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:07 VCS INFO V-16-1-10304 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:08 VCS INFO V-16-1-10304 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:08 VCS INFO V-16-1-10304 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:08 VCS INFO V-16-1-10304 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:08 VCS INFO V-16-1-10304 Resource vcsnotifier (Owner: Unspecified, Group: CMSApp_Notifier) is offline on cms-app-49-51 (First probe)
2012/12/06 15:27:08 VCS NOTICE V-16-1-10438 Group CMSApp_Notifier has been probed on system cms-app-49-51
2012/12/06 15:27:08 VCS NOTICE V-16-1-10438 Group CMSApp_Cluster has been probed on system cms-app-49-51
2012/12/06 15:27:09 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system cms-app-49-51
2012/12/06 15:27:09 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System cms-app-49-51 as the system is not a part of AutoStartList attribute of the group.
2012/12/06 15:28:00 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(tomcat) because the resource became OFFLINE unexpectedly, on its own.
2012/12/06 15:28:00 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(xportmanager) because the resource became OFFLINE unexpectedly, on its own.
2012/12/06 15:28:00 VCS ERROR V-16-2-13067 (cms-app-49-52) Agent is calling clean for resource(RCSSchedulerService) because the resource became OFFLINE unexpectedly, on its own.
2012/12/06 15:28:00 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:RCSSchedulerService:clean:PidFile </usr/local/tstv/RCSScheduler/config/rcsScheduler.pid> does not exist, process will not be killed
2012/12/06 15:28:00 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:tomcat:clean:PidFile </etc/init.d/tomcat.pid> does not exist, process will not be killed
2012/12/06 15:28:00 VCS WARNING V-16-10031-542 (cms-app-49-52) Application:xportmanager:clean:PidFile </etc/init.d/xportmanager.pid> does not exist, process will not be killed
2012/12/06 15:28:01 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(RCSSchedulerService) - clean completed successfully.
2012/12/06 15:28:01 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(xportmanager) - clean completed successfully.
2012/12/06 15:28:01 VCS INFO V-16-2-13068 (cms-app-49-52) Resource(tomcat) - clean completed successfully.
2012/12/06 15:28:01 VCS INFO V-16-1-10307 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/06 15:28:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:28:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:28:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:28:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:28:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:28:01 VCS INFO V-16-1-10305 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 15:28:01 VCS INFO V-16-1-10305 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 15:28:01 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 15:28:01 VCS INFO V-16-1-10306 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Previous State = OFFLINE)
2012/12/06 15:28:04 VCS INFO V-16-10031-504 (cms-app-49-52) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/06 15:28:05 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(autorsync): Output of the completed operation (offline) 
==============================================
Stopping autorsync
==============================================

2012/12/06 15:28:05 VCS INFO V-16-1-10305 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 15:28:07 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xporttomcat:offline:Executed /etc/init.d/xporttomcat as user root
2012/12/06 15:28:07 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xporttomcat): Output of the completed operation (offline) 
==============================================
Stopping XportTomcat
==============================================

2012/12/06 15:28:07 VCS INFO V-16-1-10305 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 15:28:10 VCS INFO V-16-10031-504 (cms-app-49-52) Application:cms:offline:Executed /etc/init.d/cms as user root
2012/12/06 15:28:11 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(cms): Output of the completed operation (offline) 
==============================================
Stopping Content Management System
Stopping ADI NameServer
==============================================

2012/12/06 15:28:11 VCS INFO V-16-1-10305 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 15:28:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:28:12 VCS INFO V-16-10031-504 (cms-app-49-52) Application:alerts:offline:Executed /etc/init.d/alerts as user root
2012/12/06 15:28:13 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(alerts): Output of the completed operation (offline) 
==============================================
Stopping Alerts Services... Done
==============================================

2012/12/06 15:28:13 VCS INFO V-16-1-10305 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 15:28:13 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:28:14 VCS INFO V-16-1-10305 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (VCS initiated)
2012/12/06 15:28:14 VCS ERROR V-16-1-10205 Group CMSApp_Cluster is faulted on system cms-app-49-52
2012/12/06 15:28:14 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-52
2012/12/06 15:28:14 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/06 15:28:14 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/06 15:28:14 VCS INFO V-16-1-50010 Group CMSApp_Cluster is online or faulted on system cms-app-49-52
2012/12/06 15:28:14 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:23 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:23 VCS INFO V-16-10031-504 (cms-app-49-51) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/06 15:28:23 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/06 15:28:23 VCS INFO V-16-10031-504 (cms-app-49-51) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/06 15:28:24 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xportmanager): Output of the completed operation (online) 
==============================================
Starting XportManager
==============================================

2012/12/06 15:28:24 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(tomcat): Output of the completed operation (online) 
==============================================
Starting Tomcat
==============================================

2012/12/06 15:28:24 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [60G[[0;32m  OK  [0;39m]
==============================================

2012/12/06 15:28:24 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:24 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:24 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:28 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:online:Executed /etc/init.d/xporttomcat as user root
2012/12/06 15:28:29 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (online) 
==============================================
Starting XportTomcat
==============================================

2012/12/06 15:28:29 VCS INFO V-16-1-10298 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:31 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:online:Executed /etc/init.d/autorsync as user root
2012/12/06 15:28:32 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (online) 
==============================================
Starting autorsync...
   Updates are being sent from cms-app-49-51 to cms-app-49-52 in 5 seconds intervals
==============================================

2012/12/06 15:28:32 VCS INFO V-16-1-10298 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:34 VCS INFO V-16-10031-504 (cms-app-49-51) Application:alerts:online:Executed /etc/init.d/alerts as user root
2012/12/06 15:28:35 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(alerts): Output of the completed operation (online) 
==============================================
nameservice (pid  3553) is running...
settings server (pid  3583) is running...
alerts (pid  3985) is running...
==============================================

2012/12/06 15:28:35 VCS INFO V-16-1-10298 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:28:37 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:online:Executed /etc/init.d/cms as user root
2012/12/06 15:28:38 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (online) 
==============================================
Starting Content Management System
Starting ADI NameServer
==============================================

2012/12/06 15:28:38 VCS INFO V-16-1-10298 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-51 (VCS initiated)
2012/12/06 15:28:38 VCS NOTICE V-16-1-10447 Group CMSApp_Cluster is online on system cms-app-49-51
2012/12/06 15:28:38 VCS NOTICE V-16-1-10448 Group CMSApp_Cluster failed over to system cms-app-49-51
2012/12/06 15:29:13 VCS INFO V-16-1-10307 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-52 (Not initiated by VCS)
2012/12/06 15:30:24 VCS ERROR V-16-2-13067 (cms-app-49-51) Agent is calling clean for resource(RCSSchedulerService) because the resource became OFFLINE unexpectedly, on its own.
2012/12/06 15:30:24 VCS ERROR V-16-2-13067 (cms-app-49-51) Agent is calling clean for resource(xportmanager) because the resource became OFFLINE unexpectedly, on its own.
2012/12/06 15:30:24 VCS ERROR V-16-2-13067 (cms-app-49-51) Agent is calling clean for resource(tomcat) because the resource became OFFLINE unexpectedly, on its own.
2012/12/06 15:30:24 VCS WARNING V-16-10031-542 (cms-app-49-51) Application:RCSSchedulerService:clean:PidFile </usr/local/tstv/RCSScheduler/config/rcsScheduler.pid> does not exist, process will not be killed
2012/12/06 15:30:24 VCS WARNING V-16-10031-542 (cms-app-49-51) Application:tomcat:clean:PidFile </etc/init.d/tomcat.pid> does not exist, process will not be killed
2012/12/06 15:30:24 VCS WARNING V-16-10031-542 (cms-app-49-51) Application:xportmanager:clean:PidFile </etc/init.d/xportmanager.pid> does not exist, process will not be killed
2012/12/06 15:30:25 VCS INFO V-16-2-13068 (cms-app-49-51) Resource(tomcat) - clean completed successfully.
2012/12/06 15:30:25 VCS INFO V-16-2-13068 (cms-app-49-51) Resource(xportmanager) - clean completed successfully.
2012/12/06 15:30:25 VCS INFO V-16-2-13068 (cms-app-49-51) Resource(RCSSchedulerService) - clean completed successfully.
2012/12/06 15:30:25 VCS INFO V-16-1-10307 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (Not initiated by VCS)
2012/12/06 15:30:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:30:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:30:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:30:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:30:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:30:25 VCS INFO V-16-1-10305 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/06 15:30:25 VCS INFO V-16-1-10305 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/06 15:30:25 VCS INFO V-16-1-10306 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (Previous State = OFFLINE)
2012/12/06 15:30:25 VCS INFO V-16-1-10306 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (Previous State = OFFLINE)
2012/12/06 15:30:28 VCS INFO V-16-10031-504 (cms-app-49-51) Application:autorsync:offline:Executed /etc/init.d/autorsync as user root
2012/12/06 15:30:29 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(autorsync): Output of the completed operation (offline) 
==============================================
Stopping autorsync
==============================================

2012/12/06 15:30:29 VCS INFO V-16-1-10305 Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/06 15:30:31 VCS INFO V-16-10031-504 (cms-app-49-51) Application:xporttomcat:offline:Executed /etc/init.d/xporttomcat as user root
2012/12/06 15:30:31 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(xporttomcat): Output of the completed operation (offline) 
==============================================
Stopping XportTomcat
==============================================

2012/12/06 15:30:31 VCS INFO V-16-1-10305 Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/06 15:30:35 VCS ERROR V-16-2-13067 (cms-app-49-51) Agent is calling clean for resource(alerts) because the resource became OFFLINE unexpectedly, on its own.
2012/12/06 15:30:35 VCS WARNING V-16-10031-542 (cms-app-49-51) Application:alerts:clean:PidFile </etc/init.d/alerts.pid> does not exist, process will not be killed
2012/12/06 15:30:35 VCS WARNING V-16-10031-542 (cms-app-49-51) Application:alerts:clean:PidFile </etc/init.d/alerts-nameservice.pid> does not exist, process will not be killed
2012/12/06 15:30:35 VCS WARNING V-16-10031-542 (cms-app-49-51) Application:alerts:clean:PidFile </etc/init.d/alerts-settingserver.pid> does not exist, process will not be killed
2012/12/06 15:30:36 VCS INFO V-16-10031-504 (cms-app-49-51) Application:cms:offline:Executed /etc/init.d/cms as user root
2012/12/06 15:30:36 VCS INFO V-16-2-13068 (cms-app-49-51) Resource(alerts) - clean completed successfully.
2012/12/06 15:30:36 VCS INFO V-16-1-10307 Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (Not initiated by VCS)
2012/12/06 15:30:36 VCS INFO V-16-2-13716 (cms-app-49-51) Resource(cms): Output of the completed operation (offline) 
==============================================
Stopping Content Management System
Stopping ADI NameServer
==============================================

2012/12/06 15:30:36 VCS INFO V-16-1-10305 Resource cms (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/06 15:30:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-51
2012/12/06 15:30:38 VCS INFO V-16-1-10305 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is offline on cms-app-49-51 (VCS initiated)
2012/12/06 15:30:38 VCS ERROR V-16-1-10205 Group CMSApp_Cluster is faulted on system cms-app-49-51
2012/12/06 15:30:38 VCS NOTICE V-16-1-10446 Group CMSApp_Cluster is offline on system cms-app-49-51
2012/12/06 15:30:38 VCS INFO V-16-1-10493 Evaluating cms-app-49-51 as potential target node for group CMSApp_Cluster
2012/12/06 15:30:38 VCS INFO V-16-1-50010 Group CMSApp_Cluster is online or faulted on system cms-app-49-51
2012/12/06 15:30:38 VCS INFO V-16-1-10493 Evaluating cms-app-49-52 as potential target node for group CMSApp_Cluster
2012/12/06 15:30:38 VCS NOTICE V-16-1-10301 Initiating Online of Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:30:46 VCS INFO V-16-1-10298 Resource Virtual_IP (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 15:30:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource xporttomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:30:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:30:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:30:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource autorsync (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:30:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource alerts (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:30:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) on System cms-app-49-52
2012/12/06 15:30:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:RCSSchedulerService:online:Executed /etc/init.d/RCSSchedulerService as user root
2012/12/06 15:30:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:xportmanager:online:Executed /etc/init.d/xportmanager as user root
2012/12/06 15:30:46 VCS INFO V-16-10031-504 (cms-app-49-52) Application:tomcat:online:Executed /etc/init.d/tomcat as user root
2012/12/06 15:30:47 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(xportmanager): Output of the completed operation (online) 
==============================================
XportManager is already running...
==============================================

2012/12/06 15:30:47 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(tomcat): Output of the completed operation (online) 
==============================================
Tomcat is already running...
==============================================

2012/12/06 15:30:47 VCS INFO V-16-2-13716 (cms-app-49-52) Resource(RCSSchedulerService): Output of the completed operation (online) 
==============================================
Starting RCSScheduler service: [60G[[0;32m  OK  [0;39m]
==============================================

2012/12/06 15:30:47 VCS INFO V-16-1-10298 Resource RCSSchedulerService (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 15:30:47 VCS INFO V-16-1-10298 Resource xportmanager (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
2012/12/06 15:30:47 VCS INFO V-16-1-10298 Resource tomcat (Owner: Unspecified, Group: CMSApp_Cluster) is online on cms-app-49-52 (VCS initiated)
