# # Log Name: engine # System: MILWB02S # SysInfo: Solaris:MILWB02S,Generic_144488-17,5.10,sun4u # Created: 2015/05/26 06:42:49 # 2015/05/26 06:42:49 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 06:42:49 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 06:42:49 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 06:42:49 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 06:42:49 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 06:42:49 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 06:42:49 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 06:42:49 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 06:42:54 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/26 06:42:54 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 06:42:54 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 06:42:54 VCS ERROR V-16-1-10621 Local cluster configuration error 2015/05/26 06:42:54 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 06:42:54 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 06:42:54 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/26 06:42:54 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 06:42:54 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from STALE_DISCOVER_WAIT to STALE_ADMIN_WAIT 2015/05/26 06:44:57 VCS ERROR V-16-1-10069 All systems have configuration files marked STALE. Unable to form cluster. 2015/05/26 06:44:57 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 06:44:57 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from STALE_ADMIN_WAIT to EXITED 2015/05/26 06:49:13 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 06:49:13 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 06:49:13 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 06:49:13 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 06:49:13 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 06:49:13 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 06:49:14 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 06:49:14 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 06:49:19 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/26 06:49:19 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 06:49:19 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 06:49:20 VCS ERROR V-16-1-10621 Local cluster configuration error 2015/05/26 06:49:20 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 06:49:20 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 06:49:20 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/26 06:49:20 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 06:49:20 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from STALE_DISCOVER_WAIT to STALE_ADMIN_WAIT 2015/05/26 06:49:52 VCS ERROR V-16-1-10069 All systems have configuration files marked STALE. Unable to form cluster. 2015/05/26 06:49:52 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 06:49:52 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from STALE_ADMIN_WAIT to EXITED 2015/05/26 07:00:10 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 07:00:10 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 07:00:10 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 07:00:10 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 07:00:10 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 07:00:10 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 07:00:11 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 07:00:12 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 07:00:18 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/26 07:00:18 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 07:00:18 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 07:00:20 VCS ERROR V-16-1-10621 Local cluster configuration error 2015/05/26 07:00:20 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 07:00:21 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 07:00:21 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/26 07:00:21 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 07:00:21 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from STALE_DISCOVER_WAIT to STALE_ADMIN_WAIT 2015/05/26 07:49:16 VCS ERROR V-16-1-10069 All systems have configuration files marked STALE. Unable to form cluster. 2015/05/26 07:49:16 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_FORCE_LOCAL MILWB02S from localhost 2015/05/26 07:49:16 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from STALE_ADMIN_WAIT to STALE_PEER_WAIT 2015/05/26 07:49:16 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from STALE_PEER_WAIT to LOCAL_BUILD 2015/05/26 07:49:17 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/26 07:49:17 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/26 07:49:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Tue May 26 07:49:18 2015 2015/05/26 07:49:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Tue May 26 07:49:18 2015 2015/05/26 07:49:18 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/26 07:49:18 VCS CRITICAL V-16-1-51135 GlobalCounter not updated 2015/05/26 07:49:18 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=PRC VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/26 07:49:18 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/26 07:49:18 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/26 07:49:18 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/26 07:49:18 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/26 07:49:18 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/26 07:49:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 07:49:18 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/26 07:49:19 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/26 07:49:19 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/26 07:49:19 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/26 07:49:20 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/26 07:49:20 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/26 07:49:20 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/26 07:49:20 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/26 07:49:28 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 07:49:28 VCS INFO V-16-1-50135 User root fired command: hagrp -add VVRService from localhost 2015/05/26 07:49:28 VCS INFO V-16-1-50135 User root fired command: hagrp -add AppService from localhost 2015/05/26 07:49:28 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService AutoStart 1 from localhost 2015/05/26 07:49:28 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService ManualOps 1 from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService ClusterFailOverPolicy Auto from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService FailOverPolicy Priority from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService SourceFile ./main.cf from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService ManageFaults ALL from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService FaultPropagation 1 from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService AutoStart 1 from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService ManualOps 1 from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService ClusterFailOverPolicy Auto from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService FailOverPolicy Priority from localhost 2015/05/26 07:49:29 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService SourceFile ./main.cf from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService ManageFaults ALL from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService FaultPropagation 1 from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService SystemList MILWB02S 0 from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService AutoStartList MILWB02S from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService AutoStart 0 from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ManualOps 1 from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterFailOverPolicy Auto from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService FailOverPolicy Priority from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService SourceFile ./main.cf from localhost 2015/05/26 07:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ManageFaults ALL from localhost 2015/05/26 07:49:31 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService FaultPropagation 1 from localhost 2015/05/26 07:49:31 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService SystemList MILWB02S 0 from localhost 2015/05/26 07:49:31 VCS INFO V-16-1-50135 User root fired command: hagrp -link AppService VVRService 1 1 from localhost 2015/05/26 07:49:31 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Enabled 1 from localhost 2015/05/26 07:49:31 VCS INFO V-16-1-50135 User root fired command: hares -modify wac User root from localhost 2015/05/26 07:49:31 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Critical 1 from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -modify wac StartProgram /opt/VRTSvcs/bin/wacstart from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -modify wac MonitorProcesses /opt/VRTSvcs/bin/wac from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -modify wac StopProgram /opt/VRTSvcs/bin/wacstop from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE wac MILWB02S from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -add datarvg RVG VVRService from localhost 2015/05/26 07:49:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Tue May 26 07:49:32 2015 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg RVG datarvg from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 0 from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg DiskGroup datadg from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Critical 0 from localhost 2015/05/26 07:49:32 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE datarvg MILWB02S from localhost 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg RVG datarvg from localhost 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 0 from localhost 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg DiskGroup datadg from localhost 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Critical 0 from localhost 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE datarvg MILWB02S from localhost 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -add RVGPrimary RVGPrimary AppService from localhost 2015/05/26 07:49:33 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Tue May 26 07:49:33 2015 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -add DatabaseServer Sybase AppService from localhost 2015/05/26 07:49:33 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Tue May 26 07:49:33 2015 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -add BackupServer SybaseBk AppService from localhost 2015/05/26 07:49:33 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Tue May 26 07:49:33 2015 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -add NMSServer Application AppService from localhost 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -add DataFilesystem Mount AppService from localhost 2015/05/26 07:49:33 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Tue May 26 07:49:33 2015 2015/05/26 07:49:33 VCS INFO V-16-1-50135 User root fired command: hares -add appNIC NIC AppService from localhost 2015/05/26 07:49:33 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Tue May 26 07:49:33 2015 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary RvgResourceName datarvg from localhost 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Critical 1 from localhost 2015/05/26 07:49:34 VCS ERROR V-16-20010-401 (MILWB02S) RVGPrimary:RVGPrimary:open:RVG attribute not assigned for resource ; please check the assignment of RvgResource name attribute for this resource or RVG attribute for 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary AutoResync 0 from localhost 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary AutoTakeover 1 from localhost 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE RVGPrimary MILWB02S from localhost 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SApswd sybase from localhost 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Server DBSVR from localhost 2015/05/26 07:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Enabled 0 from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Home /opt/sybase from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Critical 1 from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Owner sybase from localhost 2015/05/26 07:49:35 VCS INFO V-16-2-13716 (MILWB02S) Resource(RVGPrimary): Output of the completed operation (open) ============================================== VCS WARNING V-16-1-10260 Resource does not exist: RVG ============================================== 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Version 15.0 from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SA sa from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE DatabaseServer MILWB02S from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SApswd sybase from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Server DBSVR from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Enabled 0 from localhost 2015/05/26 07:49:35 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Home /opt/sybase from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Critical 0 from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Owner sybase from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Version 15.0 from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SA sa from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Backupserver DBSVR_back from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE BackupServer MILWB02S from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Enabled 1 from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer CleanProgram /opt/HWENGR/NMSApp/haClean from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer User root from localhost 2015/05/26 07:49:36 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Critical 1 from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer StartProgram /opt/HWENGR/NMSApp/haStart from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer MonitorProgram /opt/HWENGR/NMSApp/haMonitor from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer StopProgram /opt/HWENGR/NMSApp/haStop from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MILWB02S from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem MountPoint /opt/sybase/data from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem SecondLevelMonitor 0 from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Enabled 1 from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Critical 1 from localhost 2015/05/26 07:49:37 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem BlockDevice /dev/vx/dsk/datadg/lv_nms_data from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem CkptUmount 1 from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem FsckOpt -y from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem SecondLevelTimeout 30 from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem FSType ufs from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE DataFilesystem MILWB02S from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/26 07:49:38 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC NetworkHosts 10.66.28.67 from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Device bge1 from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Enabled 1 from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC PingOptimize 1 from localhost 2015/05/26 07:49:38 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Critical 0 from localhost 2015/05/26 07:49:39 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE appNIC MILWB02S from localhost 2015/05/26 07:49:39 VCS INFO V-16-1-50135 User root fired command: hares -link RVGPrimary appNIC 0 0 from localhost 2015/05/26 07:49:39 VCS INFO V-16-1-50135 User root fired command: hares -link DatabaseServer DataFilesystem 0 0 from localhost 2015/05/26 07:49:39 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 07:49:39 VCS INFO V-16-1-50135 User root fired command: hares -link BackupServer DatabaseServer 0 0 from localhost 2015/05/26 07:49:39 VCS INFO V-16-1-50135 User root fired command: hares -link NMSServer BackupServer 0 0 from localhost 2015/05/26 07:49:39 VCS INFO V-16-1-50135 User root fired command: hares -link DataFilesystem RVGPrimary 0 0 from localhost 2015/05/26 07:49:42 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 07:49:45 VCS INFO V-16-1-50135 User root fired command: hagrp -offline -force ClusterService MILWB02S localclus from localhost 2015/05/26 07:49:45 VCS NOTICE V-16-1-10167 Initiating manual offline of group ClusterService on system MILWB02S 2015/05/26 07:49:45 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 07:49:46 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/26 07:49:46 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/26 07:49:47 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 07:49:47 VCS INFO V-16-1-50135 User root fired command: haclus -modify ClusterName MILWB02SCluster from localhost 2015/05/26 07:49:47 VCS INFO V-16-1-50135 User root fired command: haclus -modify ClusterAddress 10.66.28.53 from localhost 2015/05/26 07:49:47 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/26 07:49:47 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/26 07:49:50 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 07:49:53 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS_FORCE from localhost 2015/05/26 07:49:53 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/26 07:49:53 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING_FORCIBLY 2015/05/26 07:49:53 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/26 07:49:53 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING_FORCIBLY to EXITED 2015/05/26 07:49:53 VCS INFO V-16-1-50016 Setting SYSSTATE to RSM_EXITED for node : MILWB02S. 2015/05/26 07:49:58 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 07:49:58 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 07:49:58 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 07:49:58 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 07:49:58 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 07:49:58 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 07:49:58 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 07:49:58 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 07:50:03 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/26 07:50:03 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 07:50:03 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 07:50:03 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 07:50:03 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 07:50:03 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/26 07:50:03 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 07:50:03 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/26 07:50:04 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/26 07:50:04 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Tue May 26 07:50:04 2015 2015/05/26 07:50:04 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/26 07:50:04 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin PWD=/var/VRTSvcs/diag/had TERM=sun TZ=PRC VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=MILWB02S _INIT_UTS_PLATFORM=SUNW,SPARC-Enterprise _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_144488-17 _INIT_ZONENAME=global 2015/05/26 07:50:04 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/26 07:50:04 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/26 07:50:04 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/26 07:50:04 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/26 07:50:04 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/26 07:50:04 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/26 07:50:04 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/26 07:50:04 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 07:50:04 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/26 07:50:04 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/26 07:50:04 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/26 07:50:04 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/26 07:50:06 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/26 07:50:06 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/26 07:50:06 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/26 07:50:06 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/26 07:50:06 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 07:50:06 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 07:50:06 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/26 07:50:06 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/26 08:18:12 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze AppService from localhost 2015/05/26 08:25:03 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 08:25:03 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SApswd DLGpLPePRpKPiPQpINsJEjFJg from localhost 2015/05/26 08:25:03 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SApswd DLGpLPePRpKPiPQpINsJEjFJg from localhost 2015/05/26 08:25:03 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 08:50:15 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 08:50:15 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SApswd emhQmqFqsQlqJqrQjoTkfKgkH from localhost 2015/05/26 08:50:15 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SApswd emhQmqFqsQlqJqrQjoTkfKgkH from localhost 2015/05/26 08:50:15 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 09:05:25 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 09:06:25 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 09:07:26 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 09:12:32 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 09:12:32 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 09:12:32 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 09:12:32 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 09:12:32 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 09:12:32 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 09:12:32 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 09:12:33 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 09:12:38 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/26 09:12:38 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 09:12:38 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 09:12:39 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 09:12:39 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 09:12:39 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/26 09:12:39 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 09:12:39 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/26 09:12:40 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/26 09:12:40 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:40 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Tue May 26 09:12:40 2015 2015/05/26 09:12:41 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/26 09:12:41 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 09:12:41 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/26 09:12:41 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/26 09:12:41 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/26 09:12:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/26 09:12:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 09:12:42 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=PRC VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/26 09:12:42 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/26 09:12:42 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/26 09:12:42 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/26 09:12:42 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/26 09:12:42 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/26 09:12:43 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/26 09:12:43 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/26 09:12:43 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/26 09:12:43 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/26 09:12:43 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 09:12:43 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 09:12:43 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/26 09:12:43 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/26 09:19:25 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 09:19:25 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/26 09:19:25 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/26 09:19:25 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 09:19:25 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/26 09:19:25 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/26 09:19:25 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 09:19:25 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 09:19:25 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/26 09:19:26 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/05/26 09:19:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/26 09:19:26 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/26 09:19:26 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/26 09:19:26 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/26 09:19:26 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/26 09:19:26 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/26 09:19:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/26 09:19:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:19:26 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 09:19:26 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 09:19:26 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 09:19:27 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 09:19:28 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/26 09:19:28 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 09:19:30 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:19:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:19:31 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:19:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:19:33 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/26 09:19:33 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/26 09:19:33 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/26 09:19:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/26 09:19:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/26 09:19:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/26 09:19:55 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/26 09:19:55 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5757/psinfo 2015/05/26 09:19:55 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:19:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:20:08 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/10818/psinfo 2015/05/26 09:20:08 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:20:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:20:18 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/26 09:20:19 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 5757 5754 2 09:19:34 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 8971 5757 0 09:19:51 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8986, 10000000000, 0x1c39 sybase 8917 5757 1 09:19:51 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8986, 10000000000, 0x1c39 sybase 9020 5757 0 09:19:51 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8986, 10000000000, 0x1c39 15-05-26 09:20:08 || haStart:sybase is starting, break! 15-05-26 09:20:08 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-26 09:20:18 || haStart:The imap is running, break! 15-05-26 09:20:18 || haStart:Success to start the nms! ============================================== 2015/05/26 09:20:27 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:20:27 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 09:35:00 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.69 has wrong password 2015/05/26 09:35:06 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.69 has wrong password 2015/05/26 09:35:10 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.69 has wrong password 2015/05/26 09:35:19 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/26 09:49:18 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB02S from ::ffff:10.66.28.69 2015/05/26 09:49:18 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:50:20 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/26 09:50:21 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... NmslogzipDm_4440....................stopping... PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM................ ============================================== 2015/05/26 09:50:23 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 09:51:25 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 09:51:25 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze -persistent AppService from localhost 2015/05/26 09:51:25 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 09:51:55 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/26 09:51:55 VCS INFO V-16-1-10307 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/26 09:52:07 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/26 09:52:07 VCS INFO V-16-1-10307 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/26 09:54:07 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 09:54:07 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze -persistent AppService from localhost 2015/05/26 09:54:07 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 09:54:07 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 09:54:07 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze -persistent AppService from localhost 2015/05/26 09:54:07 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 09:55:28 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 09:55:28 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze -persistent AppService from localhost 2015/05/26 09:55:28 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 09:56:17 VCS INFO V-16-1-50135 User admin fired command: hares -clear BackupServer MILWB02S from ::ffff:10.66.28.69 2015/05/26 09:56:17 VCS INFO V-16-1-10307 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/26 09:56:22 VCS INFO V-16-1-50135 User admin fired command: hares -clear DatabaseServer MILWB02S from ::ffff:10.66.28.69 2015/05/26 09:56:22 VCS INFO V-16-1-10307 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/26 09:56:29 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MILWB02S from ::ffff:10.66.28.69 2015/05/26 09:56:29 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/26 09:56:29 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:56:29 VCS ERROR V-16-20018-13 (MILWB02S) Sybase:DatabaseServer:online:Sybase service DBSVR is already running 2015/05/26 09:56:30 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6327/psinfo 2015/05/26 09:56:30 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:56:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:56:30 VCS ERROR V-16-20018-24 (MILWB02S) SybaseBk:BackupServer:online:Sybase Sysbackup service DBSVR_back is already running 2015/05/26 09:56:32 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/6328/psinfo 2015/05/26 09:56:32 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:56:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 09:56:42 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/26 09:56:43 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 6405 6327 0 09:55:19 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6327 6324 0 09:55:08 ? 0:14 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 6389 6327 0 09:55:19 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 6406 6327 0 09:55:19 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 15-05-26 09:56:32 || haStart:sybase is starting, break! 15-05-26 09:56:32 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-26 09:56:42 || haStart:The imap is running, break! 15-05-26 09:56:42 || haStart:Success to start the nms! ============================================== 2015/05/26 09:56:49 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 09:56:49 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 11:12:33 VCS INFO V-16-1-50135 User admin fired command: haconf -dump from ::ffff:10.66.28.69 2015/05/26 13:09:18 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/26 17:09:20 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/26 20:05:38 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 20:05:38 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/26 20:05:38 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 20:05:38 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 20:05:38 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/26 20:05:38 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/26 20:05:39 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/26 20:05:39 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/26 20:05:39 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/26 20:05:39 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/26 20:05:39 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/26 20:06:12 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/26 20:06:13 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/26 20:06:38 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 20:06:40 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/26 20:06:41 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopping... PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped ToolkitService......................stopped TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopped EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopped FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopped SystemService.......................stopp ============================================== 2015/05/26 20:06:43 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 20:06:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 20:06:43 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/26 20:06:44 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/26 20:06:44 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/26 20:06:44 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 20:06:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 20:06:45 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/26 20:06:45 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/26 20:06:45 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/26 20:06:45 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 20:06:45 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 20:06:47 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 20:06:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 20:06:49 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/26 20:06:51 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 20:06:51 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/26 20:06:51 VCS NOTICE V-16-1-50061 Cannot initiate online of group AppService; this group may go online after its children fail over 2015/05/26 20:06:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/26 20:06:53 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/26 20:06:54 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/26 20:06:54 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/26 20:06:54 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/26 20:06:54 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/26 20:06:54 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/26 20:06:54 VCS WARNING V-16-1-50047 Clearing parent group AppService's migrateq, since child can not go online anywhere 2015/05/26 20:06:54 VCS NOTICE V-16-1-10500 Resetting Evacuating (lastval=0) for group AppService 2015/05/26 20:06:54 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/26 20:06:54 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/26 20:06:54 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/26 14:10:53 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 14:10:53 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 14:10:53 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 14:10:53 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 14:10:53 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 14:10:53 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 14:10:53 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 14:10:53 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 14:10:58 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/26 14:10:58 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 14:10:58 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 14:10:59 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 14:10:59 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 14:10:59 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/26 14:10:59 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 14:10:59 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/26 14:11:00 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/26 14:11:00 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Tue May 26 14:11:00 2015 2015/05/26 14:11:00 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/26 14:11:01 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:11:01 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:11:01 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/26 14:11:01 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/26 14:11:01 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/26 14:11:01 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/26 14:11:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 14:11:01 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:11:01 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/26 14:11:01 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/26 14:11:01 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/26 14:11:01 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/26 14:11:01 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/26 14:11:01 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/26 14:11:01 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/26 14:11:02 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/26 14:11:02 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/26 14:11:02 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/26 14:11:02 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/26 14:11:02 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/26 14:11:02 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/26 14:11:02 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/26 14:11:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/26 14:11:02 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/26 14:11:02 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/26 14:11:03 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered ============================================== 2015/05/26 14:11:03 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:11:03 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:11:03 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/26 14:11:03 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/26 14:11:03 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/26 14:11:03 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/26 14:11:04 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/26 14:11:04 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/26 14:17:51 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/26 14:17:52 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/26 14:17:52 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/26 14:17:52 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/26 14:17:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:17:52 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:17:52 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:17:52 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:17:53 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:17:53 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/26 14:17:54 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:17:55 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:17:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:17:57 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:17:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:17:58 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/26 14:17:59 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/26 14:17:59 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/26 14:17:59 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/26 14:17:59 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/26 14:17:59 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/26 14:18:19 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/26 14:18:20 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6108/psinfo 2015/05/26 14:18:20 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:18:20 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:18:32 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/10777/psinfo 2015/05/26 14:18:32 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:18:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:18:42 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/26 14:18:43 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9199 6108 0 14:18:16 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6108 6105 2 14:17:59 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9151 6108 0 14:18:16 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 9247 6108 0 14:18:16 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 15-05-26 14:18:32 || haStart:sybase is starting, break! 15-05-26 14:18:32 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-26 14:18:42 || haStart:The imap is running, break! 15-05-26 14:18:42 || haStart:Success to start the nms! ============================================== 2015/05/26 14:18:55 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:18:55 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 14:22:08 VCS WARNING V-16-10001-7506 (MILWB02S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/26 14:22:09 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/05/26 14:22:09 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:10 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB02S (Not initiated by VCS) 2015/05/26 14:23:10 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 14:23:11 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/26 14:23:11 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... NmslogzipDm_4440....................stopping... PortTrunkingService.................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopping... UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM................ ============================================== 2015/05/26 14:23:13 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:23:13 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:14 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/26 14:23:15 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/26 14:23:15 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/26 14:23:15 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:23:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:15 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/26 14:23:16 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/26 14:23:16 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/26 14:23:16 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:23:16 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:17 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:23:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:19 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/26 14:23:22 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:23:22 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/26 14:23:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:22 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/26 14:23:24 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:23:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:25 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:23:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:26 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/26 14:23:26 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/26 14:23:26 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/26 14:23:26 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/26 14:23:26 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/26 14:23:26 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/26 14:23:42 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/26 14:23:43 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/16464/psinfo 2015/05/26 14:23:43 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:23:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:23:54 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/18353/psinfo 2015/05/26 14:23:54 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:23:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:24:05 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/26 14:24:05 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 17048 16464 0 14:23:38 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 17143 16464 0 14:23:38 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 17096 16464 0 14:23:38 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 16464 16462 2 14:23:26 ? 0:14 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-26 14:23:55 || haStart:sybase is starting, break! 15-05-26 14:23:55 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-26 14:24:05 || haStart:The imap is running, break! 15-05-26 14:24:05 || haStart:Success to start the nms! ============================================== 2015/05/26 14:24:12 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:24:12 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 14:35:28 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 14:35:28 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/26 14:35:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:35:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 14:35:28 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/26 14:35:28 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/26 14:35:29 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:35:29 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/26 14:35:29 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/26 14:35:29 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/26 14:35:29 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/26 14:35:53 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/26 14:35:54 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/26 14:36:27 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 14:36:30 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/26 14:36:30 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopping... PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped ToolkitService......................stopped TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopped EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopped FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopped SystemService.......................stopp ============================================== 2015/05/26 14:36:32 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:36:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:36:32 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/26 14:36:33 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/26 14:36:33 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/26 14:36:33 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:36:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:36:34 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/26 14:36:34 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/26 14:36:35 VCS CRITICAL V-16-20018-4 (MILWB02S) Sybase:DatabaseServer:monitor:Process dataserver restarted 2015/05/26 14:36:35 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:36:35 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:36:36 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:36:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:36:38 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/26 14:36:40 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:36:40 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/26 14:36:40 VCS NOTICE V-16-1-50061 Cannot initiate online of group AppService; this group may go online after its children fail over 2015/05/26 14:36:40 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/26 14:36:42 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/26 14:36:43 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:36:43 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/26 14:36:43 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/26 14:36:43 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/26 14:36:43 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/26 14:36:43 VCS WARNING V-16-1-50047 Clearing parent group AppService's migrateq, since child can not go online anywhere 2015/05/26 14:36:43 VCS NOTICE V-16-1-10500 Resetting Evacuating (lastval=0) for group AppService 2015/05/26 14:36:43 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/26 14:36:43 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/26 14:36:43 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/26 14:40:40 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 14:40:40 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 14:40:40 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 14:40:40 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 14:40:40 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 14:40:40 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 14:40:40 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 14:40:40 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 14:40:45 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/26 14:40:45 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 14:40:45 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 14:40:46 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 14:40:46 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 14:40:46 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/26 14:40:46 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 14:40:46 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/26 14:40:47 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/26 14:40:47 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Tue May 26 14:40:47 2015 2015/05/26 14:40:48 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/26 14:40:48 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:40:48 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:40:48 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:40:48 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/26 14:40:48 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/26 14:40:48 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/26 14:40:48 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/26 14:40:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/26 14:40:49 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/26 14:40:49 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/26 14:40:49 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/26 14:40:49 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/26 14:40:49 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/26 14:40:49 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/26 14:40:49 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/26 14:40:49 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/26 14:40:49 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/26 14:40:49 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/26 14:40:49 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/26 14:40:49 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/26 14:40:49 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/26 14:40:49 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/26 14:40:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/26 14:40:50 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/26 14:40:50 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/26 14:40:50 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered ============================================== 2015/05/26 14:40:50 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:40:50 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/26 14:40:50 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/26 14:40:51 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/26 14:40:51 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/26 14:40:51 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/26 14:40:51 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/26 14:40:51 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/26 14:47:40 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/26 14:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/26 14:47:41 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/26 14:47:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/26 14:47:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:47:41 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:47:41 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:47:41 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:47:42 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:47:42 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/26 14:47:43 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:47:44 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:47:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:47:45 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:47:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:47:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/26 14:47:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/26 14:47:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/26 14:47:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/26 14:47:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/26 14:47:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/26 14:48:08 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/26 14:48:09 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6119/psinfo 2015/05/26 14:48:09 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:48:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:48:21 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/10831/psinfo 2015/05/26 14:48:21 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:48:21 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:48:31 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/26 14:48:32 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9254 6119 0 14:48:05 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6119 6117 2 14:47:48 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9207 6119 0 14:48:05 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 9308 6119 0 14:48:05 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 15-05-26 14:48:21 || haStart:sybase is starting, break! 15-05-26 14:48:21 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-26 14:48:31 || haStart:The imap is running, break! 15-05-26 14:48:31 || haStart:Success to start the nms! ============================================== 2015/05/26 14:48:41 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:48:41 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 14:51:56 VCS WARNING V-16-10001-7506 (MILWB02S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/26 14:51:57 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/05/26 14:51:57 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:52:58 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB02S (Not initiated by VCS) 2015/05/26 14:52:58 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 14:52:59 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/26 14:52:59 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... NmslogzipDm_4440....................stopping... PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopping... UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM................ ============================================== 2015/05/26 14:53:01 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:53:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:02 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/26 14:53:02 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/26 14:53:03 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/26 14:53:03 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:53:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:03 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/26 14:53:04 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/26 14:53:04 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/26 14:53:04 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:53:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:05 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:53:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:07 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/26 14:53:09 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/26 14:53:09 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/26 14:53:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:10 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/26 14:53:12 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:53:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:13 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:53:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:14 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/26 14:53:14 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/26 14:53:14 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/26 14:53:14 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/26 14:53:14 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/26 14:53:14 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/26 14:53:29 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/26 14:53:30 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/17838/psinfo 2015/05/26 14:53:30 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:53:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:40 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/19458/psinfo 2015/05/26 14:53:40 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:53:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/26 14:53:51 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/26 14:53:51 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 17838 17836 2 14:53:14 ? 0:13 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 18254 17838 0 14:53:25 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 18303 17838 0 14:53:25 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 18208 17838 0 14:53:25 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-26 14:53:41 || haStart:sybase is starting, break! 15-05-26 14:53:41 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-26 14:53:51 || haStart:The imap is running, break! 15-05-26 14:53:51 || haStart:Success to start the nms! ============================================== 2015/05/26 14:53:57 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/26 14:53:57 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/26 18:37:30 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/26 22:37:31 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 02:37:32 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 06:37:33 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 08:13:55 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 08:13:55 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 08:13:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:13:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 08:13:56 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/27 08:13:56 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 08:13:57 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:13:57 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/27 08:13:57 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/27 08:13:57 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 08:13:57 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/27 08:14:42 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/27 08:14:43 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/27 08:14:55 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 08:14:57 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 08:14:58 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopping... PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped ToolkitService......................stopped TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopped EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopped FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopped SystemService.......................stopp ============================================== 2015/05/27 08:15:01 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:15:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:15:01 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 08:15:02 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 08:15:02 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 08:15:02 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:15:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:15:02 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 08:15:03 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 08:15:03 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 08:15:03 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:15:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:15:04 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:15:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:15:06 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 08:15:09 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:15:09 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/27 08:15:09 VCS NOTICE V-16-1-50061 Cannot initiate online of group AppService; this group may go online after its children fail over 2015/05/27 08:15:09 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/27 08:15:10 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 08:15:11 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:15:11 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/27 08:15:11 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/27 08:15:11 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/27 08:15:11 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 08:15:11 VCS WARNING V-16-1-50047 Clearing parent group AppService's migrateq, since child can not go online anywhere 2015/05/27 08:15:11 VCS NOTICE V-16-1-10500 Resetting Evacuating (lastval=0) for group AppService 2015/05/27 08:15:11 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/27 08:15:11 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/27 08:15:11 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/27 08:19:11 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 08:19:11 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 08:19:11 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 08:19:11 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 08:19:11 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 08:19:11 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 08:19:11 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 08:19:11 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 08:19:16 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/27 08:19:16 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 08:19:16 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 08:19:17 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 08:19:17 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 08:19:17 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/27 08:19:17 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 08:19:17 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 08:19:18 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 08:19:18 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 08:19:18 2015 2015/05/27 08:19:18 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 08:19:19 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:19:19 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:19:19 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:19:19 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/27 08:19:19 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/27 08:19:19 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 08:19:19 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 08:19:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 08:19:19 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 08:19:19 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 08:19:19 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 08:19:19 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/27 08:19:19 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 08:19:19 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 08:19:19 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/27 08:19:20 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/27 08:19:20 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/27 08:19:20 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/27 08:19:20 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/27 08:19:20 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/27 08:19:20 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/27 08:19:20 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 08:19:20 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/27 08:19:20 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/27 08:19:20 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/27 08:19:21 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered ============================================== 2015/05/27 08:19:21 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/27 08:19:21 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 08:19:21 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:19:21 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:19:21 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/27 08:19:21 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 08:19:22 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/27 08:19:22 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/27 08:24:55 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 08:24:55 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 08:24:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 08:24:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/27 08:24:55 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/27 08:24:55 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 08:24:56 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:24:56 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/27 08:24:56 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/27 08:24:56 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 08:24:56 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/27 08:24:57 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/27 08:24:57 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/27 08:24:58 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 08:24:59 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:24:59 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/27 08:24:59 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/27 08:24:59 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/27 08:24:59 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 08:24:59 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/27 08:24:59 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/27 08:24:59 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/27 08:33:04 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 08:33:04 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 08:33:04 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 08:33:04 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 08:33:04 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 08:33:04 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 08:33:04 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 08:33:04 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 08:33:09 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/27 08:33:09 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 08:33:09 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 08:33:10 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 08:33:10 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 08:33:10 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/27 08:33:10 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 08:33:10 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 08:33:11 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 08:33:11 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:11 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 08:33:11 2015 2015/05/27 08:33:12 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 08:33:12 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:33:12 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:33:12 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:33:12 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/27 08:33:12 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/27 08:33:12 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 08:33:12 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 08:33:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 08:33:12 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 08:33:12 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/27 08:33:12 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 08:33:12 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 08:33:13 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 08:33:13 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 08:33:13 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/27 08:33:13 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/27 08:33:13 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/27 08:33:13 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/27 08:33:13 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/27 08:33:13 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/27 08:33:13 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/27 08:33:13 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 08:33:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/27 08:33:14 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/27 08:33:14 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/27 08:33:14 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:33:14 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered ============================================== 2015/05/27 08:33:14 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/27 08:33:14 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 08:33:14 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 08:33:14 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/27 08:33:14 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 08:33:15 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/27 08:33:15 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/27 08:40:03 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/27 08:40:03 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/27 08:40:03 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 08:40:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:40:03 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:40:03 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:40:03 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:40:04 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:40:04 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/27 08:40:05 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:40:06 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:40:06 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:40:08 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:40:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:40:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 08:40:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 08:40:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 08:40:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 08:40:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/27 08:40:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 08:40:31 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 08:40:32 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6218/psinfo 2015/05/27 08:40:32 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:40:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:40:43 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/10860/psinfo 2015/05/27 08:40:43 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:40:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:40:54 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 08:40:54 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9374 6218 0 08:40:28 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 9280 6218 0 08:40:27 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 6218 6216 2 08:40:10 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9327 6218 0 08:40:28 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 15-05-27 08:40:44 || haStart:sybase is starting, break! 15-05-27 08:40:44 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 08:40:54 || haStart:The imap is running, break! 15-05-27 08:40:54 || haStart:Success to start the nms! ============================================== 2015/05/27 08:41:06 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:41:06 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 08:44:19 VCS WARNING V-16-10001-7506 (MILWB02S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 08:44:20 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/05/27 08:44:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:22 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB02S (Not initiated by VCS) 2015/05/27 08:45:22 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 08:45:22 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 08:45:23 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopping... PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped ToolkitService......................stopped TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopped SystemService... ============================================== 2015/05/27 08:45:25 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:45:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:25 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 08:45:26 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 08:45:26 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 08:45:26 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:45:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:26 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 08:45:27 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 08:45:27 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 08:45:27 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:45:27 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:28 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:45:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:31 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 08:45:33 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 08:45:33 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/27 08:45:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:33 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/27 08:45:35 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:45:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:37 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:45:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:45:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 08:45:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 08:45:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 08:45:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 08:45:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/27 08:45:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 08:45:52 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 08:45:53 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/16519/psinfo 2015/05/27 08:45:53 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:45:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:46:04 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/18209/psinfo 2015/05/27 08:46:04 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:46:04 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 08:46:14 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 08:46:15 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 16519 16517 2 08:45:38 ? 0:14 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 16970 16519 0 08:45:49 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 17015 16519 0 08:45:49 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 16927 16519 0 08:45:49 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-27 08:46:04 || haStart:sybase is starting, break! 15-05-27 08:46:04 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 08:46:14 || haStart:The imap is running, break! 15-05-27 08:46:14 || haStart:Success to start the nms! ============================================== 2015/05/27 08:46:21 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 08:46:21 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 09:41:54 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 09:41:54 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 09:41:54 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:41:54 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 09:41:55 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/27 09:41:55 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 09:41:56 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:41:56 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/27 09:41:56 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/27 09:41:56 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 09:41:56 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/27 09:42:05 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/27 09:42:05 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/27 09:42:55 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 09:42:57 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 09:42:58 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopping... PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped ToolkitService......................stopped TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopped EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopped FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopped SystemService.......................stopp ============================================== 2015/05/27 09:43:00 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:43:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:43:00 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 09:43:01 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 09:43:01 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 09:43:01 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:43:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:43:01 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 09:43:02 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 09:43:02 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 09:43:02 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:43:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:43:03 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:43:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:43:06 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 09:43:08 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:43:08 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/27 09:43:08 VCS NOTICE V-16-1-50061 Cannot initiate online of group AppService; this group may go online after its children fail over 2015/05/27 09:43:08 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/27 09:43:09 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 09:43:10 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:43:10 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/27 09:43:10 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/27 09:43:10 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/27 09:43:10 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 09:43:10 VCS WARNING V-16-1-50047 Clearing parent group AppService's migrateq, since child can not go online anywhere 2015/05/27 09:43:10 VCS NOTICE V-16-1-10500 Resetting Evacuating (lastval=0) for group AppService 2015/05/27 09:43:10 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/27 09:43:10 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/27 09:43:10 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/27 09:47:10 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 09:47:10 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 09:47:10 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 09:47:10 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 09:47:10 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 09:47:10 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 09:47:10 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 09:47:11 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 09:47:16 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/27 09:47:16 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 09:47:16 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 09:47:16 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 09:47:16 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 09:47:16 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/27 09:47:16 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 09:47:16 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 09:47:17 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 09:47:17 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 09:47:18 2015 2015/05/27 09:47:18 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 09:47:19 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 09:47:19 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 09:47:19 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 09:47:19 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 09:47:19 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/27 09:47:19 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/27 09:47:19 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/27 09:47:19 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 09:47:19 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 09:47:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 09:47:19 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 09:47:19 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 09:47:19 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 09:47:19 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 09:47:19 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/27 09:47:20 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/27 09:47:20 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/27 09:47:20 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/27 09:47:20 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/27 09:47:20 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/27 09:47:20 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/27 09:47:20 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 09:47:20 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/27 09:47:20 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/27 09:47:20 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/27 09:47:21 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 09:47:21 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/27 09:47:21 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/27 09:47:21 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 09:47:21 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/27 09:47:21 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 09:47:21 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered ============================================== 2015/05/27 09:47:22 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/27 09:47:22 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/27 09:54:09 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 09:54:09 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 09:54:09 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/27 09:54:09 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 09:54:10 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/27 09:54:10 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/27 09:54:10 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/27 09:54:10 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/27 09:54:10 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/27 09:54:10 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/27 09:54:10 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/27 09:54:10 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 09:54:10 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:54:10 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:54:10 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:54:10 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:54:11 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:54:11 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/27 09:54:12 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 09:54:13 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 09:54:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:54:14 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 09:54:14 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:54:16 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 09:54:17 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 09:54:17 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 09:54:17 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 09:54:17 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/27 09:54:17 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 09:54:38 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 09:54:39 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6152/psinfo 2015/05/27 09:54:39 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 09:54:39 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:54:49 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/10973/psinfo 2015/05/27 09:54:49 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 09:54:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 09:55:00 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 09:55:00 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9461 6152 0 09:54:35 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 9414 6152 0 09:54:35 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6152 6149 2 09:54:17 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9368 6152 0 09:54:34 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-27 09:54:50 || haStart:sybase is starting, break! 15-05-27 09:54:50 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 09:55:00 || haStart:The imap is running, break! 15-05-27 09:55:00 || haStart:Success to start the nms! ============================================== 2015/05/27 09:55:08 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 09:55:08 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 10:18:48 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.70 has wrong password 2015/05/27 10:18:57 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.70 has wrong password 2015/05/27 10:19:03 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.70 has wrong password 2015/05/27 10:19:10 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/27 10:53:04 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB02S from ::ffff:10.66.28.70 2015/05/27 10:53:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 10:54:07 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 10:54:08 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... NmslogzipDm_4440....................stopping... PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopped TopoService.........................stopping... UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................s ============================================== 2015/05/27 10:54:10 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 10:54:21 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MILWB02S from ::ffff:10.66.28.70 2015/05/27 10:54:21 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 10:54:21 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 10:54:32 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 10:54:32 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9461 6152 0 09:54:35 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 9414 6152 0 09:54:35 ? 0:16 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6152 6149 0 09:54:17 ? 0:53 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9368 6152 0 09:54:34 ? 0:26 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-27 10:54:22 || haStart:sybase is starting, break! 15-05-27 10:54:22 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 10:54:32 || haStart:The imap is running, break! 15-05-27 10:54:32 || haStart:Success to start the nms! ============================================== 2015/05/27 10:54:38 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 10:54:38 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 10:58:13 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB02S from ::ffff:10.66.28.70 2015/05/27 10:58:13 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 10:59:15 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 10:59:16 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... NmslogzipDm_4440....................stopping... PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopped PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM................ ============================================== 2015/05/27 10:59:18 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 10:59:31 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MILWB02S from ::ffff:10.66.28.70 2015/05/27 10:59:31 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 10:59:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 10:59:42 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 10:59:42 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9461 6152 0 09:54:35 ? 0:23 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 9414 6152 0 09:54:35 ? 0:24 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6152 6149 0 09:54:17 ? 1:02 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9368 6152 0 09:54:34 ? 0:32 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-27 10:59:31 || haStart:sybase is starting, break! 15-05-27 10:59:32 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 10:59:42 || haStart:The imap is running, break! 15-05-27 10:59:42 || haStart:Success to start the nms! ============================================== 2015/05/27 10:59:48 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 10:59:48 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 11:37:42 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB02S from ::ffff:10.66.28.70 2015/05/27 11:37:42 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 11:38:44 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 11:38:45 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... DCServer............................stopping... DmsNetAclDm.........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopped NmslogzipDm_4440....................stopping... Notify_Service......................stopped PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopping... UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ Agent_CORBA.........................stopping... DCServer............................stopping... DmsNetAclDm.........................stopped EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_monitomcat......................stopping... FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopped ItmService................ ============================================== 2015/05/27 11:38:48 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 12:31:10 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 12:31:11 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze AppService localclus from localhost 2015/05/27 12:31:11 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 12:44:58 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 12:44:59 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze AppService localclus from localhost 2015/05/27 12:44:59 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 12:51:40 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MILWB02S from ::ffff:10.66.28.70 2015/05/27 12:51:40 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 12:51:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 12:51:50 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 12:51:51 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9461 6152 0 09:54:35 ? 4:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 9414 6152 0 09:54:35 ? 1:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6152 6149 0 09:54:17 ? 3:32 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9368 6152 0 09:54:34 ? 2:43 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-27 12:51:40 || haStart:sybase is starting, break! 15-05-27 12:51:40 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 12:51:50 || haStart:The imap is running, break! 15-05-27 12:51:50 || haStart:Success to start the nms! ============================================== 2015/05/27 12:52:01 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 12:52:01 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 12:52:23 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 85% 2015/05/27 12:56:31 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 12:56:31 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 12:56:32 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources ClusterService from localhost 2015/05/27 12:56:32 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/27 12:56:32 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 12:57:22 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 12:57:23 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/27 12:57:42 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/27 12:57:58 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 12:58:35 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 12:58:36 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/27 12:58:36 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 12:59:55 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 12:59:55 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/27 12:59:56 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 12:59:57 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS_FORCE from localhost 2015/05/27 12:59:57 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 12:59:57 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 12:59:57 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/27 12:59:57 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 12:59:58 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/27 12:59:58 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/27 12:59:58 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/27 12:59:58 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 12:59:58 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/27 12:59:58 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/27 12:59:58 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/27 13:00:19 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 13:00:19 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 13:00:19 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 13:00:19 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 13:00:19 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 13:00:19 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 13:00:19 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 13:00:19 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 13:00:23 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/27 13:00:23 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 13:00:23 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 13:00:23 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 13:00:23 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 13:00:23 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/27 13:00:23 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 13:00:23 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 13:00:24 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 13:00:24 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 13:00:25 2015 2015/05/27 13:00:25 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 13:00:25 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger VCS_LOG=/var/VRTSvcs ########## Other Environment Variables ########## ACE_ROOT=/opt/U2000/server/3rdTools/ACE_wrappers AGENTPP_ROOT=/opt/U2000/server/3rdTools/agent++ CC_COMPILER=ForteCC CC_SW=cc CLRCASE_ROOT=/usr/atria CPPUNIT_ROOT=/opt/U2000/server/3rdTools/cppunit-1.8.0 CPU_SW=SUN_CPU DBSVR=DBSVR DES_ROOT=/opt/U2000/server/3rdTools/libdes DOCGEN_ROOT=/opt/doxygen-1.2.0 EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs FILEIO_LOCAL_CHARSET=1 FLEX_ROOT=/opt/U2000/server/3rdTools/flex-2.5.4 HWENGRROOT=/opt/HWENGR HWICMRROOT=/opt/HWICMR HWNMSJRE=/opt/HWNMSJRE ILOG_LICENSE_FILE=/opt/U2000/server/3rdTools/ilog/ilm/access.ilm ILOG_ROOT=/opt/U2000/server/3rdTools/ilog IMAP_CONFIG_PATH=/opt/U2000/server/etc/conf IMAP_DBTYPE=SYBASE IMAP_FAST_BUILD=1 IMAP_JAVA_HOME=/opt/HWNMSJRE/jre_sol IMAP_LOCALE=UTF-8 IMAP_MO_XML_TEMPLATE_DIR=/opt/U2000/server/etc/conf IMAP_ORBTYPE=TAO IMAP_ROOT=/opt/U2000/server INCLUDE=/opt/sybase/OCS-15_0/include:/opt/U2000/server/3rdTools/sybase/include:/opt/U2000/server/3rdTools/sybase/include _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=MILWB02S _INIT_UTS_PLATFORM=SUNW,SPARC-Enterprise _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_144488-17 _INIT_ZONENAME=global IT_CONFIG_DOMAINS_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/domains IT_DOMAIN_NAME=nbiorbix IT_LICENSE_FILE=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/licenses.txt IT_PRODUCT_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix LANG=en_US.UTF-8 LAN_IFNAME_0=e1000g0 LAN_IFNAME_1=e1000g1 LAN_IFNAME_2=e1000g2 LAN_IFNAME_3=e1000g3 LAN_nxge_0=nxge0 LAN_nxge_1=nxge1 LAN_nxge_2=nxge2 LAN_nxge_3=nxge3 LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH=/opt/VRTSvcs/lib:/opt/HWNMSJRE/jre_sol/lib/sparc/server:/opt/HWNMSJRE/jre_sol/lib/sparc:/opt/HWNMSJRE/jre_sol/../lib/sparc:/opt/sybase/OCS-15_0/lib:/opt/sybase/OCS-15_0/lib3p64:/opt/sybase/OCS-15_0/lib3p:/opt/sybase/ASE-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/opt/U2000/server/common/toolkit/server:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/lib:/usr/sfw/lib:/opt/SUNWspro/WS6U2/lib:/usr/local/lib:/opt/U2000/server/3rdTools/postgresql/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/usr/lib:/usr/dt/lib:/usr/openwin/lib:/opt/U2000/client/lib:/opt/U2000/server/cbb/frame/lib:/lib:/usr/ucblib LD_POINTER_GUARD=1 LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 LD_PRELOAD=libumem.so LIB=/opt/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib LOCALHOST_FIXSERVERNAME=10.66.28.69 LOCALHOST_FLOATSERVERNAME=10.66.28.69 MAKEFILE=makefile.tao MANPATH=:/opt/U2000/server/doc/cxx/man MASTERHOST_FLOATSERVERNAME=10.66.28.69 MED_SEND_ALARM_DIRECTLY=1 NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat NO_CLEARMAKE=1 OPERATION_SYSTEM=SUN_SVR4_OPSYS OSS_ROOT=/opt/U2000 PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/sybase/RPL-15_0/bin:/opt/sybase/ASEP/bin:/opt/sybase/DBISQL/bin:/opt/sybase/UAF-2_0/bin:/opt/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/bin:/opt/sybase/ASE-15_0/install:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/opt/U2000/server/3rdTools/bin:.:/export/forte6/SUNWspro/bin:/opt/U2000/server/3rdTools/postgresql/bin:/opt/U2000/server/3rdTools/usr/local/bin:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/3rdTools/flex-2.5.4/bin:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/usr/bin:/usr/sbin:/usr/openwin/bin:/usr/dt/bin:/usr/sfw/bin:/sbin:/opt/U2000/server/bin:/opt/U2000/client:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/U2000/server/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/U2000/server/3rdTools/sms/bin PGDATA=/op 2015/05/27 13:00:25 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/27 13:00:25 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/10973/psinfo 2015/05/27 13:00:25 VCS INFO V-16-1-10297 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:00:25 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:00:25 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6152/psinfo 2015/05/27 13:00:25 VCS INFO V-16-1-10297 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:00:25 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:00:25 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:00:25 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:00:25 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/27 13:00:25 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/27 13:00:25 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 13:00:25 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 13:00:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 13:00:25 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 13:00:25 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 13:00:25 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 13:00:25 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/27 13:00:26 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/05/27 13:00:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 13:00:26 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/27 13:00:26 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/27 13:00:27 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/27 13:00:27 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/27 13:00:27 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/27 13:00:27 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 13:00:27 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:00:27 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:00:29 VCS INFO V-16-1-10297 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:00:29 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:00:29 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/27 13:00:29 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 13:00:29 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 13:00:30 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 13:00:30 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 13:00:31 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 13:00:32 VCS INFO V-16-1-50135 User root fired command: hahb -add Icmp from localhost 2015/05/27 13:00:34 VCS INFO V-16-1-50135 User root fired command: haclus -add MILWB03SCluster 10.66.28.54 from localhost 2015/05/27 13:00:34 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/27 13:00:34 VCS INFO V-16-1-50135 User root fired command: hahb -modify ... -add Icmp ClusterList MILWB03SCluster from localhost 2015/05/27 13:00:34 VCS INFO V-16-1-50135 User root fired command: hahb -modify Icmp AYARetryLimit 9 from localhost 2015/05/27 13:00:35 VCS INFO V-16-1-50135 User root fired command: hahb -modify Icmp Arguments 10.66.28.54 MILWB03SCluster from localhost 2015/05/27 13:00:35 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB03SCluster 0 MILWB02SCluster 1 from localhost 2015/05/27 13:00:35 VCS NOTICE V-16-1-50920 Local cluster is assuming Authority for group AppService since Authority has not been defined and group is online in local cluster 2015/05/27 13:00:35 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/27 13:00:35 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 13:00:36 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 13:00:36 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS_FORCE from localhost 2015/05/27 13:00:36 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 13:00:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 13:00:36 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/27 13:00:36 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 13:00:38 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/27 13:00:38 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/27 13:00:38 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/27 13:00:38 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/27 13:00:38 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/27 13:00:38 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/27 13:00:38 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/27 13:00:58 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 13:00:58 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 13:00:58 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 13:00:58 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 13:00:58 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 13:00:58 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 13:00:58 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 13:00:58 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 13:01:03 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/27 13:01:03 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 13:01:03 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 13:01:03 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 13:01:03 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 13:01:03 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/27 13:01:03 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 13:01:03 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 13:01:04 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 13:01:04 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 13:01:04 2015 2015/05/27 13:01:04 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 13:01:05 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger VCS_LOG=/var/VRTSvcs ########## Other Environment Variables ########## ACE_ROOT=/opt/U2000/server/3rdTools/ACE_wrappers AGENTPP_ROOT=/opt/U2000/server/3rdTools/agent++ CC_COMPILER=ForteCC CC_SW=cc CLRCASE_ROOT=/usr/atria CPPUNIT_ROOT=/opt/U2000/server/3rdTools/cppunit-1.8.0 CPU_SW=SUN_CPU DBSVR=DBSVR DES_ROOT=/opt/U2000/server/3rdTools/libdes DOCGEN_ROOT=/opt/doxygen-1.2.0 EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs FILEIO_LOCAL_CHARSET=1 FLEX_ROOT=/opt/U2000/server/3rdTools/flex-2.5.4 HWENGRROOT=/opt/HWENGR HWICMRROOT=/opt/HWICMR HWNMSJRE=/opt/HWNMSJRE ILOG_LICENSE_FILE=/opt/U2000/server/3rdTools/ilog/ilm/access.ilm ILOG_ROOT=/opt/U2000/server/3rdTools/ilog IMAP_CONFIG_PATH=/opt/U2000/server/etc/conf IMAP_DBTYPE=SYBASE IMAP_FAST_BUILD=1 IMAP_JAVA_HOME=/opt/HWNMSJRE/jre_sol IMAP_LOCALE=UTF-8 IMAP_MO_XML_TEMPLATE_DIR=/opt/U2000/server/etc/conf IMAP_ORBTYPE=TAO IMAP_ROOT=/opt/U2000/server INCLUDE=/opt/sybase/OCS-15_0/include:/opt/U2000/server/3rdTools/sybase/include:/opt/U2000/server/3rdTools/sybase/include _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=MILWB02S _INIT_UTS_PLATFORM=SUNW,SPARC-Enterprise _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_144488-17 _INIT_ZONENAME=global IT_CONFIG_DOMAINS_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/domains IT_DOMAIN_NAME=nbiorbix IT_LICENSE_FILE=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/licenses.txt IT_PRODUCT_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix LANG=en_US.UTF-8 LAN_IFNAME_0=e1000g0 LAN_IFNAME_1=e1000g1 LAN_IFNAME_2=e1000g2 LAN_IFNAME_3=e1000g3 LAN_nxge_0=nxge0 LAN_nxge_1=nxge1 LAN_nxge_2=nxge2 LAN_nxge_3=nxge3 LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH=/opt/VRTSvcs/lib:/opt/HWNMSJRE/jre_sol/lib/sparc/server:/opt/HWNMSJRE/jre_sol/lib/sparc:/opt/HWNMSJRE/jre_sol/../lib/sparc:/opt/sybase/OCS-15_0/lib:/opt/sybase/OCS-15_0/lib3p64:/opt/sybase/OCS-15_0/lib3p:/opt/sybase/ASE-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/opt/U2000/server/common/toolkit/server:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/lib:/usr/sfw/lib:/opt/SUNWspro/WS6U2/lib:/usr/local/lib:/opt/U2000/server/3rdTools/postgresql/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/usr/lib:/usr/dt/lib:/usr/openwin/lib:/opt/U2000/client/lib:/opt/U2000/server/cbb/frame/lib:/lib:/usr/ucblib LD_POINTER_GUARD=1 LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 LD_PRELOAD=libumem.so LIB=/opt/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib LOCALHOST_FIXSERVERNAME=10.66.28.69 LOCALHOST_FLOATSERVERNAME=10.66.28.69 MAKEFILE=makefile.tao MANPATH=:/opt/U2000/server/doc/cxx/man MASTERHOST_FLOATSERVERNAME=10.66.28.69 MED_SEND_ALARM_DIRECTLY=1 NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat NO_CLEARMAKE=1 OPERATION_SYSTEM=SUN_SVR4_OPSYS OSS_ROOT=/opt/U2000 PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/sybase/RPL-15_0/bin:/opt/sybase/ASEP/bin:/opt/sybase/DBISQL/bin:/opt/sybase/UAF-2_0/bin:/opt/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/bin:/opt/sybase/ASE-15_0/install:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/opt/U2000/server/3rdTools/bin:.:/export/forte6/SUNWspro/bin:/opt/U2000/server/3rdTools/postgresql/bin:/opt/U2000/server/3rdTools/usr/local/bin:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/3rdTools/flex-2.5.4/bin:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/usr/bin:/usr/sbin:/usr/openwin/bin:/usr/dt/bin:/usr/sfw/bin:/sbin:/opt/U2000/server/bin:/opt/U2000/client:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/U2000/server/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/U2000/server/3rdTools/sms/bin PGDATA=/op 2015/05/27 13:01:05 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/27 13:01:05 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6152/psinfo 2015/05/27 13:01:05 VCS INFO V-16-1-10297 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:01:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:01:05 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/10973/psinfo 2015/05/27 13:01:05 VCS INFO V-16-1-10297 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:01:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:01:05 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:01:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:01:05 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/27 13:01:05 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/27 13:01:05 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 13:01:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 13:01:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/27 13:01:05 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 13:01:05 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 13:01:05 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/27 13:01:05 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/27 13:01:05 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 13:01:05 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/27 13:01:06 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/05/27 13:01:06 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 13:01:06 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/27 13:01:06 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/27 13:01:06 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/27 13:01:06 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/27 13:01:07 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/27 13:01:07 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 13:01:07 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:01:07 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:01:09 VCS INFO V-16-1-10297 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/27 13:01:09 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 13:01:09 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/27 13:01:09 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 13:01:09 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 13:01:12 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/27 13:01:12 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:12 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:13 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:51 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:51 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:51 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:52 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:52 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:01:53 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 13:02:02 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 13:02:02 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 13:02:03 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources ClusterService from localhost 2015/05/27 13:02:03 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/27 13:02:04 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 13:02:39 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/27 13:02:43 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/27 13:02:53 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 13:02:53 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 13:02:53 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 13:03:47 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.70 has wrong password 2015/05/27 13:03:54 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/27 13:43:57 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 14:50:30 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/27 14:50:30 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/27 14:52:59 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/27 14:54:59 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/27 14:54:59 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 14:54:59 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/27 14:54:59 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/27 14:57:04 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/27 15:42:52 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/27 15:42:52 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/27 15:47:17 VCS WARNING V-16-3-18325 (MILWB02S) Cluster (MILWB03SCluster) refused connection. This is because all heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are currently down. 2015/05/27 15:47:23 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/27 15:47:23 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:47:23 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/27 15:55:11 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/27 15:55:11 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/27 15:59:35 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/27 15:59:35 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:35 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/27 15:59:37 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 15:59:37 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 17:19:36 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/27 17:32:40 VCS NOTICE V-16-1-50086 CPU usage on MILWB02S is 73% 2015/05/27 17:41:43 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB02S from ::ffff:10.66.28.69 2015/05/27 17:41:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 17:42:46 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 17:42:46 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopped TrapReceiver........................stopping... UniteUitlDM................ ============================================== 2015/05/27 17:42:48 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 17:43:33 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MILWB02S from ::ffff:10.66.28.69 2015/05/27 17:43:33 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 17:43:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 17:43:44 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 17:43:44 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9461 6152 0 09:54:35 ? 7:27 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 9414 6152 0 09:54:35 ? 6:39 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 6152 6149 0 09:54:17 ? 8:25 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9368 6152 0 09:54:34 ? 5:44 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-27 17:43:33 || haStart:sybase is starting, break! 15-05-27 17:43:33 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 17:43:44 || haStart:The imap is running, break! 15-05-27 17:43:44 || haStart:Success to start the nms! ============================================== 2015/05/27 17:43:56 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 17:43:56 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 17:43:58 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 17:44:10 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 87% 2015/05/27 17:57:09 VCS INFO V-16-1-50135 User admin fired command: hagrp -offline AppService MILWB02S localclus from ::ffff:10.66.28.69 2015/05/27 17:57:09 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/05/27 17:57:09 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 17:58:11 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 17:58:12 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopping... InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopped Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopped Notify_Service......................stopping... PMDataRefiner.......................stopped PMSDm...............................stopped PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping.. ============================================== 2015/05/27 17:58:14 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 17:58:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 17:58:15 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 17:58:15 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 17:58:15 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 17:58:15 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 17:58:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 17:58:17 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 17:58:18 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 17:58:18 VCS CRITICAL V-16-20018-4 (MILWB02S) Sybase:DatabaseServer:monitor:Process dataserver restarted 2015/05/27 17:58:18 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 17:58:18 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 17:58:19 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 17:58:19 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 17:58:21 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 17:58:23 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 17:58:23 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/27 17:58:41 VCS INFO V-16-1-50135 User admin fired command: haconf -dump from ::ffff:10.66.28.69 2015/05/27 17:59:31 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB03SCluster to online group AppService on system MILWB03S 2015/05/27 17:59:31 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB03SCluster to online group AppService on system MILWB03S 2015/05/27 17:59:31 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/27 17:59:31 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 17:59:31 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/27 17:59:31 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 17:59:43 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 17:59:44 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:00:12 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:00:24 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:00:40 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:00:40 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:51:07 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:51:08 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:51:11 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:51:12 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:51:16 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:51:16 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 18:51:46 VCS INFO V-16-1-50135 User admin fired command: hagrp -online AppService MILWB02S localclus from ::ffff:10.66.28.69 2015/05/27 18:51:46 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/05/27 18:51:46 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/27 18:51:46 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/27 18:51:46 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 18:51:46 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/27 18:51:46 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 18:51:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 18:51:46 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/27 18:51:56 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/27 18:51:58 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 18:51:58 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 18:51:59 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 18:51:59 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 18:52:00 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase/ASE-15_0/install/startserver -f /opt/sybase/ASE-15_0/install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 18:52:00 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 18:52:00 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 18:52:00 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 18:52:00 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/27 18:52:01 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 18:52:23 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 18:52:24 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/12948/psinfo 2015/05/27 18:52:24 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 18:52:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 18:52:35 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/16275/psinfo 2015/05/27 18:52:35 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 18:52:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 18:52:46 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 18:52:46 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 12948 12946 2 18:52:01 ? 0:18 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 14391 12948 0 18:52:16 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 14359 12948 1 18:52:16 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 14422 12948 1 18:52:16 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 15-05-27 18:52:35 || haStart:sybase is starting, break! 15-05-27 18:52:35 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 18:52:45 || haStart:The imap is running, break! 15-05-27 18:52:45 || haStart:Success to start the nms! ============================================== 2015/05/27 18:53:00 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 18:53:00 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 18:53:10 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 80% 2015/05/27 18:53:40 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 97% 2015/05/27 19:09:25 VCS INFO V-16-1-50860 Attempting to switch group AppService from system MILWB02S to best possible system in cluster MILWB03SCluster 2015/05/27 19:09:25 VCS INFO V-16-1-50135 User admin fired command: hagrp -switch -any AppService MILWB03SCluster from ::ffff:10.66.28.69 2015/05/27 19:09:25 VCS NOTICE V-16-1-50930 Initial tests indicate group AppService is able to switch to remote cluster MILWB03SCluster. Initiating offline of group on system MILWB02S 2015/05/27 19:09:25 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/05/27 19:09:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:10:28 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 19:10:29 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopping... Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopped TrapReceiver........................stopping... UniteUitl ============================================== 2015/05/27 19:10:31 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 19:10:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:10:31 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 19:10:32 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 19:10:32 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 19:10:32 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 19:10:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:10:34 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 19:10:34 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 19:10:34 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 19:10:34 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 19:10:34 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:10:35 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 19:10:35 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:10:38 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 19:10:40 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/27 19:10:40 VCS INFO V-16-1-50998 Proceeding to online Group AppService on ANY system in local cluster MILWB03SCluster 2015/05/27 19:10:40 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/27 19:10:40 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/27 19:10:40 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/27 19:10:40 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/27 19:10:40 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 19:10:40 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/27 19:10:40 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 19:11:06 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 19:11:07 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 19:11:29 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 19:11:41 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 19:12:07 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 19:12:07 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 19:44:35 VCS ERROR V-16-3-18313 (MILWB02S) TCP/IP connection via connector from cluster MILWB02SCluster to cluster MILWB03SCluster is hung; intentionally disconnecting. Auto-reconnect will occur however you may wish to examine the wac resource on system 10.66.28.54 2015/05/27 19:44:35 VCS ERROR V-16-3-18311 (MILWB02S) MILWB02SCluster lost connection to cluster MILWB03SCluster 2015/05/27 19:51:26 VCS ERROR V-16-3-18211 (MILWB02S) Cluster MILWB02SCluster lost heartbeat Icmp to cluster MILWB03SCluster 2015/05/27 19:51:26 VCS NOTICE V-16-3-18213 (MILWB02S) Cluster MILWB02SCluster lost all heartbeats to cluster MILWB03SCluster: effecting inquiry 2015/05/27 19:51:26 VCS ERROR V-16-3-18314 (MILWB02S) Cluster MILWB02SCluster lost all heartbeats to cluster MILWB03SCluster: no inquiry peers 2015/05/27 19:51:26 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/05/27 19:51:26 VCS ERROR V-16-1-50908 Remote cluster MILWB03SCluster has faulted. Determining if global group AppService should be failed over to local cluster 2015/05/27 19:51:26 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/05/27 19:51:26 VCS INFO V-16-1-50904 Cluster MILWB03SCluster is not in RUNNING state. Skipping... 2015/05/27 19:51:26 VCS INFO V-16-1-50899 Evaluating local cluster MILWB02SCluster as potential target for group AppService 2015/05/27 19:51:26 VCS INFO V-16-1-50902 Found a possible target system in local cluster 2015/05/27 19:51:26 VCS ERROR V-16-1-50910 Local cluster was determined to be the next suitable failover target for Group AppService. Initiating online of group 2015/05/27 19:51:26 VCS CRITICAL V-16-1-50513 Remote cluster 'MILWB03SCluster' has faulted 2015/05/27 19:51:26 VCS ERROR V-16-5-51071 ALERT[CFAULT-MILWB03SCluster-MILWB03SCluster] Remote cluster MILWB03SCluster has faulted. Administrative action is required. 2015/05/27 19:51:26 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/27 19:51:26 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 19:51:26 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/05/27 19:51:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 19:51:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:51:26 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/27 19:51:26 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/27 19:51:26 VCS INFO V-16-20010-79 (MILWB02S) RVGPrimary:RVGPrimary:online:_p_p_mode = AS 2015/05/27 19:51:26 VCS INFO V-16-20010-79 (MILWB02S) RVGPrimary:RVGPrimary:online:It is a AS mode. 2015/05/27 19:51:27 VCS INFO V-16-20010-79 (MILWB02S) RVGPrimary:RVGPrimary:online:touch /opt/HWENGR/NMSApp/flag/To_Stop. 2015/05/27 19:51:27 VCS INFO V-16-20010-80 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg takeover successful; use vxrvg resync command to resynchronize the original primary upon its restoration 2015/05/27 19:51:28 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 19:51:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:51:30 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 19:51:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:51:30 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase/ASE-15_0/install/startserver -f /opt/sybase/ASE-15_0/install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 19:51:30 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 19:51:30 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 19:51:30 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 19:51:30 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/27 19:51:30 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 19:52:04 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 19:52:04 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/14245/psinfo 2015/05/27 19:52:04 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 19:52:04 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:52:15 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/20264/psinfo 2015/05/27 19:52:15 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 19:52:15 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/27 19:52:26 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 19:52:27 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 14245 14243 1 19:51:31 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 15563 14245 1 19:51:46 ? 0:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 15496 14245 0 19:51:45 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 15531 14245 2 19:51:46 ? 0:15 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 15-05-27 19:52:16 || haStart:sybase is starting, break! 15-05-27 19:52:16 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 19:52:26 || haStart:The imap is running, break! 15-05-27 19:52:26 || haStart:Success to start the nms! ============================================== 2015/05/27 19:52:41 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/27 19:52:41 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/27 19:53:10 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 100% 2015/05/27 20:06:06 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/27 20:06:06 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/27 20:06:06 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:06:06 VCS ERROR V-16-1-50981 Resource appNIC is faulted on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:06:06 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50840 ALERT[CFAULT-MILWB03SCluster-MILWB03SCluster] was deleted automatically by the engine because alert no longer exists on cluster MILWB03SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/27 20:06:50 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:06:50 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:13:31 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:13:31 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:13:31 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:13:32 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:13:32 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:13:33 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:13:48 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:13:49 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/27 20:14:08 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/27 20:14:24 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:14:48 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:14:48 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:14:48 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:14:49 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:14:49 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:14:50 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:15:51 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/05/27 20:15:51 VCS WARNING V-16-20010-301 (MILWB02S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/05/27 20:15:52 VCS ERROR V-16-20010-302 (MILWB02S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/05/27 20:15:52 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/05/27 20:15:52 VCS ERROR V-16-2-13069 (MILWB02S) Resource(datarvg) - clean failed. 2015/05/27 20:16:32 VCS NOTICE V-16-1-50517 Resource NMSServer has been deleted from cluster MILWB03SCluster 2015/05/27 20:16:32 VCS NOTICE V-16-1-50517 Resource DataFilesystem has been deleted from cluster MILWB03SCluster 2015/05/27 20:16:32 VCS NOTICE V-16-1-50517 Resource appNIC has been deleted from cluster MILWB03SCluster 2015/05/27 20:16:32 VCS NOTICE V-16-1-50517 Resource RVGPrimary has been deleted from cluster MILWB03SCluster 2015/05/27 20:16:32 VCS NOTICE V-16-1-50517 Resource DatabaseServer has been deleted from cluster MILWB03SCluster 2015/05/27 20:16:32 VCS NOTICE V-16-1-50517 Resource BackupServer has been deleted from cluster MILWB03SCluster 2015/05/27 20:16:32 VCS NOTICE V-16-1-50518 Group AppService has been deleted from cluster MILWB03SCluster 2015/05/27 20:16:32 VCS INFO V-16-1-50957 Updating ClusterList for group AppService in local cluster configuration to reflect group deletion on cluster MILWB03SCluster 2015/05/27 20:16:32 VCS INFO V-16-1-50958 Local cluster dumping configuration to save ClusterList modification for group AppService 2015/05/27 20:16:33 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:16:35 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/27 20:16:35 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:16:39 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:16:39 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB02SCluster and remote cluster MILWB03SCluster 2015/05/27 20:16:39 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:16:39 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:16:39 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:16:39 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:16:39 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:16:39 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/27 20:16:39 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB02SCluster and remote cluster MILWB03SCluster 2015/05/27 20:16:41 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:16:42 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB03SCluster 0 MILWB02SCluster 1 from localhost 2015/05/27 20:16:43 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:16:43 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:16:45 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/27 20:17:04 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/27 20:17:20 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:22:22 VCS INFO V-16-1-50135 User admin fired command: haconf -dump from ::ffff:10.66.28.69 2015/05/27 21:43:59 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 01:44:00 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 05:44:01 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 08:36:04 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/28 08:36:59 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB02S from ::ffff:10.66.28.69 2015/05/28 08:36:59 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 08:38:02 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/28 08:38:02 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopped NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopped TrapReceiver........................stopping... U ============================================== 2015/05/28 08:38:04 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 08:50:10 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MILWB02S from ::ffff:10.66.28.69 2015/05/28 08:50:10 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 08:50:10 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 08:50:21 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 08:50:22 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 14245 14243 0 19:51:31 ? 23:06 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 15563 14245 0 19:51:46 ? 31:31 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 15496 14245 0 19:51:45 ? 15:25 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 15531 14245 0 19:51:46 ? 17:45 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 15-05-28 08:50:10 || haStart:sybase is starting, break! 15-05-28 08:50:10 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 08:50:21 || haStart:The imap is running, break! 15-05-28 08:50:21 || haStart:Success to start the nms! ============================================== 2015/05/28 08:50:40 VCS INFO V-16-1-50086 CPU usage on MILWB02S is 63% 2015/05/28 08:50:46 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 08:50:46 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/28 08:51:10 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 97% 2015/05/28 08:53:40 VCS NOTICE V-16-1-50086 CPU usage on MILWB02S is 73% 2015/05/28 09:44:02 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 10:13:33 VCS INFO V-16-1-50135 User admin fired command: haconf -makerw from ::ffff:10.66.28.69 2015/05/28 10:13:33 VCS INFO V-16-1-50135 User admin fired command: hagrp -freeze -persistent AppService from ::ffff:10.66.28.69 2015/05/28 10:14:11 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/28 10:14:11 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/28 10:18:47 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/28 10:18:47 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:47 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/28 10:18:47 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:20:08 VCS INFO V-16-1-50135 User admin fired command: hagrp -unfreeze -persistent AppService from ::ffff:10.66.28.69 2015/05/28 10:20:56 VCS INFO V-16-1-50860 Attempting to switch group AppService from system MILWB02S to best possible system in cluster MILWB03SCluster 2015/05/28 10:20:56 VCS INFO V-16-1-50135 User admin fired command: hagrp -switch -any AppService MILWB03SCluster from ::ffff:10.66.28.69 2015/05/28 10:20:56 VCS NOTICE V-16-1-50930 Initial tests indicate group AppService is able to switch to remote cluster MILWB03SCluster. Initiating offline of group on system MILWB02S 2015/05/28 10:20:56 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/05/28 10:20:56 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 10:21:58 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/28 10:21:59 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopped Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopped SyslogCollectorDM...................stopped SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... Unite ============================================== 2015/05/28 10:22:01 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 10:22:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 10:22:22 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/28 10:22:22 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/28 10:22:22 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/28 10:22:22 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 10:22:22 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 10:22:23 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/28 10:22:24 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/28 10:22:25 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/28 10:22:25 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 10:22:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 10:22:26 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 10:22:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 10:22:28 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/28 10:22:30 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 10:22:30 VCS INFO V-16-1-50998 Proceeding to online Group AppService on ANY system in local cluster MILWB03SCluster 2015/05/28 10:22:30 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/28 10:22:30 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/28 10:22:30 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/28 10:22:30 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/28 10:22:30 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 10:22:30 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/28 10:22:30 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 10:23:19 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:23:20 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:23:56 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:24:07 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:24:49 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:24:49 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:29:50 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/28 10:29:50 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/28 10:29:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/28 10:29:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/28 10:29:51 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/28 10:29:51 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/28 10:29:52 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/28 10:29:52 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/28 10:29:52 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/28 10:29:52 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/28 10:29:52 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/28 10:29:52 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/28 10:29:53 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/28 10:29:54 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 10:29:54 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/28 10:29:54 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/28 10:29:54 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/28 10:29:54 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/28 10:29:54 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/28 10:29:54 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/28 10:29:54 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/28 10:29:54 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/28 10:34:06 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/28 10:34:06 VCS INFO V-16-1-10196 Cluster logger started 2015/05/28 10:34:06 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/28 10:34:06 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/28 10:34:06 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/28 10:34:06 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/28 10:34:06 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/28 10:34:07 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/28 10:34:11 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/28 10:34:11 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/28 10:34:11 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/28 10:34:12 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/28 10:34:12 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/28 10:34:12 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/28 10:34:12 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/28 10:34:12 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/28 10:34:13 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/28 10:34:13 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu May 28 10:34:14 2015 2015/05/28 10:34:14 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/28 10:34:15 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 10:34:15 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 10:34:15 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 10:34:15 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/28 10:34:15 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/28 10:34:15 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/28 10:34:15 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/28 10:34:15 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/28 10:34:15 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 10:34:15 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/28 10:34:15 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/28 10:34:15 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/28 10:34:16 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/28 10:34:16 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/28 10:34:16 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/28 10:34:16 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/28 10:34:16 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/28 10:34:16 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/28 10:34:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/28 10:34:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/28 10:34:16 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/28 10:34:16 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/28 10:34:16 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/28 10:34:16 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/28 10:34:16 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/28 10:34:16 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/28 10:34:17 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/28 10:34:17 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/28 10:34:17 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/28 10:34:18 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 10:34:18 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/28 10:34:18 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/28 10:34:18 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 10:34:19 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/28 10:34:19 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/28 10:34:21 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/28 10:34:21 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/28 10:34:21 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 10:34:21 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/28 10:38:39 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/28 11:40:30 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/05/28 11:40:30 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/05/28 11:41:35 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:41:36 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:41:37 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:41:39 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:41:43 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:41:43 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:41:43 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/05/28 11:41:43 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/28 11:41:43 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/28 11:41:43 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 11:41:43 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/28 11:41:43 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 11:41:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 11:41:44 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/28 11:42:08 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/28 11:42:10 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 11:42:10 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 11:42:11 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 11:42:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 11:42:13 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/28 11:42:13 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/28 11:42:13 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/28 11:42:13 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/28 11:42:13 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/28 11:42:13 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/28 11:42:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/28 11:42:37 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/14984/psinfo 2015/05/28 11:42:37 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 11:42:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 11:42:49 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/20526/psinfo 2015/05/28 11:42:49 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 11:42:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 11:43:01 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 11:43:01 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 14984 14982 2 11:42:13 ? 0:18 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 17766 14984 0 11:42:30 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 17720 14984 0 11:42:30 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 17811 14984 1 11:42:30 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 15-05-28 11:42:50 || haStart:sybase is starting, break! 15-05-28 11:42:50 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 11:43:00 || haStart:The imap is running, break! 15-05-28 11:43:00 || haStart:Success to start the nms! ============================================== 2015/05/28 11:43:36 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 11:43:36 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/28 11:43:50 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 93% 2015/05/28 11:46:38 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/28 11:46:38 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/28 11:54:31 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/28 11:54:31 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 11:54:31 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/28 11:54:31 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/28 12:03:07 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/28 12:03:07 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/28 12:07:44 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/28 12:07:44 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:44 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/28 12:07:44 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:10:11 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/28 12:10:11 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/28 12:18:29 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/28 12:18:29 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:29 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/28 12:18:29 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/28 12:18:31 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:18:31 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 12:55:36 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MILWB02S from localhost 2015/05/28 12:55:36 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/05/28 12:55:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 12:56:38 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/28 12:56:38 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopping... TrapReceiver........................stopping... UniteUitlDM.. ============================================== 2015/05/28 12:56:40 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 12:56:40 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 12:56:40 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/28 12:56:41 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/28 12:56:41 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/28 12:56:41 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 12:56:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 12:56:42 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/28 12:56:43 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/28 12:56:43 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/28 12:56:43 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 12:56:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 12:56:44 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 12:56:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 12:56:46 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/28 12:56:48 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 12:56:48 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/28 12:58:34 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/28 12:58:46 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze -persistent AppService from localhost 2015/05/28 12:58:55 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/28 13:01:17 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/28 13:01:17 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/28 13:01:17 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/28 13:01:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/28 13:01:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/28 13:01:17 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/28 13:01:17 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/28 13:01:18 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/28 13:01:18 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/28 13:01:18 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/28 13:01:18 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/28 13:01:18 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/28 13:01:18 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/28 13:01:20 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 13:01:21 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/28 13:01:21 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/28 13:01:21 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/28 13:01:21 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/28 13:01:21 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/28 13:01:21 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/28 13:01:21 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/28 13:01:21 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/28 13:10:06 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/28 13:10:06 VCS INFO V-16-1-10196 Cluster logger started 2015/05/28 13:10:06 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/28 13:10:06 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/28 13:10:06 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/28 13:10:06 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/28 13:10:06 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/28 13:10:06 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/28 13:10:11 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/28 13:10:11 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/28 13:10:11 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/28 13:10:12 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/28 13:10:12 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/28 13:10:12 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/28 13:10:12 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/28 13:10:12 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/28 13:10:13 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/28 13:10:13 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:13 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu May 28 13:10:13 2015 2015/05/28 13:10:14 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/28 13:10:14 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:10:15 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:10:15 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:10:15 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/28 13:10:15 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/28 13:10:15 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/28 13:10:15 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/28 13:10:15 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/28 13:10:15 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/28 13:10:15 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 13:10:15 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/28 13:10:15 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/28 13:10:15 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/28 13:10:15 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/28 13:10:15 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/28 13:10:15 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/28 13:10:16 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/28 13:10:16 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/28 13:10:16 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/28 13:10:16 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/28 13:10:16 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/28 13:10:16 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/28 13:10:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/28 13:10:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/28 13:10:16 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/28 13:10:16 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/28 13:10:16 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/28 13:10:17 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/28 13:10:17 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/28 13:10:17 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/28 13:10:17 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:10:18 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:10:18 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/28 13:10:18 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/28 13:10:18 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/28 13:10:18 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/28 13:10:18 VCS NOTICE V-16-1-10482 Cannot initiate online of group AppService; group is frozen 2015/05/28 13:10:22 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/28 13:10:22 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:10:22 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/28 13:11:00 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/28 13:11:00 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/05/28 13:11:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/28 13:11:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/28 13:11:00 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/05/28 13:11:00 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/28 13:11:02 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/05/28 13:11:02 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/05/28 13:11:02 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group ClusterService 2015/05/28 13:11:02 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/28 13:11:02 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB02S 2015/05/28 13:11:02 VCS INFO V-16-20010-222 (MILWB02S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/28 13:11:02 VCS INFO V-16-20010-223 (MILWB02S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/28 13:11:03 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 13:11:04 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (VCS initiated) 2015/05/28 13:11:04 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB02S 2015/05/28 13:11:04 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/28 13:11:04 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/28 13:11:04 VCS INFO V-16-1-10494 System MILWB02S not in RUNNING state 2015/05/28 13:11:04 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB02S 2015/05/28 13:11:04 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/05/28 13:11:04 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/05/28 13:21:00 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/28 13:21:00 VCS INFO V-16-1-10196 Cluster logger started 2015/05/28 13:21:00 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/28 13:21:00 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/28 13:21:00 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/28 13:21:00 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/28 13:21:00 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/28 13:21:00 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/28 13:21:05 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/28 13:21:05 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/28 13:21:05 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/28 13:21:05 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/28 13:21:05 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/28 13:21:05 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/28 13:21:05 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/28 13:21:05 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/28 13:21:06 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/28 13:21:06 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:06 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu May 28 13:21:06 2015 2015/05/28 13:21:07 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/28 13:21:07 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:21:07 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:21:08 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:21:08 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/28 13:21:08 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/28 13:21:08 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/28 13:21:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/28 13:21:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/28 13:21:08 VCS WARNING V-16-20010-224 (MILWB02S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 13:21:08 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/28 13:21:08 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/28 13:21:08 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/28 13:21:08 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB02S (First probe) 2015/05/28 13:21:08 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/28 13:21:08 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/28 13:21:08 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group VVRService 2015/05/28 13:21:08 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB02S; hence child group VVRService may not come online on system MILWB02S 2015/05/28 13:21:08 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB02S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/28 13:21:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/28 13:21:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB02S 2015/05/28 13:21:08 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/28 13:21:08 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/28 13:21:08 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/28 13:21:08 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/28 13:21:09 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/28 13:21:09 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/28 13:21:09 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/28 13:21:09 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/28 13:21:09 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/28 13:21:11 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:21:11 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (VCS initiated) 2015/05/28 13:21:11 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/28 13:21:11 VCS NOTICE V-16-1-10482 Cannot initiate online of group AppService; group is frozen 2015/05/28 13:21:11 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/28 13:21:11 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/28 13:21:11 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/28 13:21:13 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/28 13:21:15 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/28 13:21:15 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 13:21:15 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/28 13:22:54 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/28 13:23:02 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze -persistent AppService from localhost 2015/05/28 13:23:09 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/28 13:23:32 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/28 13:23:32 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/28 13:23:32 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 13:23:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 13:23:32 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/28 13:23:34 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 13:23:34 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 13:23:35 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 13:23:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 13:23:37 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/28 13:23:38 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/28 13:23:38 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/28 13:23:38 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/28 13:23:38 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/28 13:23:38 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/28 13:24:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/28 13:24:03 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5520/psinfo 2015/05/28 13:24:03 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 13:24:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 13:24:14 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/11614/psinfo 2015/05/28 13:24:14 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 13:24:14 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 13:24:25 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 13:24:26 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 8621 5520 1 13:23:55 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 8528 5520 0 13:23:55 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 8575 5520 0 13:23:55 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 5520 5517 2 13:23:38 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-28 13:24:15 || haStart:sybase is starting, break! 15-05-28 13:24:15 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 13:24:25 || haStart:The imap is running, break! 15-05-28 13:24:25 || haStart:Success to start the nms! ============================================== 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/28 13:24:30 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/28 13:24:30 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/28 13:24:30 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 13:24:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 13:25:00 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 13:25:00 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/28 13:25:12 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 87% 2015/05/28 13:28:13 VCS INFO V-16-1-50086 CPU usage on MILWB02S is 68% 2015/05/28 14:46:22 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MILWB02S from localhost 2015/05/28 14:46:22 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/05/28 14:46:22 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:47:24 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/28 14:47:24 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopping... TrapReceiver........................stopping... UniteUitlDM...... ============================================== 2015/05/28 14:47:26 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 14:47:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:47:47 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/28 14:47:47 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/28 14:47:47 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/28 14:47:47 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 14:47:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:47:48 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/28 14:47:49 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/28 14:47:49 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/28 14:47:49 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 14:47:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:47:50 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 14:47:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:47:52 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/28 14:47:54 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 14:47:54 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/28 14:48:22 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/28 14:48:22 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/28 14:48:22 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 14:48:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:48:23 VCS INFO V-16-20010-72 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/28 14:48:25 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 14:48:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:48:26 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 14:48:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:48:27 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/28 14:48:27 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/28 14:48:27 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/28 14:48:27 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/28 14:48:27 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/28 14:48:27 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/28 14:48:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/28 14:48:48 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/28066/psinfo 2015/05/28 14:48:48 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 14:48:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:49:00 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/2616/psinfo 2015/05/28 14:49:00 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 14:49:00 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:49:11 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 14:49:12 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 29994 28066 0 14:48:42 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 94 28066 1 14:48:42 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 47 28066 0 14:48:42 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 28066 28065 2 14:48:28 ? 0:18 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-28 14:49:00 || haStart:sybase is starting, break! 15-05-28 14:49:01 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 14:49:11 || haStart:The imap is running, break! 15-05-28 14:49:11 || haStart:Success to start the nms! ============================================== 2015/05/28 14:49:23 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 14:49:23 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/28 14:49:43 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 99% 2015/05/28 14:53:50 VCS INFO V-16-1-50135 User root fired command: hares -offline NMSServer MILWB02S from localhost 2015/05/28 14:53:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:54:52 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/28 14:54:52 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopped PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping... U ============================================== 2015/05/28 14:54:54 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 14:55:49 VCS INFO V-16-1-50135 User root fired command: hares -online NMSServer MILWB02S from localhost 2015/05/28 14:55:49 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 14:55:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 14:56:00 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 14:56:00 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 29994 28066 0 14:48:42 ? 1:32 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 94 28066 0 14:48:42 ? 1:22 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 47 28066 0 14:48:42 ? 0:49 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 28066 28065 0 14:48:28 ? 2:04 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-28 14:55:49 || haStart:sybase is starting, break! 15-05-28 14:55:49 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 14:55:59 || haStart:The imap is running, break! 15-05-28 14:56:00 || haStart:Success to start the nms! ============================================== 2015/05/28 14:56:12 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 14:56:12 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/28 14:56:42 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 99% 2015/05/28 15:27:32 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 15:27:32 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 15:27:33 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/05/28 15:27:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:28:33 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 15:28:33 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 15:29:33 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 15:29:33 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 15:30:33 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 15:30:33 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 15:31:33 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 15:31:33 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 15:32:33 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 15:32:33 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 15:32:34 VCS INFO V-16-2-13003 (MILWB02S) Resource(NMSServer): Output of the timed out operation (offline) Oracle Corporation SunOS 5.10 Generic Patch January 2005 2015/05/28 15:32:34 VCS WARNING V-16-2-13011 (MILWB02S) Resource(NMSServer): offline procedure did not complete within the expected time. 2015/05/28 15:32:34 VCS ERROR V-16-2-13063 (MILWB02S) Agent is calling clean for resource(NMSServer) because offline did not complete within the expected time. 2015/05/28 15:32:40 VCS INFO V-16-2-13068 (MILWB02S) Resource(NMSServer) - clean completed successfully. 2015/05/28 15:32:42 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 15:32:42 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:33:33 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 15:33:33 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 15:33:43 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/28 15:33:44 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/28 15:33:44 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/28 15:33:44 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 15:33:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:33:44 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/28 15:33:45 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/28 15:33:45 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/28 15:33:45 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 15:33:45 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:33:46 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 15:33:46 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:33:48 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/28 15:33:51 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 15:33:51 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/05/28 15:33:51 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/28 15:33:51 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/05/28 15:33:51 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/05/28 15:33:51 VCS NOTICE V-16-1-10235 Restart is set for group AppService. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2015/05/28 15:33:51 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/05/28 15:33:51 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/05/28 15:33:51 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/05/28 15:33:51 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/05/28 15:33:51 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/05/28 15:33:51 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/28 15:33:51 VCS NOTICE V-16-1-10231 Clearing start attributes for resources of group AppService on node MILWB02S 2015/05/28 15:33:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 15:33:51 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/28 15:33:51 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/28 15:33:51 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/28 15:33:51 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 15:33:51 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/28 15:33:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 15:34:02 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:34:04 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:34:34 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB02S (Not initiated by VCS) 2015/05/28 15:34:38 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:34:49 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:35:30 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:35:30 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:43:21 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/05/28 15:43:21 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/05/28 15:44:27 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:44:28 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:44:31 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:44:32 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:44:37 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:44:37 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/28 15:44:37 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/05/28 15:44:37 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/28 15:44:37 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/28 15:44:37 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 15:44:37 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/28 15:44:37 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 15:44:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:44:37 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/28 15:44:46 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/28 15:44:48 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 15:44:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:44:50 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 15:44:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:44:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/28 15:44:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/28 15:44:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/28 15:44:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/28 15:44:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/28 15:44:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/28 15:45:11 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/28 15:45:12 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/19581/psinfo 2015/05/28 15:45:12 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 15:45:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:45:23 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/24130/psinfo 2015/05/28 15:45:23 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 15:45:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 15:45:34 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 15:45:34 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 19581 19579 2 15:44:51 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 21710 19581 1 15:45:06 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 21661 19581 0 15:45:05 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 21613 19581 0 15:45:05 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-28 15:45:23 || haStart:sybase is starting, break! 15-05-28 15:45:23 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 15:45:34 || haStart:The imap is running, break! 15-05-28 15:45:34 || haStart:Success to start the nms! ============================================== 2015/05/28 15:45:46 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/28 15:45:46 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/28 15:46:12 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 100% 2015/05/28 16:04:12 VCS INFO V-16-1-50086 CPU usage on MILWB02S is 60% 2015/05/28 16:04:43 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 97% 2015/05/28 16:14:35 VCS ERROR V-16-2-13027 (MILWB02S) Resource(NMSServer) - monitor procedure did not complete within the expected time. 2015/05/28 16:14:36 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:14:36 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:14:37 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/05/28 16:14:37 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 16:15:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:15:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:16:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:16:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:17:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:17:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:18:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:18:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:19:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:19:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:19:38 VCS INFO V-16-2-13003 (MILWB02S) Resource(NMSServer): Output of the timed out operation (offline) Oracle Corporation SunOS 5.10 Generic Patch January 2005 2015/05/28 16:19:38 VCS WARNING V-16-2-13011 (MILWB02S) Resource(NMSServer): offline procedure did not complete within the expected time. 2015/05/28 16:19:38 VCS ERROR V-16-2-13063 (MILWB02S) Agent is calling clean for resource(NMSServer) because offline did not complete within the expected time. 2015/05/28 16:19:44 VCS INFO V-16-2-13068 (MILWB02S) Resource(NMSServer) - clean completed successfully. 2015/05/28 16:19:47 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 16:19:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 16:20:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:20:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:21:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:21:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:22:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:22:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:23:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:23:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:24:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:24:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:24:48 VCS INFO V-16-2-13003 (MILWB02S) Resource(BackupServer): Output of the timed out operation (offline) Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. 2015/05/28 16:24:48 VCS WARNING V-16-2-13011 (MILWB02S) Resource(BackupServer): offline procedure did not complete within the expected time. 2015/05/28 16:24:48 VCS ERROR V-16-2-13063 (MILWB02S) Agent is calling clean for resource(BackupServer) because offline did not complete within the expected time. 2015/05/28 16:24:48 VCS ERROR V-16-20018-20 (MILWB02S) SybaseBk:BackupServer:clean:kill -15 of 24130 2015/05/28 16:24:49 VCS INFO V-16-2-13068 (MILWB02S) Resource(BackupServer) - clean completed successfully. 2015/05/28 16:24:49 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/28 16:24:49 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 16:24:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 16:25:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:25:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:26:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:26:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:27:09 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/28 16:27:10 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/28 16:27:10 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/28 16:27:10 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 16:27:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 16:27:11 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 16:27:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/28 16:27:13 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/28 16:27:16 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/28 16:27:16 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/05/28 16:27:16 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/28 16:27:16 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/05/28 16:27:16 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/05/28 16:27:16 VCS NOTICE V-16-1-10235 Restart is set for group AppService. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2015/05/28 16:27:16 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/05/28 16:27:16 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/05/28 16:27:16 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/05/28 16:27:16 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/05/28 16:27:16 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/05/28 16:27:16 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/28 16:27:16 VCS NOTICE V-16-1-10231 Clearing start attributes for resources of group AppService on node MILWB02S 2015/05/28 16:27:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 16:27:16 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/28 16:27:16 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/28 16:27:16 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/28 16:27:16 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 16:27:16 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/28 16:27:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 16:27:26 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 16:27:28 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 16:27:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:27:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:27:57 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 16:28:08 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 16:28:34 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 16:28:34 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/28 16:28:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:28:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:29:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:29:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:30:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:30:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:31:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:31:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:32:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:32:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:33:38 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:33:38 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:34:38 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:34:38 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:35:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:35:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:36:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:36:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:37:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:37:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:38:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:38:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:39:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:39:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:40:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:40:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:41:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:41:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:42:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:42:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:43:37 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/05/28 16:43:37 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/05/28 16:44:38 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB02S (Not initiated by VCS) 2015/05/28 17:17:49 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 21:17:49 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 01:17:50 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 05:17:51 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 07:20:01 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/05/29 07:20:01 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/05/29 07:20:01 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/05/29 07:21:05 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 07:21:06 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 07:21:07 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 07:21:09 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 07:21:13 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 07:21:13 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 07:21:13 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/05/29 07:21:13 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 07:21:13 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 07:21:13 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 07:21:13 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/29 07:21:13 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 07:21:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 07:21:14 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 07:21:38 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/29 07:21:40 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 07:21:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 07:21:41 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 07:21:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 07:21:42 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 07:21:42 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 07:21:42 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 07:21:42 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 07:21:42 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/29 07:21:42 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 07:22:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 07:22:03 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/10436/psinfo 2015/05/29 07:22:03 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 07:22:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 07:22:15 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/14774/psinfo 2015/05/29 07:22:15 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 07:22:15 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 07:22:26 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 07:22:26 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 12268 10436 0 07:21:57 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 12315 10436 1 07:21:57 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 10436 10434 2 07:21:42 ? 0:17 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 12220 10436 0 07:21:56 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-29 07:22:16 || haStart:sybase is starting, break! 15-05-29 07:22:16 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 07:22:26 || haStart:The imap is running, break! 15-05-29 07:22:26 || haStart:Success to start the nms! ============================================== 2015/05/29 07:22:39 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 07:22:39 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/29 07:23:12 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 99% 2015/05/29 09:15:34 VCS ERROR V-16-3-18313 (MILWB02S) TCP/IP connection via connector from cluster MILWB02SCluster to cluster MILWB03SCluster is hung; intentionally disconnecting. Auto-reconnect will occur however you may wish to examine the wac resource on system 10.66.28.54 2015/05/29 09:15:34 VCS ERROR V-16-3-18311 (MILWB02S) MILWB02SCluster lost connection to cluster MILWB03SCluster 2015/05/29 09:17:52 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 09:22:29 VCS ERROR V-16-3-18211 (MILWB02S) Cluster MILWB02SCluster lost heartbeat Icmp to cluster MILWB03SCluster 2015/05/29 09:22:29 VCS NOTICE V-16-3-18213 (MILWB02S) Cluster MILWB02SCluster lost all heartbeats to cluster MILWB03SCluster: effecting inquiry 2015/05/29 09:22:29 VCS ERROR V-16-3-18314 (MILWB02S) Cluster MILWB02SCluster lost all heartbeats to cluster MILWB03SCluster: no inquiry peers 2015/05/29 09:22:29 VCS CRITICAL V-16-1-50513 Remote cluster 'MILWB03SCluster' has faulted 2015/05/29 09:22:29 VCS ERROR V-16-5-51071 ALERT[CFAULT-MILWB03SCluster-MILWB03SCluster] Remote cluster MILWB03SCluster has faulted. Administrative action is required. 2015/05/29 09:22:29 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/29 09:28:42 VCS INFO V-16-1-50086 CPU usage on MILWB02S is 61% 2015/05/29 09:29:42 VCS INFO V-16-1-50086 CPU usage on MILWB02S is 60% 2015/05/29 09:32:42 VCS INFO V-16-1-50086 CPU usage on MILWB02S is 61% 2015/05/29 09:34:09 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/29 09:34:09 VCS WARNING V-16-3-18325 (MILWB02S) Cluster (MILWB03SCluster) refused connection. This is because all heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are currently down. 2015/05/29 09:34:33 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/29 09:34:33 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:34:33 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50840 ALERT[CFAULT-MILWB03SCluster-MILWB03SCluster] was deleted automatically by the engine because alert no longer exists on cluster MILWB03SCluster 2015/05/29 09:34:33 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/05/29 09:34:33 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/29 09:35:54 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:35:55 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:35:58 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:35:59 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:36:04 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 09:36:04 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/05/29 09:36:04 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:09:17 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:09:17 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:09:17 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:09:18 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:09:18 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:09:19 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:09:33 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:09:35 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/29 10:09:53 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/29 10:10:09 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 10:10:31 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:10:31 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:10:31 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:10:32 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:10:32 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:10:33 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:11:37 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/05/29 10:11:37 VCS WARNING V-16-20010-301 (MILWB02S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/05/29 10:11:37 VCS ERROR V-16-20010-302 (MILWB02S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/05/29 10:11:38 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/05/29 10:11:38 VCS ERROR V-16-2-13069 (MILWB02S) Resource(datarvg) - clean failed. 2015/05/29 10:12:11 VCS NOTICE V-16-1-50517 Resource NMSServer has been deleted from cluster MILWB03SCluster 2015/05/29 10:12:11 VCS NOTICE V-16-1-50517 Resource DataFilesystem has been deleted from cluster MILWB03SCluster 2015/05/29 10:12:11 VCS NOTICE V-16-1-50517 Resource appNIC has been deleted from cluster MILWB03SCluster 2015/05/29 10:12:11 VCS NOTICE V-16-1-50517 Resource RVGPrimary has been deleted from cluster MILWB03SCluster 2015/05/29 10:12:11 VCS NOTICE V-16-1-50517 Resource DatabaseServer has been deleted from cluster MILWB03SCluster 2015/05/29 10:12:11 VCS NOTICE V-16-1-50517 Resource BackupServer has been deleted from cluster MILWB03SCluster 2015/05/29 10:12:11 VCS NOTICE V-16-1-50518 Group AppService has been deleted from cluster MILWB03SCluster 2015/05/29 10:12:11 VCS INFO V-16-1-50957 Updating ClusterList for group AppService in local cluster configuration to reflect group deletion on cluster MILWB03SCluster 2015/05/29 10:12:11 VCS INFO V-16-1-50958 Local cluster dumping configuration to save ClusterList modification for group AppService 2015/05/29 10:12:12 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:12:13 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/29 10:12:14 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 10:12:17 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:12:17 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB02SCluster and remote cluster MILWB03SCluster 2015/05/29 10:12:17 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:12:17 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:12:17 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:12:17 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:12:17 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:12:17 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 10:12:17 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB02SCluster and remote cluster MILWB03SCluster 2015/05/29 10:12:18 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:12:20 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB03SCluster 0 MILWB02SCluster 1 from localhost 2015/05/29 10:12:20 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 10:12:21 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:12:22 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/29 10:12:41 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/29 10:12:57 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 11:16:41 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 11:16:41 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(DatabaseServer) because the resource became OFFLINE unexpectedly, on its own. 2015/05/29 11:16:41 VCS INFO V-16-20018-51 (MILWB02S) Sybase:DatabaseServer:clean:Could not open directory [/var/tmp/sybase_shm/DBSVR] for read [No such file or directory] 2015/05/29 11:16:41 VCS NOTICE V-16-20018-78 (MILWB02S) Sybase:DatabaseServer:clean:Cannot remove Shared Memory. 2015/05/29 11:16:42 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (clean) ============================================== Password: CT-LIBRARY error: ct_connect(): network packet layer: internal net library error: Net-Lib protocol driver call to connect two endpoints failed ============================================== 2015/05/29 11:16:42 VCS INFO V-16-2-13068 (MILWB02S) Resource(DatabaseServer) - clean completed successfully. 2015/05/29 11:16:42 VCS INFO V-16-1-10307 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/29 11:16:42 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:18:44 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 11:18:45 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopping... InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopped Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopped Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... Unite ============================================== 2015/05/29 11:18:47 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:18:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:18:47 VCS ERROR V-16-20018-25 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back cannot be stopped by isql - shutdown 2015/05/29 11:18:47 VCS ERROR V-16-20018-17 (MILWB02S) SybaseBk:BackupServer:offline:clean scripts should do the process kill sequence 2015/05/29 11:18:48 VCS ERROR V-16-2-13064 (MILWB02S) Agent is calling clean for resource(BackupServer) because the resource is up even after offline completed. 2015/05/29 11:18:49 VCS ERROR V-16-20018-20 (MILWB02S) SybaseBk:BackupServer:clean:kill -15 of 14774 2015/05/29 11:19:09 VCS INFO V-16-2-13068 (MILWB02S) Resource(BackupServer) - clean completed successfully. 2015/05/29 11:19:09 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 11:19:09 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:19:09 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:19:11 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:19:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:19:13 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 11:19:15 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:19:15 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/05/29 11:19:15 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/29 11:19:15 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/05/29 11:19:15 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/05/29 11:19:15 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/05/29 11:19:15 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/05/29 11:19:15 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/05/29 11:19:15 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/05/29 11:19:15 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/05/29 11:19:15 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 11:19:15 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:19:15 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/29 11:19:15 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/29 11:19:15 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 11:19:15 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:19:15 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 11:19:15 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:19:42 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:19:43 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:20:24 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:20:35 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:20:58 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:20:58 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:28:27 VCS INFO V-16-1-50135 User root fired command: hares -clear DatabaseServer MILWB02S from localhost 2015/05/29 11:28:27 VCS INFO V-16-1-10307 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/29 11:31:21 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/05/29 11:31:21 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/05/29 11:31:21 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/05/29 11:32:25 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:32:27 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:32:30 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:32:31 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:32:36 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:32:36 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:32:36 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/05/29 11:32:36 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 11:32:36 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 11:32:36 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:32:36 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/29 11:32:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:32:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:32:36 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 11:32:43 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/29 11:32:45 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:32:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:32:46 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:32:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:32:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 11:32:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 11:32:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 11:32:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 11:32:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/29 11:32:48 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 11:33:13 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 11:33:14 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/18951/psinfo 2015/05/29 11:33:14 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:33:14 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:33:25 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/24804/psinfo 2015/05/29 11:33:25 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:33:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:33:36 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 11:33:37 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 18951 18948 2 11:32:48 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 21909 18951 1 11:33:06 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8988, 10000000000, 0x1c39 sybase 22006 18951 1 11:33:06 ? 0:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8988, 10000000000, 0x1c39 sybase 21959 18951 0 11:33:06 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8988, 10000000000, 0x1c39 15-05-29 11:33:25 || haStart:sybase is starting, break! 15-05-29 11:33:25 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 11:33:36 || haStart:The imap is running, break! 15-05-29 11:33:36 || haStart:Success to start the nms! ============================================== 2015/05/29 11:33:50 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:33:50 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/29 11:34:12 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 99% 2015/05/29 11:41:38 VCS ERROR V-16-2-13027 (MILWB02S) Resource(NMSServer) - monitor procedure did not complete within the expected time. 2015/05/29 11:42:39 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(NMSServer) because the resource became OFFLINE unexpectedly, on its own. 2015/05/29 11:42:44 VCS INFO V-16-2-13068 (MILWB02S) Resource(NMSServer) - clean completed successfully. 2015/05/29 11:42:46 VCS INFO V-16-1-10307 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/29 11:42:46 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:42:46 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 11:42:47 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 11:42:47 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 11:42:47 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:42:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:42:49 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 11:42:49 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 11:42:50 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 11:42:50 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:42:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:42:51 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:42:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:42:53 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 11:42:55 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 11:42:55 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/05/29 11:42:55 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/29 11:42:55 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/05/29 11:42:55 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/05/29 11:42:55 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/05/29 11:42:55 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/05/29 11:42:55 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/05/29 11:42:55 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/05/29 11:42:55 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/05/29 11:42:55 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 11:42:55 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:42:55 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/29 11:42:55 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/29 11:42:55 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 11:42:55 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:42:55 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 11:42:55 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:43:21 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:43:22 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:43:52 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:44:02 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:44:29 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:44:29 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:51:11 VCS INFO V-16-1-50135 User root fired command: hares -clear NMSServer MILWB02S from localhost 2015/05/29 11:51:11 VCS INFO V-16-1-10307 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/29 11:54:15 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/05/29 11:54:15 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/05/29 11:54:15 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/05/29 11:55:22 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:55:24 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:55:27 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:55:28 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:55:33 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:55:33 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 11:55:33 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/05/29 11:55:33 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 11:55:33 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 11:55:33 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:55:33 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/29 11:55:33 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:55:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:55:33 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 11:55:40 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/29 11:55:42 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:55:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:55:44 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:55:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:55:44 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 11:55:44 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 11:55:44 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 11:55:44 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 11:55:44 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/29 11:55:44 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 11:56:05 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 11:56:06 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/7858/psinfo 2015/05/29 11:56:06 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:56:06 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:56:16 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/12092/psinfo 2015/05/29 11:56:16 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:56:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 11:56:27 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 11:56:27 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9549 7858 0 11:55:59 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 9598 7858 0 11:55:59 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 9647 7858 1 11:55:59 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 7858 7856 2 11:55:44 ? 0:18 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-29 11:56:17 || haStart:sybase is starting, break! 15-05-29 11:56:17 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 11:56:27 || haStart:The imap is running, break! 15-05-29 11:56:27 || haStart:Success to start the nms! ============================================== 2015/05/29 11:56:39 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 11:56:39 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/29 11:57:12 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 99% 2015/05/29 12:45:16 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 12:45:16 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(BackupServer) because the resource became OFFLINE unexpectedly, on its own. 2015/05/29 12:45:17 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (clean) ============================================== Password: Msg 7205, Level 17, State 2: Server 'DBSVR', Line 1: Can't open a connection to site 'SYB_BACKUP'. See the error log file in the ASE boot directory. Msg 7205, Level 18, State 3: Server 'DBSVR', Line 1: Can't open a connection to site 'SYB_BACKUP'. See the error log file in the ASE boot directory. ============================================== 2015/05/29 12:45:17 VCS INFO V-16-2-13068 (MILWB02S) Resource(BackupServer) - clean completed successfully. 2015/05/29 12:45:18 VCS INFO V-16-1-10307 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/29 12:45:18 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 12:46:20 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 12:46:20 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... UniteUitlDM.. ============================================== 2015/05/29 12:46:22 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 12:46:22 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 12:46:22 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 12:46:23 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 12:46:23 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 12:46:23 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 12:46:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 12:46:24 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 12:46:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 12:46:26 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 12:46:29 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 12:46:29 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/05/29 12:46:29 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/29 12:46:29 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/05/29 12:46:29 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/05/29 12:46:29 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/05/29 12:46:29 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/05/29 12:46:29 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/05/29 12:46:29 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/05/29 12:46:29 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/05/29 12:46:29 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 12:46:29 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 12:46:29 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/29 12:46:29 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/05/29 12:46:29 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 12:46:29 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 12:46:29 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 12:46:29 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 12:46:55 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 12:46:57 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 12:47:26 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 12:47:37 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 12:48:01 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 12:48:01 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 12:53:15 VCS INFO V-16-1-50135 User root fired command: hares -clear BackupServer MILWB02S from localhost 2015/05/29 12:53:15 VCS INFO V-16-1-10307 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/05/29 12:59:22 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/05/29 12:59:22 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/05/29 13:00:28 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:00:29 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:00:32 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:00:33 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:00:38 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:00:38 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:00:38 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/05/29 13:00:38 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 13:00:38 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 13:00:38 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 13:00:38 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/29 13:00:38 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 13:00:38 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 13:00:38 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 13:00:46 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/29 13:00:48 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 13:00:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 13:00:50 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 13:00:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 13:00:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 13:00:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 13:00:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 13:00:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 13:00:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/29 13:00:50 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 13:01:12 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 13:01:12 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/3622/psinfo 2015/05/29 13:01:12 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 13:01:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 13:01:23 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/8145/psinfo 2015/05/29 13:01:23 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 13:01:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 13:01:35 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 13:01:36 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 5621 3622 1 13:01:05 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 5527 3622 0 13:01:05 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 5573 3622 0 13:01:05 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 3622 3620 2 13:00:51 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-29 13:01:24 || haStart:sybase is starting, break! 15-05-29 13:01:24 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 13:01:34 || haStart:The imap is running, break! 15-05-29 13:01:35 || haStart:Success to start the nms! ============================================== 2015/05/29 13:01:47 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 13:01:47 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/29 13:02:12 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 100% 2015/05/29 13:17:53 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 13:19:35 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB03SCluster' has exited 2015/05/29 13:19:35 VCS INFO V-16-3-18309 (MILWB02S) Cluster MILWB03SCluster exited 2015/05/29 13:24:52 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/29 13:24:52 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 13:24:52 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/29 13:24:52 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/29 14:30:02 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/29 14:30:02 VCS INFO V-16-1-10196 Cluster logger started 2015/05/29 14:30:02 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/29 14:30:02 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/29 14:30:02 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/29 14:30:02 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/29 14:30:03 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/29 14:30:03 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/29 14:30:08 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/29 14:30:08 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/29 14:30:08 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/29 14:30:09 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/29 14:30:09 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/29 14:30:09 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/29 14:30:09 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/29 14:30:09 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/29 14:30:10 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/29 14:30:10 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Fri May 29 14:30:10 2015 2015/05/29 14:30:11 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/29 14:30:11 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 14:30:11 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 14:30:12 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 14:30:12 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/29 14:30:12 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/29 14:30:12 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/29 14:30:12 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/29 14:30:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/29 14:30:12 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/29 14:30:12 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/29 14:30:12 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/29 14:30:12 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/29 14:30:12 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/29 14:30:12 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/29 14:30:12 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/29 14:30:12 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/05/29 14:30:12 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/29 14:30:12 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/29 14:30:12 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/29 14:30:12 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/29 14:30:13 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/29 14:30:13 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/29 14:30:14 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/29 14:30:14 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 14:30:14 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/29 14:30:14 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/29 14:30:14 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 14:30:14 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/29 14:30:14 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/29 14:30:18 VCS WARNING V-16-3-18325 (MILWB02S) Cluster (MILWB03SCluster) refused connection. This is because all heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are currently down. 2015/05/29 14:30:54 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/29 14:30:54 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 14:30:54 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/29 14:30:54 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/29 14:33:31 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 14:33:31 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/29 14:33:31 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/29 14:33:31 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 14:33:31 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/29 14:33:32 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/29 14:33:32 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/29 14:33:32 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/29 14:33:32 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/29 14:33:32 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/29 14:33:32 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/29 14:33:32 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 14:33:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 14:33:32 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 14:33:32 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 14:33:32 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 14:33:33 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 14:33:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 14:33:34 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 14:33:35 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 14:33:35 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 14:33:35 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 14:33:35 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 14:33:35 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/29 14:33:35 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 14:34:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 14:34:11 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5743/psinfo 2015/05/29 14:34:11 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 14:34:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 14:34:22 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/16653/psinfo 2015/05/29 14:34:22 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 14:34:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 14:34:34 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 14:34:34 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 5743 5741 2 14:33:35 ? 0:29 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 8939 5743 2 14:33:53 ? 0:14 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 8890 5743 0 14:33:53 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 8842 5743 0 14:33:53 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-05-29 14:34:24 || haStart:sybase is starting, break! 15-05-29 14:34:24 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 14:34:34 || haStart:The imap is running, break! 15-05-29 14:34:34 || haStart:Success to start the nms! ============================================== 2015/05/29 14:35:11 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 14:35:11 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/29 14:35:16 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 86% 2015/05/29 14:38:16 VCS NOTICE V-16-1-50086 CPU usage on MILWB02S is 76% 2015/05/29 15:46:00 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/29 15:46:00 VCS INFO V-16-1-10196 Cluster logger started 2015/05/29 15:46:00 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/29 15:46:00 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/29 15:46:00 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/29 15:46:00 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/29 15:46:00 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/29 15:46:01 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/29 15:46:05 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/05/29 15:46:05 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/29 15:46:05 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/29 15:46:06 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/29 15:46:06 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/29 15:46:06 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/05/29 15:46:06 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/29 15:46:06 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/29 15:46:08 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/29 15:46:08 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Fri May 29 15:46:08 2015 2015/05/29 15:46:08 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/29 15:46:09 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 15:46:09 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 15:46:09 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 15:46:09 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/05/29 15:46:09 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/05/29 15:46:09 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/29 15:46:09 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/29 15:46:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/05/29 15:46:10 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/29 15:46:10 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/29 15:46:10 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/29 15:46:10 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/29 15:46:10 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/05/29 15:46:10 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/05/29 15:46:10 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/05/29 15:46:10 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/05/29 15:46:10 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/05/29 15:46:11 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/05/29 15:46:11 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/05/29 15:46:11 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/29 15:46:11 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/05/29 15:46:11 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/05/29 15:46:11 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/05/29 15:46:11 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/05/29 15:46:12 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/05/29 15:46:12 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 15:46:12 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/05/29 15:46:12 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/05/29 15:46:12 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/29 15:46:15 VCS WARNING V-16-3-18325 (MILWB02S) Cluster (MILWB03SCluster) refused connection. This is because all heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are currently down. 2015/05/29 15:46:51 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/05/29 15:46:51 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:46:51 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/05/29 15:46:51 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/05/29 15:46:51 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/05/29 15:49:30 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/05/29 15:49:30 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/05/29 15:49:30 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 15:49:30 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/05/29 15:49:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 15:49:30 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 15:49:30 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 15:49:30 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 15:49:32 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 15:49:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 15:49:32 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 15:49:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 15:49:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 15:49:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 15:49:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 15:49:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/05/29 15:49:34 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 15:50:00 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 15:50:01 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5656/psinfo 2015/05/29 15:50:01 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 15:50:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 15:50:12 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/12657/psinfo 2015/05/29 15:50:12 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 15:50:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 15:50:24 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 15:50:25 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 8885 5656 1 15:49:52 ? 0:07 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 5656 5653 2 15:49:34 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 8786 5656 1 15:49:51 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 8836 5656 0 15:49:52 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 15-05-29 15:50:14 || haStart:sybase is starting, break! 15-05-29 15:50:15 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 15:50:24 || haStart:The imap is running, break! 15-05-29 15:50:24 || haStart:Success to start the nms! ============================================== 2015/05/29 15:50:58 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/05/29 15:50:58 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/05/29 15:50:58 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/05/29 15:51:13 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 92% 2015/05/29 15:54:13 VCS INFO V-16-1-50086 CPU usage on MILWB02S is 60% 2015/05/29 15:56:58 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:56:59 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:57:01 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:57:02 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:57:06 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 15:57:06 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/05/29 15:57:06 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:21:44 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:21:45 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/29 16:22:04 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/29 16:22:20 VCS INFO V-16-1-50135 User root fired command: hagrp -offline -force AppService MILWB02S localclus from localhost 2015/05/29 16:22:20 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/05/29 16:22:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 16:23:22 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 16:23:23 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopped Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopped Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... Unite ============================================== 2015/05/29 16:23:25 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 16:23:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 16:23:25 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 16:23:26 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 16:23:26 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 16:23:26 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 16:23:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 16:23:27 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 16:23:27 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 16:23:27 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 16:23:27 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 16:23:27 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 16:23:29 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 16:23:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/05/29 16:23:31 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 16:23:33 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/05/29 16:23:33 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/05/29 16:23:42 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 16:24:02 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:02 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:02 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:03 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:04 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:04 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:18 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB03SCluster to online group AppService on system MILWB03S 2015/05/29 16:24:18 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB03SCluster to online group AppService on system MILWB03S 2015/05/29 16:24:18 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 16:24:18 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 16:24:18 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 16:24:18 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 16:24:21 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:22 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:51 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:24:55 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:24:56 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/05/29 16:25:02 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:25:14 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/05/29 16:25:15 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/05/29 16:25:15 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/05/29 16:25:15 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/05/29 16:25:16 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/05/29 16:25:16 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/05/29 16:25:25 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:25:25 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:25:30 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 16:25:59 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/05/29 16:25:59 VCS WARNING V-16-20010-301 (MILWB02S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/05/29 16:25:59 VCS ERROR V-16-20010-302 (MILWB02S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/05/29 16:26:00 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/05/29 16:26:00 VCS ERROR V-16-2-13069 (MILWB02S) Resource(datarvg) - clean failed. 2015/05/29 16:26:56 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:26:58 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/29 16:26:58 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 16:27:02 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:27:04 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB03SCluster 0 MILWB02SCluster 1 from localhost 2015/05/29 16:27:04 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 16:27:07 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:07 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:07 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:07 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:07 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:07 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:07 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:28 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:28 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:28 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:29 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:29 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 16:27:33 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/05/29 19:42:46 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 23:42:47 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 03:42:48 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 07:42:49 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 11:42:50 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 15:42:51 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 19:42:52 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 23:42:53 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 03:42:54 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 07:42:55 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 11:42:56 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 15:42:57 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 19:42:58 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 23:42:59 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 03:43:00 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 07:43:01 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 08:39:34 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/06/01 08:39:34 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/06/01 08:39:34 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/06/01 08:40:41 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/01 08:40:42 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/01 08:40:44 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/01 08:40:45 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/01 08:40:49 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/01 08:40:49 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/01 08:40:49 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/06/01 08:40:49 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/01 08:40:49 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/01 08:40:49 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/01 08:40:49 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/01 08:40:49 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/01 08:40:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/01 08:40:50 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/01 08:40:58 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/06/01 08:41:00 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/01 08:41:00 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/01 08:41:01 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/01 08:41:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/01 08:41:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/01 08:41:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/01 08:41:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/01 08:41:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/01 08:41:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/06/01 08:41:02 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/01 08:41:24 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/01 08:41:25 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/18591/psinfo 2015/06/01 08:41:25 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/01 08:41:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/01 08:41:37 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/23551/psinfo 2015/06/01 08:41:37 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/01 08:41:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/01 08:41:48 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/01 08:41:49 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 20720 18591 1 08:41:18 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 20676 18591 0 08:41:17 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 18591 18589 2 08:41:02 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 20628 18591 0 08:41:17 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-06-01 08:41:37 || haStart:sybase is starting, break! 15-06-01 08:41:37 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-01 08:41:47 || haStart:The imap is running, break! 15-06-01 08:41:48 || haStart:Success to start the nms! ============================================== 2015/06/01 08:42:00 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/01 08:42:00 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/06/01 08:42:13 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 82% 2015/06/01 08:42:43 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 98% 2015/06/01 11:43:01 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 15:43:02 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 19:43:03 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 23:43:04 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 03:43:05 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 07:43:06 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 11:43:07 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 15:43:08 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 19:43:09 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 23:43:10 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 03:43:11 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 07:43:12 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 11:06:03 VCS INFO V-16-1-10196 Cluster logger started 2015/06/03 11:06:03 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/03 11:06:03 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/06/03 11:06:03 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/03 11:06:03 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/03 11:06:03 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/03 11:06:03 VCS NOTICE V-16-1-10115 Using GABSIM 2015/06/03 11:06:03 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/06/03 11:06:08 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/03 11:06:08 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/06/03 11:06:08 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/06/03 11:06:09 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/03 11:06:09 VCS INFO V-16-1-10077 Received new cluster membership 2015/06/03 11:06:09 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/06/03 11:06:09 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/06/03 11:06:09 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/06/03 11:06:10 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/06/03 11:06:10 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:10 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed Jun 3 11:06:10 2015 2015/06/03 11:06:11 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/06/03 11:06:11 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/03 11:06:11 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/03 11:06:12 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/03 11:06:12 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/06/03 11:06:12 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/06/03 11:06:12 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/06/03 11:06:12 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/06/03 11:06:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/06/03 11:06:12 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/06/03 11:06:12 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/06/03 11:06:12 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/06/03 11:06:12 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/06/03 11:06:12 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/06/03 11:06:12 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/06/03 11:06:12 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/06/03 11:06:12 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/06/03 11:06:12 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/06/03 11:06:12 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/06/03 11:06:12 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/06/03 11:06:13 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/06/03 11:06:13 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/06/03 11:06:13 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/06/03 11:06:13 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/06/03 11:06:13 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/06/03 11:06:13 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/06/03 11:06:13 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/03 11:06:15 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/03 11:06:15 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/06/03 11:06:15 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/06/03 11:06:19 VCS WARNING V-16-3-18325 (MILWB02S) Cluster (MILWB03SCluster) refused connection. This is because all heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are currently down. 2015/06/03 11:06:54 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/06/03 11:06:54 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:06:54 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/06/03 11:06:54 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/03 11:06:54 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/06/03 11:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/06/03 11:09:32 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/03 11:09:32 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/03 11:09:32 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/03 11:09:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:09:32 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:09:32 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:09:32 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:09:33 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 11:09:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:09:34 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:09:35 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/03 11:09:36 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/03 11:09:36 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/03 11:09:36 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/03 11:09:36 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/06/03 11:09:36 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/03 11:10:03 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/03 11:10:05 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5598/psinfo 2015/06/03 11:10:05 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 11:10:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:10:15 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/13108/psinfo 2015/06/03 11:10:15 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 11:10:15 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:10:26 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/03 11:10:27 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 8762 5598 0 11:09:53 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 8809 5598 1 11:09:53 ? 0:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 8706 5598 0 11:09:53 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 5598 5596 2 11:09:36 ? 0:23 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-06-03 11:10:18 || haStart:sybase is starting, break! 15-06-03 11:10:18 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-03 11:10:26 || haStart:The imap is running, break! 15-06-03 11:10:26 || haStart:Success to start the nms! ============================================== 2015/06/03 11:10:58 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 11:10:58 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/03 11:10:58 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/06/03 11:11:14 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 83% 2015/06/03 11:14:14 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 86% 2015/06/03 11:33:50 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/06/03 11:33:50 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/03 11:33:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:34:52 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/03 11:34:53 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................st ============================================== 2015/06/03 11:34:55 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:34:55 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/03 11:34:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:34:55 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/03 11:34:56 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/03 11:34:56 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/03 11:34:56 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:34:56 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:34:57 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/03 11:34:57 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/03 11:34:58 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/03 11:34:58 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:34:58 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:34:59 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:34:59 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 11:35:01 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/03 11:35:03 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/03 11:35:03 VCS INFO V-16-1-50918 Local cluster MILWB02SCluster and remote cluster MILWB03SCluster have Authority for group AppService. Giving up Authority since group is online on remote cluster and offline in local cluster 2015/06/03 11:35:03 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/03 11:35:03 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/03 11:35:03 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/06/03 11:43:04 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:43:05 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/06/03 11:43:24 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/06/03 11:43:24 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:43:24 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:43:24 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:43:25 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:43:26 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:43:39 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:44:00 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:44:01 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:44:01 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:44:02 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:44:02 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:44:04 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:44:18 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:44:19 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/06/03 11:44:38 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/06/03 11:44:38 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:44:38 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:44:38 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:44:39 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:44:40 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/03 11:44:54 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:45:22 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/06/03 11:45:22 VCS WARNING V-16-20010-301 (MILWB02S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/06/03 11:45:22 VCS ERROR V-16-20010-302 (MILWB02S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/06/03 11:45:23 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/06/03 11:45:23 VCS ERROR V-16-2-13069 (MILWB02S) Resource(datarvg) - clean failed. 2015/06/03 11:46:20 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:46:21 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/06/03 11:46:21 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:46:26 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:46:27 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB03SCluster 0 MILWB02SCluster 1 from localhost 2015/06/03 11:46:28 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:46:31 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:31 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:31 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:31 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:31 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:31 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:31 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:54 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:54 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:54 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:55 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:55 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 11:46:59 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/03 15:02:44 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 16:38:36 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/06/03 16:38:36 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/06/03 16:43:45 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/03 16:43:46 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/03 16:43:47 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/03 16:43:49 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/03 16:43:53 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/03 16:43:53 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/03 16:43:53 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/06/03 16:43:53 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/03 16:43:53 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/03 16:43:53 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/03 16:43:53 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/03 16:43:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/03 16:43:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 16:43:53 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/03 16:44:05 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/06/03 16:44:07 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 16:44:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 16:44:09 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 16:44:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 16:44:09 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/03 16:44:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/03 16:44:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/03 16:44:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/03 16:44:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/06/03 16:44:10 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/03 16:44:33 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/03 16:44:34 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/24069/psinfo 2015/06/03 16:44:34 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 16:44:34 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 16:44:45 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/29153/psinfo 2015/06/03 16:44:45 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 16:44:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/03 16:44:56 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/03 16:44:57 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 26398 24069 0 16:44:26 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 26354 24069 1 16:44:26 ? 0:03 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 24069 24066 2 16:44:10 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 26441 24069 1 16:44:26 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 15-06-03 16:44:45 || haStart:sybase is starting, break! 15-06-03 16:44:45 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-03 16:44:56 || haStart:The imap is running, break! 15-06-03 16:44:56 || haStart:Success to start the nms! ============================================== 2015/06/03 16:45:10 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/03 16:45:10 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/06/03 16:45:44 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 98% 2015/06/03 19:02:45 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 23:02:46 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 03:02:47 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 07:02:48 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 10:41:20 VCS INFO V-16-1-10196 Cluster logger started 2015/06/04 10:41:20 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/04 10:41:20 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/06/04 10:41:20 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/04 10:41:20 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/04 10:41:20 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/04 10:41:21 VCS NOTICE V-16-1-10115 Using GABSIM 2015/06/04 10:41:21 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/06/04 10:41:26 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/04 10:41:26 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/06/04 10:41:26 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/06/04 10:41:27 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/04 10:41:27 VCS INFO V-16-1-10077 Received new cluster membership 2015/06/04 10:41:27 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/06/04 10:41:27 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/06/04 10:41:27 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/06/04 10:41:28 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/06/04 10:41:28 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:28 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Jun 4 10:41:28 2015 2015/06/04 10:41:29 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/06/04 10:41:29 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/04 10:41:30 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/04 10:41:30 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/04 10:41:30 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/06/04 10:41:30 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/06/04 10:41:30 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/06/04 10:41:30 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/06/04 10:41:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/06/04 10:41:30 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/06/04 10:41:30 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/06/04 10:41:30 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/06/04 10:41:31 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/06/04 10:41:31 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/06/04 10:41:31 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/06/04 10:41:31 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/06/04 10:41:31 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/06/04 10:41:31 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/06/04 10:41:31 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/06/04 10:41:31 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/06/04 10:41:31 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/06/04 10:41:31 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/06/04 10:41:31 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/06/04 10:41:32 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (First probe) 2015/06/04 10:41:32 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 10:41:32 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/06/04 10:41:32 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/06/04 10:41:34 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/04 10:41:34 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/06/04 10:41:34 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/06/04 10:41:36 VCS WARNING V-16-3-18325 (MILWB02S) Cluster (MILWB03SCluster) refused connection. This is because all heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are currently down. 2015/06/04 10:41:54 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/06/04 10:41:54 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:41:54 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/06/04 10:41:54 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/04 10:41:54 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S 0 from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB02S from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S 0 from localhost 2015/06/04 10:44:50 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB02S from localhost 2015/06/04 10:44:50 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/04 10:44:50 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 10:44:50 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/04 10:44:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:44:50 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:44:50 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:44:50 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:44:52 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/04 10:44:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:44:52 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:44:54 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/04 10:44:54 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/04 10:44:54 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/04 10:44:54 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/04 10:44:54 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/06/04 10:44:54 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/04 10:45:25 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/04 10:45:26 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5623/psinfo 2015/06/04 10:45:26 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/04 10:45:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:45:38 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/14927/psinfo 2015/06/04 10:45:38 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/04 10:45:38 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:45:49 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/04 10:45:49 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 8942 5623 1 10:45:12 ? 0:12 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 5623 5621 1 10:44:54 ? 0:20 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 8839 5623 0 10:45:12 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 8888 5623 0 10:45:12 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 15-06-04 10:45:41 || haStart:sybase is starting, break! 15-06-04 10:45:41 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-04 10:45:49 || haStart:The imap is running, break! 15-06-04 10:45:49 || haStart:Success to start the nms! ============================================== 2015/06/04 10:46:24 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/04 10:46:24 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/04 10:46:24 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/06/04 10:46:32 VCS WARNING V-16-1-50086 CPU usage on MILWB02S is 80% 2015/06/04 10:49:32 VCS NOTICE V-16-1-50086 CPU usage on MILWB02S is 73% 2015/06/04 10:52:11 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:52:12 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:52:13 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:52:15 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:52:19 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:52:19 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB03SCluster, MILWB02SCluster] 2015/06/04 10:52:19 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:56:41 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 10:56:42 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/06/04 10:57:00 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/06/04 10:57:16 VCS INFO V-16-1-50135 User root fired command: hagrp -offline -force AppService MILWB02S localclus from localhost 2015/06/04 10:57:16 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB02S 2015/06/04 10:57:16 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:58:18 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/04 10:58:19 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... Unite ============================================== 2015/06/04 10:58:21 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:58:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:58:22 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/04 10:58:23 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/04 10:58:23 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/04 10:58:23 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:58:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:58:24 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/04 10:58:25 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/04 10:58:25 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/04 10:58:25 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:58:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:58:26 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:58:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 10:58:28 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/04 10:58:31 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/04 10:58:31 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/06/04 10:58:38 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 10:59:00 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:00 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:00 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:01 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:01 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:01 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:15 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB03SCluster to online group AppService on system MILWB03S 2015/06/04 10:59:15 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB03SCluster to online group AppService on system MILWB03S 2015/06/04 10:59:15 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/04 10:59:15 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 10:59:15 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/04 10:59:15 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 10:59:18 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:19 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:49 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 10:59:53 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 10:59:55 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB02S localclus 0 from localhost 2015/06/04 11:00:00 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:00:13 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB02S localclus 0 from localhost 2015/06/04 11:00:13 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/04 11:00:13 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/04 11:00:13 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/04 11:00:14 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/04 11:00:15 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Previous State = OFFLINE) 2015/06/04 11:00:24 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:00:24 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:00:29 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 11:00:57 VCS ERROR V-16-2-13067 (MILWB02S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/06/04 11:00:57 VCS WARNING V-16-20010-301 (MILWB02S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/06/04 11:00:57 VCS ERROR V-16-20010-302 (MILWB02S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/06/04 11:00:58 VCS INFO V-16-2-13716 (MILWB02S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/06/04 11:00:58 VCS ERROR V-16-2-13069 (MILWB02S) Resource(datarvg) - clean failed. 2015/06/04 11:02:01 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 11:02:03 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/06/04 11:02:03 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 11:02:09 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 11:02:10 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB03SCluster 0 MILWB02SCluster 1 from localhost 2015/06/04 11:02:10 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 11:02:14 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:14 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:14 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:14 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:14 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:14 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:14 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:36 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:36 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:36 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:37 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:38 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:02:41 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:04:26 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/06/04 11:04:26 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/06/04 11:05:31 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:05:33 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:05:37 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:05:38 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:05:43 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:05:43 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:05:43 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/06/04 11:05:43 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/04 11:05:43 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/04 11:05:43 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 11:05:43 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/04 11:05:43 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 11:05:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/04 11:05:44 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/04 11:05:44 VCS ERROR V-16-20010-71 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk is inconsistent; please restore datarvg from a consistent snapshot or backup and promote it to primary, clear its service group, and online group again 2015/06/04 11:07:46 VCS ERROR V-16-2-13066 (MILWB02S) Agent is calling clean for resource(RVGPrimary) because the resource is not up even after online completed. 2015/06/04 11:07:47 VCS INFO V-16-2-13068 (MILWB02S) Resource(RVGPrimary) - clean completed successfully. 2015/06/04 11:07:47 VCS INFO V-16-2-13072 (MILWB02S) Resource(RVGPrimary): Agent is retrying online (attempt number 1 of 1). 2015/06/04 11:07:47 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/04 11:07:47 VCS ERROR V-16-20010-71 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk is inconsistent; please restore datarvg from a consistent snapshot or backup and promote it to primary, clear its service group, and online group again 2015/06/04 11:09:49 VCS ERROR V-16-2-13066 (MILWB02S) Agent is calling clean for resource(RVGPrimary) because the resource is not up even after online completed. 2015/06/04 11:09:50 VCS INFO V-16-2-13068 (MILWB02S) Resource(RVGPrimary) - clean completed successfully. 2015/06/04 11:09:50 VCS INFO V-16-2-13071 (MILWB02S) Resource(RVGPrimary): reached OnlineRetryLimit(1). 2015/06/04 11:09:51 VCS ERROR V-16-1-10303 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/06/04 11:09:51 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/06/04 11:09:51 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/06/04 11:09:51 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/06/04 11:09:51 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/06/04 11:09:51 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/06/04 11:09:51 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/06/04 11:09:51 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/06/04 11:09:51 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/06/04 11:09:51 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/06/04 11:09:51 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/04 11:09:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 11:09:51 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/06/04 11:09:51 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/06/04 11:09:51 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/04 11:09:51 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 11:09:51 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/04 11:09:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 11:09:53 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:09:55 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:10:25 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:10:36 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:11:00 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 11:11:00 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/04 12:04:58 VCS INFO V-16-1-50135 User root fired command: hares -clear RVGPrimary MILWB02S from localhost 2015/06/04 12:04:58 VCS INFO V-16-1-10307 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (Not initiated by VCS) 2015/06/04 14:38:03 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 18:38:04 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 22:38:05 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 02:38:06 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 06:38:07 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 10:38:08 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 14:38:09 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 18:38:10 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 22:38:11 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 02:38:12 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 06:38:13 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 10:38:14 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 14:38:15 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 18:38:16 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 22:38:17 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 02:38:18 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 06:38:19 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 10:38:19 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 14:38:21 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 18:38:22 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 22:38:22 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 02:38:23 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 06:38:24 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 09:57:19 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/06/08 10:16:29 VCS INFO V-16-1-50135 User admin fired command: haconf -makerw from ::ffff:10.66.28.69 2015/06/08 10:16:29 VCS INFO V-16-1-50135 User admin fired command: hagrp -freeze -persistent AppService from ::ffff:10.66.28.69 2015/06/08 10:21:06 VCS INFO V-16-1-50135 User admin fired command: hares -add neNIC NIC AppService from ::ffff:10.66.28.69 2015/06/08 10:21:06 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Critical 1 from ::ffff:10.66.28.69 2015/06/08 10:21:06 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC PingOptimize 1 from ::ffff:10.66.28.69 2015/06/08 10:21:06 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Protocol IPv4 from ::ffff:10.66.28.69 2015/06/08 10:21:06 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC ExclusiveIPZone 0 from ::ffff:10.66.28.69 2015/06/08 10:21:06 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Device bge0 from ::ffff:10.66.28.69 2015/06/08 10:21:07 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC NetworkHosts 10.66.28.19 from ::ffff:10.66.28.69 2015/06/08 10:21:07 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC NetworkType from ::ffff:10.66.28.69 2015/06/08 10:21:07 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Enabled 0 from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:24:24 VCS NOTICE V-16-1-50517 Resource neNIC has been deleted from cluster MILWB03SCluster 2015/06/08 10:26:44 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:26:57 VCS INFO V-16-1-50135 User admin fired command: hagrp -unfreeze -persistent AppService from ::ffff:10.66.28.69 2015/06/08 10:27:37 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_ALL_FORCE from localhost 2015/06/08 10:27:37 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from RUNNING to LEAVING 2015/06/08 10:27:37 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/06/08 10:27:37 VCS NOTICE V-16-1-50402 VCS engine on system MILWB02S has lost connection with wide area connector 2015/06/08 10:27:37 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/06/08 10:27:38 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (VCS initiated) 2015/06/08 10:27:38 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB02S 2015/06/08 10:27:38 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LEAVING to EXITING 2015/06/08 10:27:38 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from EXITING to EXITED 2015/06/08 10:27:41 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:27:41 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:27:41 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:27:41 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:27:41 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:27:41 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:27:41 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/08 10:27:41 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/06/08 10:27:41 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/06/08 10:27:41 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:27:41 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:27:41 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:27:52 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:27:52 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:27:52 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:27:52 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:27:52 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:27:52 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:27:52 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:27:52 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/08 10:27:52 VCS WARNING V-16-1-11030 HAD not ready to receive this command. Message was: 0x915 2015/06/08 10:27:52 VCS WARNING V-16-1-11030 HAD not ready to receive this command. Message was: 0x311 2015/06/08 10:27:52 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:27:52 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:27:52 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:28:17 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:28:17 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:28:17 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:28:17 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:28:17 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:28:17 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:28:17 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:28:17 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/08 10:28:17 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:28:17 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:28:17 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:28:52 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:28:52 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:28:52 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:28:52 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:28:52 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:28:52 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:28:52 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:28:52 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/08 10:28:52 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:28:52 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:28:52 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:29:37 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:29:37 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:29:37 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:29:37 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:29:37 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:29:37 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:29:37 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:29:37 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/08 10:29:37 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:29:37 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:29:37 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:30:32 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:30:32 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:30:32 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:30:32 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:30:32 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:30:32 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:30:32 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:30:32 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/08 10:30:32 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:30:32 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:30:32 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:40:57 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:40:57 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/06/08 10:40:57 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:40:57 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:40:57 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:40:57 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:40:57 VCS NOTICE V-16-1-10115 Using GABSIM 2015/06/08 10:40:57 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/06/08 10:41:01 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB02S 2015/06/08 10:41:01 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/06/08 10:41:01 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/06/08 10:41:01 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:41:01 VCS INFO V-16-1-10077 Received new cluster membership 2015/06/08 10:41:01 VCS NOTICE V-16-1-10112 System (MILWB02S) - Membership: 0x1, DDNA: 0x0 2015/06/08 10:41:01 VCS NOTICE V-16-1-10086 System MILWB02S (Node '0') is in Regular Membership - Membership: 0x1 2015/06/08 10:41:01 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/06/08 10:41:01 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 10:41:02 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/06/08 10:41:02 VCS NOTICE V-16-1-10322 System MILWB02S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/06/08 10:41:02 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Mon Jun 8 10:41:02 2015 2015/06/08 10:41:02 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Mon Jun 8 10:41:02 2015 2015/06/08 10:41:02 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Mon Jun 8 10:41:02 2015 2015/06/08 10:41:02 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Mon Jun 8 10:41:02 2015 2015/06/08 10:41:02 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Mon Jun 8 10:41:02 2015 2015/06/08 10:41:03 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Mon Jun 8 10:41:02 2015 2015/06/08 10:41:03 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Mon Jun 8 10:41:03 2015 2015/06/08 10:41:03 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Mon Jun 8 10:41:03 2015 2015/06/08 10:41:03 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/06/08 10:41:03 VCS INFO V-16-6-15023 (MILWB02S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs EDITOR=vi HOME=/ HWENGRROOT=/opt/HWENGR HWICMRROOT=/opt/HWICMR HWNMSJRE=/opt/HWNMSJRE IMAP_JAVA_HOME=/opt/HWNMSJRE/jre_sol LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib:/usr/openwin/lib LOGNAME=root MAIL=/var/mail/root MANPATH=/opt/VRTSob/man:/opt/VRTS/man:/opt/VRTSvxvm/man::/usr/share/man:/usr/sunvts/man:/opt/SUNWexplo/man OPENWINHOME=/usr/openwin PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise/sbin:/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin PS1=root@MILWB02S # PS2=root@MILWB02S > PWD=/var/VRTSvcs/diag/had SHELL=/bin/bash SHLVL=1 SSH_CLIENT=10.66.21.87 49251 22 SSH_CONNECTION=10.66.21.87 49251 10.66.28.69 22 SSH_TTY=/dev/pts/2 TERM=xterm TZ=Europe/Rome USER=root VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger _=/opt/VRTS/bin/hastart 2015/06/08 10:41:03 VCS INFO V-16-6-15002 (MILWB02S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB02S 1 successfully 2015/06/08 10:41:03 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/08 10:41:03 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/08 10:41:03 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/08 10:41:03 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB02S (First probe) 2015/06/08 10:41:03 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB02S 2015/06/08 10:41:03 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/06/08 10:41:03 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/06/08 10:41:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB02S 2015/06/08 10:41:03 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/06/08 10:41:03 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/06/08 10:41:03 VCS INFO V-16-1-50403 VCS engine on system MILWB02S has established connection with wide area connector 2015/06/08 10:41:03 VCS NOTICE V-16-3-18511 (MILWB02S) all heartbeats to cluster (MILWB03SCluster) are down. will not attempt to connect to cluster (MILWB03SCluster) until at least one heartbeat to this cluster is alive 2015/06/08 10:41:03 VCS INFO V-16-10001-88 (MILWB02S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/06/08 10:41:03 VCS INFO V-16-3-18210 (MILWB02S) Cluster MILWB02SCluster heartbeat Icmp to cluster MILWB03SCluster is alive 2015/06/08 10:41:04 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB02S (First probe) 2015/06/08 10:41:04 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/06/08 10:41:04 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB02S 2015/06/08 10:41:04 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB02S 2015/06/08 10:41:04 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/08 10:41:04 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB02S (VCS initiated) 2015/06/08 10:41:04 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB02S 2015/06/08 10:41:05 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB02S 2015/06/08 10:41:05 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB02S as the system is not a part of AutoStartList attribute of the group. 2015/06/08 10:41:05 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (First probe) 2015/06/08 10:41:05 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB02S 2015/06/08 10:41:05 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/06/08 10:41:09 VCS INFO V-16-3-18306 (MILWB02S) Initiating connection to cluster MILWB03SCluster at 10.66.28.54 2015/06/08 10:41:09 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50991 System MILWB03S is running in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/08 10:41:09 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB03SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB03SCluster 2015/06/08 10:42:01 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/06/08 14:41:02 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 18:41:03 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 22:41:04 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 02:41:05 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 06:41:06 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 08:24:05 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/09 08:24:20 VCS INFO V-16-1-50135 User root fired command: hares -add neNIC NIC AppService from localhost 2015/06/09 08:40:37 VCS INFO V-16-1-50135 User root fired command: hares -modify neNIC Device bge0 from localhost 2015/06/09 08:40:48 VCS INFO V-16-1-50135 User root fired command: hares -modify neNIC NetworkHosts 10.66.28.19 from localhost 2015/06/09 08:41:03 VCS INFO V-16-1-50135 User root fired command: hares -modify neNIC Enabled 1 from localhost 2015/06/09 08:41:38 VCS INFO V-16-1-50135 User root fired command: hares -online neNIC MILWB02S from localhost 2015/06/09 08:45:18 VCS INFO V-16-1-50135 User root fired command: hares -link RVGPrimary neNIC 0 0 from localhost 2015/06/09 08:47:27 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/09 09:24:19 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:24:57 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:38:40 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/06/09 09:38:40 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/06/09 09:39:45 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:39:47 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:39:48 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:39:49 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:39:54 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:39:54 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 09:39:54 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/06/09 09:39:54 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/09 09:39:54 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/09 09:39:54 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 09:39:54 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/09 09:39:54 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 09:39:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 09:39:54 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/09 09:40:04 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/06/09 09:40:06 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 09:40:06 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 09:40:07 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 09:40:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 09:40:08 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/09 09:40:08 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/09 09:40:08 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/09 09:40:08 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/09 09:40:08 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/06/09 09:40:08 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/09 09:40:30 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/09 09:40:31 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/19067/psinfo 2015/06/09 09:40:31 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 09:40:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 09:40:41 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/23702/psinfo 2015/06/09 09:40:41 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 09:40:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 09:40:52 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/09 09:40:53 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 20818 19067 0 09:40:23 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 sybase 19067 19066 2 09:40:08 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 20863 19067 1 09:40:23 ? 0:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 20769 19067 0 09:40:23 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 15-06-09 09:40:42 || haStart:sybase is starting, break! 15-06-09 09:40:42 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-09 09:40:52 || haStart:The imap is running, break! 15-06-09 09:40:52 || haStart:Success to start the nms! ============================================== 2015/06/09 09:41:07 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 09:41:07 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/06/09 09:41:38 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 100% 2015/06/09 09:44:38 VCS NOTICE V-16-1-50086 CPU usage on MILWB02S is 73% 2015/06/09 10:40:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:40:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:40:06 VCS ERROR V-16-1-10303 Resource neNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/06/09 10:40:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 10:41:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:41:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:41:07 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 10:41:08 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/09 10:41:09 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver...................... ============================================== 2015/06/09 10:41:11 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 10:41:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 10:41:31 VCS NOTICE V-16-20018-26 (MILWB02S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/09 10:41:32 VCS INFO V-16-2-13716 (MILWB02S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/09 10:41:32 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/09 10:41:32 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 10:41:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 10:41:33 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/09 10:41:33 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/09 10:41:33 VCS CRITICAL V-16-20018-4 (MILWB02S) Sybase:DatabaseServer:monitor:Process dataserver restarted 2015/06/09 10:41:33 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 10:41:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 10:41:34 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 10:41:34 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 10:41:37 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/09 10:41:39 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 10:41:39 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/06/09 10:41:39 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/06/09 10:41:39 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/06/09 10:41:39 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/06/09 10:41:39 VCS NOTICE V-16-1-10235 Restart is set for group AppService. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2015/06/09 10:41:39 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/06/09 10:41:39 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/06/09 10:41:39 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/06/09 10:41:39 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/06/09 10:41:39 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/06/09 10:41:39 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/09 10:41:39 VCS NOTICE V-16-1-10231 Clearing start attributes for resources of group AppService on node MILWB02S 2015/06/09 10:41:39 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 10:41:39 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/06/09 10:41:39 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/06/09 10:41:39 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/09 10:41:39 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 10:41:39 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/09 10:41:39 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 10:41:49 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 10:41:51 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 10:42:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:42:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:42:21 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 10:42:32 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 10:43:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:43:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:43:06 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 10:43:06 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 10:44:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:44:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:45:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:45:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:46:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:46:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:47:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:47:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:48:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:48:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:49:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:49:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:50:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:50:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:51:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:51:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:52:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:52:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:53:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:53:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:54:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:54:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:55:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:55:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:56:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:56:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:57:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:57:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:58:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:58:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 10:59:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 10:59:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:00:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:00:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:01:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:01:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:02:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:02:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:03:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:03:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:04:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:04:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:05:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:05:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:06:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:06:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:07:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:07:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:08:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:08:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:09:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:09:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:10:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:10:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:11:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:11:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:12:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:12:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:13:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:13:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:14:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:14:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:15:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:15:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:16:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:16:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:17:06 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:neNIC:monitor:Invalid Device Name 2015/06/09 11:17:06 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:neNIC:monitor:Device name (bge0) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 11:18:07 VCS INFO V-16-1-10299 Resource neNIC (Owner: Unspecified, Group: AppService) is online on MILWB02S (Not initiated by VCS) 2015/06/09 12:03:42 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/06/09 12:03:42 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/06/09 12:04:47 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 12:04:48 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 12:04:49 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 12:04:51 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 12:04:55 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 12:04:55 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/09 12:04:55 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/06/09 12:04:55 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/09 12:04:55 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/09 12:04:55 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 12:04:55 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/09 12:04:55 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 12:04:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 12:04:55 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/09 12:05:02 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/06/09 12:05:04 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 12:05:04 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 12:05:06 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 12:05:06 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 12:05:07 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/09 12:05:07 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/09 12:05:07 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/09 12:05:07 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/09 12:05:07 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/06/09 12:05:07 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/09 12:05:28 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/09 12:05:28 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/7072/psinfo 2015/06/09 12:05:28 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 12:05:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 12:05:39 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/11317/psinfo 2015/06/09 12:05:39 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 12:05:39 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 12:05:51 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/09 12:05:51 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 7072 7070 2 12:05:07 ? 0:18 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 8864 7072 1 12:05:22 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 8775 7072 0 12:05:21 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 8821 7072 0 12:05:22 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 15-06-09 12:05:40 || haStart:sybase is starting, break! 15-06-09 12:05:40 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-09 12:05:50 || haStart:The imap is running, break! 15-06-09 12:05:51 || haStart:Success to start the nms! ============================================== 2015/06/09 12:06:10 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/09 12:06:10 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/06/09 12:06:38 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 99% 2015/06/09 14:07:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:07:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:07:04 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB02S 2015/06/09 14:07:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 14:08:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:08:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:09:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:09:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:10:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:10:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:11:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:11:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:12:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:12:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:12:06 VCS INFO V-16-2-13003 (MILWB02S) Resource(NMSServer): Output of the timed out operation (offline) Oracle Corporation SunOS 5.10 Generic Patch January 2005 2015/06/09 14:12:06 VCS WARNING V-16-2-13011 (MILWB02S) Resource(NMSServer): offline procedure did not complete within the expected time. 2015/06/09 14:12:06 VCS ERROR V-16-2-13063 (MILWB02S) Agent is calling clean for resource(NMSServer) because offline did not complete within the expected time. 2015/06/09 14:12:12 VCS INFO V-16-2-13068 (MILWB02S) Resource(NMSServer) - clean completed successfully. 2015/06/09 14:12:14 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 14:12:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 14:13:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:13:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:14:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:14:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:15:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:15:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:16:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:16:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:17:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:17:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:17:16 VCS INFO V-16-2-13003 (MILWB02S) Resource(BackupServer): Output of the timed out operation (offline) Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. 2015/06/09 14:17:16 VCS WARNING V-16-2-13011 (MILWB02S) Resource(BackupServer): offline procedure did not complete within the expected time. 2015/06/09 14:17:16 VCS ERROR V-16-2-13063 (MILWB02S) Agent is calling clean for resource(BackupServer) because offline did not complete within the expected time. 2015/06/09 14:17:16 VCS ERROR V-16-20018-20 (MILWB02S) SybaseBk:BackupServer:clean:kill -15 of 11317 2015/06/09 14:17:17 VCS INFO V-16-2-13068 (MILWB02S) Resource(BackupServer) - clean completed successfully. 2015/06/09 14:17:17 VCS WARNING V-16-20018-301 (MILWB02S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/09 14:17:17 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 14:17:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 14:18:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:18:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:19:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:19:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:19:37 VCS NOTICE V-16-20018-18 (MILWB02S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/09 14:19:38 VCS INFO V-16-2-13716 (MILWB02S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. SHUTDOWN is waiting for 1 process(es) to complete. Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/09 14:19:38 VCS WARNING V-16-20018-301 (MILWB02S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/09 14:19:38 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 14:19:38 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 14:19:39 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 14:19:39 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/09 14:19:42 VCS INFO V-16-20010-176 (MILWB02S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/09 14:19:44 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB02S (VCS initiated) 2015/06/09 14:19:44 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB02S 2015/06/09 14:19:44 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB02S 2015/06/09 14:19:44 VCS INFO V-16-1-10493 Evaluating MILWB02S as potential target node for group AppService 2015/06/09 14:19:44 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB02S 2015/06/09 14:19:44 VCS NOTICE V-16-1-10235 Restart is set for group AppService. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2015/06/09 14:19:44 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/06/09 14:19:44 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB03SCluster as potential target for group AppService 2015/06/09 14:19:44 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB03SCluster. 2015/06/09 14:19:44 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB03SCluster and MILWB02SCluster 2015/06/09 14:19:44 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB03SCluster 2015/06/09 14:19:44 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/09 14:19:44 VCS NOTICE V-16-1-10231 Clearing start attributes for resources of group AppService on node MILWB02S 2015/06/09 14:19:44 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 14:19:44 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/06/09 14:19:44 VCS NOTICE V-16-1-51039 Received Authority request to online group AppService in cluster MILWB03SCluster 2015/06/09 14:19:44 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/09 14:19:44 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 14:19:44 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/09 14:19:44 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 14:19:54 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 14:19:56 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 14:20:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:20:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:20:26 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 14:20:37 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 14:21:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:21:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:21:07 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 14:21:07 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB03S in cluster MILWB03SCluster 2015/06/09 14:22:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:22:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:23:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:23:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:24:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:24:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:25:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:25:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:26:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:26:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:27:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:27:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:28:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:28:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:29:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:29:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:30:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:30:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:31:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:31:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:32:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:32:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:33:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:33:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:34:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:34:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:35:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:35:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:36:04 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:36:04 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:37:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:37:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:38:05 VCS WARNING V-16-10001-7502 (MILWB02S) NIC:appNIC:monitor:Invalid Device Name 2015/06/09 14:38:05 VCS WARNING V-16-10001-7509 (MILWB02S) NIC:appNIC:monitor:Device name (bge1) could be a typo or Device is not configured.Resource State is OFFLINE! 2015/06/09 14:38:58 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/06/09 14:39:06 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB02S (Not initiated by VCS) 2015/06/09 14:41:08 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 18:41:09 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 22:41:10 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 02:41:11 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 06:41:12 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 10:41:13 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 14:41:14 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 18:41:15 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 22:41:16 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 02:41:17 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 06:41:18 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 10:41:19 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 14:41:19 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 18:41:20 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 22:41:21 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/12 02:41:22 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/12 06:41:23 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/12 08:11:00 VCS INFO V-16-1-50803 Received request to switch group AppService from remote system MILWB03S to local system MILWB02S 2015/06/12 08:11:00 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB02S 2015/06/12 08:12:05 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/12 08:12:07 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/12 08:12:10 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/12 08:12:11 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/12 08:12:16 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/12 08:12:16 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB03S in cluster MILWB03SCluster 2015/06/12 08:12:16 VCS NOTICE V-16-1-50923 Local cluster MILWB02SCluster is assuming Authority for group AppService 2015/06/12 08:12:16 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/12 08:12:16 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/12 08:12:16 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/12 08:12:16 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB02S 2015/06/12 08:12:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/12 08:12:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/12 08:12:16 VCS INFO V-16-20010-84 (MILWB02S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/12 08:12:24 VCS INFO V-16-20010-76 (MILWB02S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/06/12 08:12:26 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/12 08:12:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/12 08:12:27 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/12 08:12:27 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/12 08:12:28 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/12 08:12:28 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/12 08:12:28 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/12 08:12:28 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/12 08:12:28 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt emhQmqFqsQlqJqrQjoTkfKgkH 2> /dev/null 2015/06/12 08:12:28 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/12 08:12:49 VCS INFO V-16-20018-999 (MILWB02S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/12 08:12:50 VCS INFO V-16-20018-7 (MILWB02S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/12899/psinfo 2015/06/12 08:12:50 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/12 08:12:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/12 08:13:01 VCS INFO V-16-20018-7 (MILWB02S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/17133/psinfo 2015/06/12 08:13:01 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/12 08:13:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB02S 2015/06/12 08:13:13 VCS INFO V-16-10001-88 (MILWB02S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/12 08:13:13 VCS INFO V-16-2-13716 (MILWB02S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 14661 12899 1 08:12:43 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x652c8952, 10000000000, 0x1c39 sybase 12899 12897 2 08:12:28 ? 0:19 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 14570 12899 0 08:12:43 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x652c8952, 10000000000, 0x1c39 sybase 14618 12899 0 08:12:43 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x652c8952, 10000000000, 0x1c39 15-06-12 08:13:02 || haStart:sybase is starting, break! 15-06-12 08:13:02 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-12 08:13:12 || haStart:The imap is running, break! 15-06-12 08:13:13 || haStart:Success to start the nms! ============================================== 2015/06/12 08:13:25 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB02S (VCS initiated) 2015/06/12 08:13:25 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB02S 2015/06/12 08:13:37 VCS NOTICE V-16-1-50086 CPU usage on MILWB02S is 79% 2015/06/12 08:14:07 VCS CRITICAL V-16-1-50086 CPU usage on MILWB02S is 96%