# # Log Name: engine # System: MILWB03S # SysInfo: Solaris:MILWB03S,Generic_144488-17,5.10,sun4u # Created: 2015/05/26 23:56:21 # 2015/05/26 23:56:21 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/26 23:56:21 VCS INFO V-16-1-10196 Cluster logger started 2015/05/26 23:56:21 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/26 23:56:21 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/26 23:56:21 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/26 23:56:21 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/26 23:56:21 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/26 23:56:21 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/26 23:56:26 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/26 23:56:26 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/26 23:56:26 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/26 23:56:26 VCS ERROR V-16-1-10621 Local cluster configuration error 2015/05/26 23:56:26 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/26 23:56:26 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/26 23:56:26 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/26 23:56:26 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/26 23:56:26 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from STALE_DISCOVER_WAIT to STALE_ADMIN_WAIT 2015/05/26 23:58:29 VCS ERROR V-16-1-10069 All systems have configuration files marked STALE. Unable to form cluster. 2015/05/26 23:58:29 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/26 23:58:29 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from STALE_ADMIN_WAIT to EXITED 2015/05/27 00:03:04 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 00:03:04 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 00:03:04 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 00:03:04 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 00:03:04 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 00:03:04 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 00:03:04 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 00:03:04 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 00:03:10 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 00:03:10 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 00:03:10 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 00:03:11 VCS ERROR V-16-1-10621 Local cluster configuration error 2015/05/27 00:03:11 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 00:03:11 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 00:03:11 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 00:03:11 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 00:03:11 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from STALE_DISCOVER_WAIT to STALE_ADMIN_WAIT 2015/05/27 00:03:37 VCS ERROR V-16-1-10069 All systems have configuration files marked STALE. Unable to form cluster. 2015/05/27 00:03:37 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 00:03:37 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from STALE_ADMIN_WAIT to EXITED 2015/05/27 00:14:18 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 00:14:18 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 00:14:18 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 00:14:18 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 00:14:18 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 00:14:18 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 00:14:18 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 00:14:18 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 00:14:22 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 00:14:22 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 00:14:22 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 00:14:24 VCS ERROR V-16-1-10621 Local cluster configuration error 2015/05/27 00:14:24 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 00:14:24 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 00:14:24 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 00:14:24 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 00:14:24 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from STALE_DISCOVER_WAIT to STALE_ADMIN_WAIT 2015/05/27 01:07:03 VCS ERROR V-16-1-10069 All systems have configuration files marked STALE. Unable to form cluster. 2015/05/27 01:07:03 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_FORCE_LOCAL MILWB03S from localhost 2015/05/27 01:07:03 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from STALE_ADMIN_WAIT to STALE_PEER_WAIT 2015/05/27 01:07:03 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from STALE_PEER_WAIT to LOCAL_BUILD 2015/05/27 01:07:04 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 01:07:04 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 01:07:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 01:07:04 2015 2015/05/27 01:07:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 01:07:04 2015 2015/05/27 01:07:05 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 01:07:05 VCS CRITICAL V-16-1-51135 GlobalCounter not updated 2015/05/27 01:07:05 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 01:07:05 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 01:07:05 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 01:07:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 01:07:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 01:07:05 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=PRC VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 01:07:05 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 01:07:05 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 01:07:06 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 01:07:06 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 01:07:06 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 01:07:07 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 01:07:07 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 01:07:07 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 01:07:07 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 01:07:15 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 01:07:15 VCS INFO V-16-1-50135 User root fired command: hagrp -add VVRService from localhost 2015/05/27 01:07:15 VCS INFO V-16-1-50135 User root fired command: hagrp -add AppService from localhost 2015/05/27 01:07:15 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService AutoStart 1 from localhost 2015/05/27 01:07:15 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService ManualOps 1 from localhost 2015/05/27 01:07:15 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService ClusterFailOverPolicy Auto from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService FailOverPolicy Priority from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService SourceFile ./main.cf from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService ManageFaults ALL from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ClusterService FaultPropagation 1 from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService AutoStart 1 from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService ManualOps 1 from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService ClusterFailOverPolicy Auto from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService FailOverPolicy Priority from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService SourceFile ./main.cf from localhost 2015/05/27 01:07:16 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService ManageFaults ALL from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService FaultPropagation 1 from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService SystemList MILWB03S 0 from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify VVRService AutoStartList MILWB03S from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService AutoStart 0 from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ManualOps 1 from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterFailOverPolicy Auto from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService FailOverPolicy Priority from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService SourceFile ./main.cf from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ManageFaults ALL from localhost 2015/05/27 01:07:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService FaultPropagation 1 from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService SystemList MILWB03S 0 from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hagrp -link AppService VVRService 1 1 from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Enabled 1 from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hares -modify wac User root from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Critical 1 from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hares -modify wac StartProgram /opt/VRTSvcs/bin/wacstart from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hares -modify wac MonitorProcesses /opt/VRTSvcs/bin/wac from localhost 2015/05/27 01:07:18 VCS INFO V-16-1-50135 User root fired command: hares -modify wac StopProgram /opt/VRTSvcs/bin/wacstop from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE wac MILWB03S from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -add datarvg RVG VVRService from localhost 2015/05/27 01:07:19 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 01:07:19 2015 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg RVG datarvg from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 0 from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg DiskGroup datadg from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Critical 0 from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE datarvg MILWB03S from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg RVG datarvg from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 0 from localhost 2015/05/27 01:07:19 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg DiskGroup datadg from localhost 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Critical 0 from localhost 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE datarvg MILWB03S from localhost 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -add RVGPrimary RVGPrimary AppService from localhost 2015/05/27 01:07:20 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 01:07:20 2015 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -add DatabaseServer Sybase AppService from localhost 2015/05/27 01:07:20 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 01:07:20 2015 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -add BackupServer SybaseBk AppService from localhost 2015/05/27 01:07:20 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 01:07:20 2015 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -add NMSServer Application AppService from localhost 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -add DataFilesystem Mount AppService from localhost 2015/05/27 01:07:20 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 01:07:20 2015 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -add appNIC NIC AppService from localhost 2015/05/27 01:07:20 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 01:07:20 2015 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary RvgResourceName datarvg from localhost 2015/05/27 01:07:20 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Critical 1 from localhost 2015/05/27 01:07:21 VCS ERROR V-16-20010-401 (MILWB03S) RVGPrimary:RVGPrimary:open:RVG attribute not assigned for resource ; please check the assignment of RvgResource name attribute for this resource or RVG attribute for 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary AutoResync 0 from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary AutoTakeover 1 from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE RVGPrimary MILWB03S from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SApswd sybase from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Server DBSVR from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Enabled 0 from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Home /opt/sybase from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Critical 1 from localhost 2015/05/27 01:07:21 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Owner sybase from localhost 2015/05/27 01:07:21 VCS INFO V-16-2-13716 (MILWB03S) Resource(RVGPrimary): Output of the completed operation (open) ============================================== VCS WARNING V-16-1-10260 Resource does not exist: RVG ============================================== 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Version 15.0 from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SA sa from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE DatabaseServer MILWB03S from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SApswd sybase from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Server DBSVR from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Enabled 0 from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Home /opt/sybase from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Critical 0 from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Owner sybase from localhost 2015/05/27 01:07:22 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Version 15.0 from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SA sa from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Backupserver DBSVR_back from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE BackupServer MILWB03S from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Enabled 1 from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer CleanProgram /opt/HWENGR/NMSApp/haClean from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer User root from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Critical 1 from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer StartProgram /opt/HWENGR/NMSApp/haStart from localhost 2015/05/27 01:07:23 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer MonitorProgram /opt/HWENGR/NMSApp/haMonitor from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer StopProgram /opt/HWENGR/NMSApp/haStop from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MILWB03S from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem MountPoint /opt/sybase/data from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem SecondLevelMonitor 0 from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Enabled 1 from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Critical 1 from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem BlockDevice /dev/vx/dsk/datadg/lv_nms_data from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem CkptUmount 1 from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem FsckOpt -y from localhost 2015/05/27 01:07:24 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem SecondLevelTimeout 30 from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem FSType ufs from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE DataFilesystem MILWB03S from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (First probe) 2015/05/27 01:07:25 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC NetworkHosts 10.66.28.67 from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Device bge1 from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Enabled 1 from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC PingOptimize 1 from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Critical 0 from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE appNIC MILWB03S from localhost 2015/05/27 01:07:25 VCS INFO V-16-1-50135 User root fired command: hares -link RVGPrimary appNIC 0 0 from localhost 2015/05/27 01:07:26 VCS INFO V-16-1-50135 User root fired command: hares -link DatabaseServer DataFilesystem 0 0 from localhost 2015/05/27 01:07:26 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 01:07:26 VCS INFO V-16-1-50135 User root fired command: hares -link BackupServer DatabaseServer 0 0 from localhost 2015/05/27 01:07:26 VCS INFO V-16-1-50135 User root fired command: hares -link NMSServer BackupServer 0 0 from localhost 2015/05/27 01:07:26 VCS INFO V-16-1-50135 User root fired command: hares -link DataFilesystem RVGPrimary 0 0 from localhost 2015/05/27 01:07:29 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 01:07:32 VCS INFO V-16-1-50135 User root fired command: hagrp -offline -force ClusterService MILWB03S localclus from localhost 2015/05/27 01:07:32 VCS NOTICE V-16-1-10167 Initiating manual offline of group ClusterService on system MILWB03S 2015/05/27 01:07:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 01:07:32 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/27 01:07:32 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 01:07:33 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 01:07:33 VCS INFO V-16-1-50135 User root fired command: haclus -modify ClusterName MILWB03SCluster from localhost 2015/05/27 01:07:33 VCS INFO V-16-1-50135 User root fired command: haclus -modify ClusterAddress 10.66.28.54 from localhost 2015/05/27 01:07:34 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/27 01:07:34 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/27 01:07:37 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 01:07:40 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS_FORCE from localhost 2015/05/27 01:07:40 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 01:07:40 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING_FORCIBLY 2015/05/27 01:07:40 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/27 01:07:40 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING_FORCIBLY to EXITED 2015/05/27 01:07:40 VCS INFO V-16-1-50016 Setting SYSSTATE to RSM_EXITED for node : MILWB03S. 2015/05/27 01:07:45 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 01:07:45 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 01:07:45 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 01:07:45 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 01:07:45 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 01:07:45 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 01:07:45 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 01:07:45 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 01:07:50 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 01:07:50 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 01:07:50 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 01:07:50 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 01:07:50 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 01:07:50 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 01:07:50 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 01:07:50 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 01:07:51 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 01:07:51 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 01:07:51 2015 2015/05/27 01:07:51 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 01:07:51 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin PWD=/var/VRTSvcs/diag/had TERM=sun TZ=PRC VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=MILWB03S _INIT_UTS_PLATFORM=SUNW,SPARC-Enterprise _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_144488-17 _INIT_ZONENAME=global 2015/05/27 01:07:51 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 01:07:51 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (First probe) 2015/05/27 01:07:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 01:07:51 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 01:07:51 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 01:07:51 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 01:07:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 01:07:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 01:07:52 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 01:07:52 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 01:07:52 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 01:07:52 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 01:07:52 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 01:07:53 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 01:07:53 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 01:07:53 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 01:07:53 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 01:07:53 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 01:07:53 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 01:07:53 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 01:16:45 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze AppService from localhost 2015/05/27 01:21:25 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 01:21:25 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SApswd FNIrNRgRTrMRkRSrKPuLGlHLi from localhost 2015/05/27 01:21:25 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SApswd FNIrNRgRTrMRkRSrKPuLGlHLi from localhost 2015/05/27 01:21:25 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 01:54:06 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 01:54:06 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer SApswd GOJsOShSUsNSlSTsLQvMHmIMj from localhost 2015/05/27 01:54:06 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer SApswd GOJsOShSUsNSlSTsLQvMHmIMj from localhost 2015/05/27 01:54:06 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 01:55:36 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 01:56:37 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 01:57:38 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 02:02:38 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 02:02:38 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 02:02:38 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 02:02:38 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 02:02:38 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 02:02:38 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 02:02:38 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 02:02:38 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 02:02:44 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 02:02:44 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 02:02:44 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 02:02:45 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 02:02:45 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 02:02:45 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 02:02:45 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 02:02:45 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 02:02:46 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 02:02:46 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:46 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 02:02:46 2015 2015/05/27 02:02:47 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 02:02:47 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:02:47 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 02:02:47 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 02:02:47 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 02:02:47 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 02:02:47 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 02:02:47 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=PRC VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 02:02:47 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 02:02:47 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 02:02:47 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 02:02:47 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 02:02:47 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 02:02:49 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 02:02:49 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 02:02:49 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 02:02:49 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 02:02:50 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:02:50 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:02:50 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 02:02:50 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 02:09:31 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 02:09:31 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 02:09:31 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/27 02:09:31 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 02:09:31 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/27 02:09:31 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:09:31 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:09:31 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S 0 from localhost 2015/05/27 02:09:31 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB03S from localhost 2015/05/27 02:09:32 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (First probe) 2015/05/27 02:09:32 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 02:09:32 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/27 02:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/27 02:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S 0 from localhost 2015/05/27 02:09:32 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB03S from localhost 2015/05/27 02:09:32 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/05/27 02:09:32 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 02:09:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:09:32 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:09:32 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:09:32 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:09:33 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:09:34 VCS INFO V-16-20010-72 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/27 02:09:34 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:09:36 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:09:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:09:37 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:09:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:09:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 02:09:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 02:09:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 02:09:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 02:09:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/27 02:09:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 02:10:08 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 02:10:09 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5756/psinfo 2015/05/27 02:10:09 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:10:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:10:20 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/14519/psinfo 2015/05/27 02:10:20 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:10:20 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:10:31 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 02:10:31 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 5756 5753 2 02:09:39 ? 0:26 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 12115 5756 1 02:10:03 ? 0:03 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 12016 5756 0 02:10:03 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 12066 5756 0 02:10:03 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 15-05-27 02:10:21 || haStart:sybase is starting, break! 15-05-27 02:10:21 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 02:10:31 || haStart:The imap is running, break! 15-05-27 02:10:31 || haStart:Success to start the nms! ============================================== 2015/05/27 02:10:40 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:10:40 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 02:24:38 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 02:24:38 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze -persistent AppService from localhost 2015/05/27 02:24:38 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 02:25:08 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 02:25:08 VCS INFO V-16-1-10307 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Not initiated by VCS) 2015/05/27 02:25:20 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 02:25:20 VCS INFO V-16-1-10307 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Not initiated by VCS) 2015/05/27 02:27:41 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 02:27:41 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze -persistent AppService from localhost 2015/05/27 02:27:41 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 02:27:42 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 02:27:42 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze -persistent AppService from localhost 2015/05/27 02:27:42 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 02:29:14 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 02:29:14 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze -persistent AppService from localhost 2015/05/27 02:29:14 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 02:30:09 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/29201/psinfo 2015/05/27 02:30:09 VCS INFO V-16-1-10299 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (Not initiated by VCS) 2015/05/27 02:30:09 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 02:30:21 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/29202/psinfo 2015/05/27 02:30:21 VCS INFO V-16-1-10299 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (Not initiated by VCS) 2015/05/27 02:30:21 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 02:30:21 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 02:44:55 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 02:44:55 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 02:44:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:44:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 02:44:55 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/27 02:44:55 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 02:44:56 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:44:56 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/27 02:44:56 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/27 02:44:56 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 02:44:56 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/27 02:45:33 VCS INFO V-16-20010-222 (MILWB03S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/27 02:45:33 VCS INFO V-16-20010-223 (MILWB03S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/27 02:45:55 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 02:45:57 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 02:45:58 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_monitomcat......................stopping... FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440....................stopping... PortTrunkingService.................stopped SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped ToolkitService......................stopped TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopped EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopped Eml_monitomcat......................stopped FaultService........................stopped ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService..........................stopped NmslogzipDm_4440................. ============================================== 2015/05/27 02:46:00 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:46:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:46:00 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 02:46:01 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 02:46:01 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 02:46:01 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:46:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:46:01 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 02:46:02 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 02:46:02 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 02:46:02 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:46:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:46:04 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:46:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:46:06 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 02:46:08 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:46:08 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/27 02:46:08 VCS NOTICE V-16-1-50061 Cannot initiate online of group AppService; this group may go online after its children fail over 2015/05/27 02:46:08 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 02:46:10 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 02:46:10 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:46:10 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/27 02:46:10 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/27 02:46:10 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 02:46:10 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 02:46:10 VCS WARNING V-16-1-50047 Clearing parent group AppService's migrateq, since child can not go online anywhere 2015/05/27 02:46:10 VCS NOTICE V-16-1-10500 Resetting Evacuating (lastval=0) for group AppService 2015/05/27 02:46:10 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/27 02:46:10 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/27 02:46:10 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/27 02:50:14 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 02:50:14 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 02:50:14 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 02:50:14 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 02:50:14 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 02:50:14 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 02:50:14 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 02:50:15 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 02:50:20 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 02:50:20 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 02:50:20 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 02:50:21 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 02:50:21 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 02:50:21 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 02:50:21 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 02:50:21 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 02:50:22 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 02:50:22 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:22 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 02:50:22 2015 2015/05/27 02:50:23 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 02:50:23 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:50:23 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:50:24 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:50:24 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 02:50:24 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 02:50:24 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 02:50:24 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 02:50:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 02:50:24 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=PRC VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 02:50:24 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 02:50:24 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 02:50:24 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 02:50:24 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 02:50:24 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 02:50:24 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 02:50:24 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/27 02:50:24 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/27 02:50:24 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/27 02:50:24 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 02:50:24 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/27 02:50:24 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/27 02:50:24 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 02:50:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 02:50:25 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 02:50:25 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 02:50:25 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered ============================================== 2015/05/27 02:50:25 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 02:50:25 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 02:50:26 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:50:27 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 02:50:27 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 02:50:27 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 02:50:27 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/27 02:50:27 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -clear VVRService from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S 0 from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -online VVRService MILWB03S from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -clear AppService from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S 0 from localhost 2015/05/27 02:57:05 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB03S from localhost 2015/05/27 02:57:05 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/05/27 02:57:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 02:57:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:57:05 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:57:05 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:57:05 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:57:06 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:57:07 VCS INFO V-16-20010-72 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/27 02:57:07 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 02:57:09 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:57:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:57:10 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:57:10 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:57:12 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 02:57:13 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 02:57:13 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 02:57:13 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 02:57:13 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/27 02:57:13 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 02:57:42 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 02:57:42 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/6085/psinfo 2015/05/27 02:57:42 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:57:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:57:54 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/14636/psinfo 2015/05/27 02:57:54 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:57:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 02:58:04 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 02:58:05 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 6085 6082 2 02:57:13 ? 0:27 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 12691 6085 0 02:57:38 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 12738 6085 1 02:57:38 ? 0:03 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 12642 6085 0 02:57:38 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 15-05-27 02:57:54 || haStart:sybase is starting, break! 15-05-27 02:57:54 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 02:58:04 || haStart:The imap is running, break! 15-05-27 02:58:04 || haStart:Success to start the nms! ============================================== 2015/05/27 02:58:17 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 02:58:17 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 03:01:21 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 03:01:22 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB03S 2015/05/27 03:01:22 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:23 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB03S (Not initiated by VCS) 2015/05/27 03:02:23 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 03:02:24 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 03:02:24 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... NmslogzipDm_4440....................stopping... PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService............ ============================================== 2015/05/27 03:02:26 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 03:02:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:26 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 03:02:27 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 03:02:27 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 03:02:27 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 03:02:27 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:28 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 03:02:28 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 03:02:29 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 03:02:29 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 03:02:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:30 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 03:02:30 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:32 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 03:02:34 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 03:02:34 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/27 03:02:34 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:35 VCS INFO V-16-20010-72 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/27 03:02:37 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 03:02:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:38 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 03:02:38 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:02:38 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 03:02:38 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 03:02:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 03:02:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 03:02:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/27 03:02:39 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 03:03:02 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 03:03:02 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/20484/psinfo 2015/05/27 03:03:02 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 03:03:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:03:14 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/22714/psinfo 2015/05/27 03:03:14 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 03:03:14 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 03:03:25 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 03:03:25 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 20484 20482 1 03:02:39 ? 0:14 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 21345 20484 1 03:02:51 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 21253 20484 0 03:02:51 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 21301 20484 0 03:02:51 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 15-05-27 03:03:15 || haStart:sybase is starting, break! 15-05-27 03:03:15 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 03:03:25 || haStart:The imap is running, break! 15-05-27 03:03:25 || haStart:Success to start the nms! ============================================== 2015/05/27 03:03:33 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 03:03:33 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 06:46:56 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 10:46:57 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 14:46:58 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 16:19:48 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/27 16:37:43 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB03S from ::ffff:10.66.28.70 2015/05/27 16:37:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 16:38:45 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 16:38:46 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 DCServer............................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... NmslogzipDm_4440....................stopping... PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopped UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ DCServer............................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopped Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopped LogService............ ============================================== 2015/05/27 16:38:48 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 16:39:53 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MILWB03S from ::ffff:10.66.28.70 2015/05/27 16:39:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 16:39:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 16:40:04 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 16:40:04 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 20484 20482 0 03:02:39 ? 7:06 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 21345 20484 0 03:02:51 ? 3:34 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 21253 20484 0 03:02:51 ? 2:46 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 21301 20484 0 03:02:51 ? 8:16 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 15-05-27 16:39:53 || haStart:sybase is starting, break! 15-05-27 16:39:53 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 16:40:03 || haStart:The imap is running, break! 15-05-27 16:40:03 || haStart:Success to start the nms! ============================================== 2015/05/27 16:40:12 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 16:40:12 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 17:37:50 VCS INFO V-16-1-50135 User admin fired command: hares -offline NMSServer MILWB03S from ::ffff:10.66.28.70 2015/05/27 17:37:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 17:38:53 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 17:38:53 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... DCServer............................stopping... DmsNetAclDm.........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... NmslogzipDm_4440....................stopping... Notify_Service......................stopping... PortTrunkingService.................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... ToolkitService......................stopping... TopoService.........................stopping... UniteUitlDM.........................stopping... XftpDm..............................stopping... dam.................................stopping... datacollectorsvr....................stopping... ds_889..............................stopping... mc..................................stopping... neLicServer.........................stopping... neproxy.............................stopping... profile.............................stopping... uflight_dispatcher..................stopping... ------------cost: 1(sec)------------ Agent_CORBA.........................stopping... DCServer............................stopping... DmsNetAclDm.........................stopped EAMService..........................stopped EmfGnlDevDm.........................stopping... Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_mo ============================================== 2015/05/27 17:38:56 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:31:03 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 18:31:03 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze AppService localclus from localhost 2015/05/27 18:31:04 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 18:39:05 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 18:45:29 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 18:45:29 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze AppService localclus from localhost 2015/05/27 18:45:30 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 18:56:34 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 18:56:36 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/27 18:56:54 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/27 18:56:56 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 18:57:10 VCS INFO V-16-1-50135 User root fired command: hagrp -offline -force AppService MILWB03S localclus from localhost 2015/05/27 18:57:10 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/27 18:57:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:57:10 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 18:57:11 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 18:57:12 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 18:57:12 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:57:12 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:57:12 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 18:57:13 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 18:57:13 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 18:57:13 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:57:13 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:57:14 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:57:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:57:16 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 18:57:19 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:57:19 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/27 18:57:21 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 18:58:33 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 18:58:34 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/27 18:58:34 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 18:58:38 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 18:58:38 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/27 18:58:39 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 18:58:40 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS_FORCE from localhost 2015/05/27 18:58:40 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 18:58:40 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 18:58:40 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/27 18:58:40 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 18:58:41 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:58:41 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/27 18:58:41 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/27 18:58:41 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 18:58:41 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/27 18:58:41 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/27 18:58:41 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/27 18:59:02 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 18:59:02 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 18:59:02 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 18:59:02 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 18:59:02 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 18:59:02 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 18:59:02 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 18:59:02 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 18:59:07 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 18:59:07 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 18:59:07 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 18:59:07 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 18:59:07 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 18:59:07 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 18:59:07 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 18:59:07 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 18:59:07 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 18:59:08 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 18:59:08 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 18:59:08 2015 2015/05/27 18:59:08 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 18:59:08 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger VCS_LOG=/var/VRTSvcs ########## Other Environment Variables ########## ACE_ROOT=/opt/U2000/server/3rdTools/ACE_wrappers AGENTPP_ROOT=/opt/U2000/server/3rdTools/agent++ CC_COMPILER=ForteCC CC_SW=cc CLRCASE_ROOT=/usr/atria CPPUNIT_ROOT=/opt/U2000/server/3rdTools/cppunit-1.8.0 CPU_SW=SUN_CPU DBSVR=DBSVR DES_ROOT=/opt/U2000/server/3rdTools/libdes DOCGEN_ROOT=/opt/doxygen-1.2.0 EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs FILEIO_LOCAL_CHARSET=1 FLEX_ROOT=/opt/U2000/server/3rdTools/flex-2.5.4 HWENGRROOT=/opt/HWENGR HWICMRROOT=/opt/HWICMR HWNMSJRE=/opt/HWNMSJRE ILOG_LICENSE_FILE=/opt/U2000/server/3rdTools/ilog/ilm/access.ilm ILOG_ROOT=/opt/U2000/server/3rdTools/ilog IMAP_CONFIG_PATH=/opt/U2000/server/etc/conf IMAP_DBTYPE=SYBASE IMAP_FAST_BUILD=1 IMAP_JAVA_HOME=/opt/HWNMSJRE/jre_sol IMAP_LOCALE=UTF-8 IMAP_MO_XML_TEMPLATE_DIR=/opt/U2000/server/etc/conf IMAP_ORBTYPE=TAO IMAP_ROOT=/opt/U2000/server INCLUDE=/opt/sybase/OCS-15_0/include:/opt/U2000/server/3rdTools/sybase/include:/opt/U2000/server/3rdTools/sybase/include _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=MILWB03S _INIT_UTS_PLATFORM=SUNW,SPARC-Enterprise _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_144488-17 _INIT_ZONENAME=global IT_CONFIG_DOMAINS_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/domains IT_DOMAIN_NAME=nbiorbix IT_LICENSE_FILE=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/licenses.txt IT_PRODUCT_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix LANG=en_US.UTF-8 LAN_IFNAME_0=e1000g0 LAN_IFNAME_1=e1000g1 LAN_IFNAME_2=e1000g2 LAN_IFNAME_3=e1000g3 LAN_nxge_0=nxge0 LAN_nxge_1=nxge1 LAN_nxge_2=nxge2 LAN_nxge_3=nxge3 LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH=/opt/VRTSvcs/lib:/opt/HWNMSJRE/jre_sol/lib/sparc/server:/opt/HWNMSJRE/jre_sol/lib/sparc:/opt/HWNMSJRE/jre_sol/../lib/sparc:/opt/sybase/OCS-15_0/lib:/opt/sybase/OCS-15_0/lib3p64:/opt/sybase/OCS-15_0/lib3p:/opt/sybase/ASE-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/opt/U2000/server/common/toolkit/server:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/lib:/usr/sfw/lib:/opt/SUNWspro/WS6U2/lib:/usr/local/lib:/opt/U2000/server/3rdTools/postgresql/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/usr/lib:/usr/dt/lib:/usr/openwin/lib:/opt/U2000/client/lib:/opt/U2000/server/cbb/frame/lib:/lib:/usr/ucblib LD_POINTER_GUARD=1 LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 LD_PRELOAD=libumem.so LIB=/opt/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib LOCALHOST_FIXSERVERNAME=10.66.28.70 LOCALHOST_FLOATSERVERNAME=10.66.28.70 MAKEFILE=makefile.tao MANPATH=:/opt/U2000/server/doc/cxx/man MASTERHOST_FLOATSERVERNAME=10.66.28.70 MED_SEND_ALARM_DIRECTLY=1 NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat NO_CLEARMAKE=1 OPERATION_SYSTEM=SUN_SVR4_OPSYS OSS_ROOT=/opt/U2000 PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/sybase/RPL-15_0/bin:/opt/sybase/ASEP/bin:/opt/sybase/DBISQL/bin:/opt/sybase/UAF-2_0/bin:/opt/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/bin:/opt/sybase/ASE-15_0/install:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/opt/U2000/server/3rdTools/bin:.:/export/forte6/SUNWspro/bin:/opt/U2000/server/3rdTools/postgresql/bin:/opt/U2000/server/3rdTools/usr/local/bin:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/3rdTools/flex-2.5.4/bin:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/usr/bin:/usr/sbin:/usr/openwin/bin:/usr/dt/bin:/usr/sfw/bin:/sbin:/opt/U2000/server/bin:/opt/U2000/client:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/U2000/server/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/U2000/server/3rdTools/sms/bin PGDATA=/op 2015/05/27 18:59:08 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 18:59:08 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:08 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:08 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:09 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 18:59:09 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 18:59:09 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 18:59:09 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 18:59:09 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 18:59:09 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 18:59:09 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 18:59:09 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 18:59:09 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 18:59:09 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (First probe) 2015/05/27 18:59:09 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 18:59:09 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/27 18:59:09 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/27 18:59:10 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 18:59:10 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 18:59:10 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 18:59:10 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 18:59:11 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:11 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:11 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 18:59:11 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 18:59:13 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 18:59:14 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 18:59:16 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 18:59:16 VCS INFO V-16-1-50135 User root fired command: hahb -add Icmp from localhost 2015/05/27 18:59:18 VCS INFO V-16-1-50135 User root fired command: haclus -add MILWB02SCluster 10.66.28.53 from localhost 2015/05/27 18:59:19 VCS INFO V-16-1-50135 User root fired command: hahb -modify ... -add Icmp ClusterList MILWB02SCluster from localhost 2015/05/27 18:59:19 VCS INFO V-16-1-50135 User root fired command: hahb -modify Icmp AYARetryLimit 9 from localhost 2015/05/27 18:59:19 VCS INFO V-16-1-50135 User root fired command: hahb -modify Icmp Arguments 10.66.28.53 MILWB02SCluster from localhost 2015/05/27 18:59:20 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB02SCluster 0 MILWB03SCluster 1 from localhost 2015/05/27 18:59:20 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 18:59:21 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS_FORCE from localhost 2015/05/27 18:59:21 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 18:59:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 18:59:21 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/27 18:59:21 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 18:59:22 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:59:22 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/27 18:59:22 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/27 18:59:22 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 18:59:22 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/27 18:59:22 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/27 18:59:22 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/27 18:59:43 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 18:59:43 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 18:59:43 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 18:59:43 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 18:59:43 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 18:59:43 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 18:59:43 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 18:59:43 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 18:59:47 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 18:59:47 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 18:59:47 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 18:59:47 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 18:59:47 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 18:59:47 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 18:59:47 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 18:59:47 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 18:59:47 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 18:59:48 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 18:59:48 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 18:59:48 2015 2015/05/27 18:59:48 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 18:59:48 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger VCS_LOG=/var/VRTSvcs ########## Other Environment Variables ########## ACE_ROOT=/opt/U2000/server/3rdTools/ACE_wrappers AGENTPP_ROOT=/opt/U2000/server/3rdTools/agent++ CC_COMPILER=ForteCC CC_SW=cc CLRCASE_ROOT=/usr/atria CPPUNIT_ROOT=/opt/U2000/server/3rdTools/cppunit-1.8.0 CPU_SW=SUN_CPU DBSVR=DBSVR DES_ROOT=/opt/U2000/server/3rdTools/libdes DOCGEN_ROOT=/opt/doxygen-1.2.0 EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs FILEIO_LOCAL_CHARSET=1 FLEX_ROOT=/opt/U2000/server/3rdTools/flex-2.5.4 HWENGRROOT=/opt/HWENGR HWICMRROOT=/opt/HWICMR HWNMSJRE=/opt/HWNMSJRE ILOG_LICENSE_FILE=/opt/U2000/server/3rdTools/ilog/ilm/access.ilm ILOG_ROOT=/opt/U2000/server/3rdTools/ilog IMAP_CONFIG_PATH=/opt/U2000/server/etc/conf IMAP_DBTYPE=SYBASE IMAP_FAST_BUILD=1 IMAP_JAVA_HOME=/opt/HWNMSJRE/jre_sol IMAP_LOCALE=UTF-8 IMAP_MO_XML_TEMPLATE_DIR=/opt/U2000/server/etc/conf IMAP_ORBTYPE=TAO IMAP_ROOT=/opt/U2000/server INCLUDE=/opt/sybase/OCS-15_0/include:/opt/U2000/server/3rdTools/sybase/include:/opt/U2000/server/3rdTools/sybase/include _INIT_NET_STRATEGY=none _INIT_PREV_LEVEL=S _INIT_RUN_LEVEL=3 _INIT_RUN_NPREV=0 _INIT_UTS_ISA=sparc _INIT_UTS_MACHINE=sun4u _INIT_UTS_NODENAME=MILWB03S _INIT_UTS_PLATFORM=SUNW,SPARC-Enterprise _INIT_UTS_RELEASE=5.10 _INIT_UTS_SYSNAME=SunOS _INIT_UTS_VERSION=Generic_144488-17 _INIT_ZONENAME=global IT_CONFIG_DOMAINS_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/domains IT_DOMAIN_NAME=nbiorbix IT_LICENSE_FILE=/../cbb/nbi/nbicbb_3p/tools/orbix/etc/licenses.txt IT_PRODUCT_DIR=/../cbb/nbi/nbicbb_3p/tools/orbix LANG=en_US.UTF-8 LAN_IFNAME_0=e1000g0 LAN_IFNAME_1=e1000g1 LAN_IFNAME_2=e1000g2 LAN_IFNAME_3=e1000g3 LAN_nxge_0=nxge0 LAN_nxge_1=nxge1 LAN_nxge_2=nxge2 LAN_nxge_3=nxge3 LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH=/opt/VRTSvcs/lib:/opt/HWNMSJRE/jre_sol/lib/sparc/server:/opt/HWNMSJRE/jre_sol/lib/sparc:/opt/HWNMSJRE/jre_sol/../lib/sparc:/opt/sybase/OCS-15_0/lib:/opt/sybase/OCS-15_0/lib3p64:/opt/sybase/OCS-15_0/lib3p:/opt/sybase/ASE-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/opt/U2000/server/common/toolkit/server:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/lib:/usr/sfw/lib:/opt/SUNWspro/WS6U2/lib:/usr/local/lib:/opt/U2000/server/3rdTools/postgresql/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p64:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib3p:/usr/lib:/usr/dt/lib:/usr/openwin/lib:/opt/U2000/client/lib:/opt/U2000/server/cbb/frame/lib:/lib:/usr/ucblib LD_POINTER_GUARD=1 LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 LD_PRELOAD=libumem.so LIB=/opt/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib:/opt/U2000/server/3rdTools/sybase/OCS-15_0/lib LOCALHOST_FIXSERVERNAME=10.66.28.70 LOCALHOST_FLOATSERVERNAME=10.66.28.70 MAKEFILE=makefile.tao MANPATH=:/opt/U2000/server/doc/cxx/man MASTERHOST_FLOATSERVERNAME=10.66.28.70 MED_SEND_ALARM_DIRECTLY=1 NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat NO_CLEARMAKE=1 OPERATION_SYSTEM=SUN_SVR4_OPSYS OSS_ROOT=/opt/U2000 PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/sybase/RPL-15_0/bin:/opt/sybase/ASEP/bin:/opt/sybase/DBISQL/bin:/opt/sybase/UAF-2_0/bin:/opt/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/bin:/opt/sybase/ASE-15_0/install:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/opt/U2000/server/3rdTools/bin:.:/export/forte6/SUNWspro/bin:/opt/U2000/server/3rdTools/postgresql/bin:/opt/U2000/server/3rdTools/usr/local/bin:/opt/U2000/server/3rdTools/lib:/opt/U2000/server/3rdTools/flex-2.5.4/bin:/opt/U2000/server/3rdTools/sybase/OCS-15_0/bin:/usr/bin:/usr/sbin:/usr/openwin/bin:/usr/dt/bin:/usr/sfw/bin:/sbin:/opt/U2000/server/bin:/opt/U2000/client:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/U2000/server/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/U2000/server/3rdTools/sms/bin PGDATA=/op 2015/05/27 18:59:49 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 18:59:49 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:49 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:49 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:49 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 18:59:49 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 18:59:49 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 18:59:49 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 18:59:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 18:59:49 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 18:59:49 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 18:59:49 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 18:59:49 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 18:59:49 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/27 18:59:50 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (First probe) 2015/05/27 18:59:50 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 18:59:50 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/27 18:59:50 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/27 18:59:50 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 18:59:50 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 18:59:51 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 18:59:51 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 18:59:51 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:51 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 18:59:51 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 18:59:51 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 19:01:09 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 19:01:09 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 19:01:10 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources ClusterService from localhost 2015/05/27 19:01:10 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources VVRService from localhost 2015/05/27 19:01:11 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 19:01:11 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/27 19:01:11 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:11 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:11 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:12 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/27 19:01:12 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:01:12 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/27 19:01:12 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/27 19:01:12 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:13 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:47 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/27 19:01:51 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/27 19:01:51 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:51 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:51 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:52 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:01:53 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 19:02:00 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 19:02:01 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost 2015/05/27 19:02:01 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 19:02:43 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:02:43 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:02:43 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:02:44 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:02:44 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:02:48 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:03:32 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/27 20:50:29 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 20:50:29 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 20:50:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 20:50:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 20:50:29 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/27 20:50:30 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 20:50:31 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/27 20:50:31 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/27 20:50:31 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/27 20:50:31 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 20:50:31 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/27 20:50:31 VCS INFO V-16-20010-222 (MILWB03S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/27 20:50:33 VCS INFO V-16-20010-223 (MILWB03S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/27 20:50:33 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 20:50:34 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/27 20:50:34 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/27 20:50:34 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/27 20:50:34 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 20:50:34 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 20:50:34 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/27 20:50:34 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/27 20:50:34 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/27 14:54:46 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 14:54:46 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 14:54:46 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 14:54:46 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 14:54:46 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 14:54:46 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 14:54:46 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 14:54:46 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 14:54:50 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 14:54:50 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 14:54:50 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 14:54:51 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 14:54:51 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 14:54:51 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 14:54:51 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 14:54:51 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 14:54:53 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 14:54:53 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 14:54:53 2015 2015/05/27 14:54:54 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 14:54:54 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 14:54:54 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 14:54:54 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 14:54:54 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 14:54:54 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 14:54:54 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 14:54:54 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 14:54:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 14:54:55 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 14:54:55 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 14:54:55 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 14:54:55 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 14:54:55 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 14:54:55 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 14:54:55 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 14:54:55 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/27 14:54:55 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/27 14:54:55 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/27 14:54:55 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/27 14:54:55 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 14:54:55 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/27 14:54:55 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/27 14:54:55 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 14:54:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 14:54:56 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 14:54:56 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 14:54:56 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/27 14:54:56 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 14:54:56 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 14:54:57 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 14:54:57 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 14:54:57 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 14:54:57 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 14:54:57 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/27 14:54:57 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/27 14:55:00 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/27 14:55:00 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 14:55:00 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/27 14:55:00 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/27 14:56:52 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/27 15:42:52 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 15:42:52 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 15:42:52 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 15:42:52 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 15:42:53 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/27 15:42:53 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 15:42:54 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/27 15:42:54 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/27 15:42:54 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/27 15:42:54 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 15:42:54 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/27 15:42:54 VCS INFO V-16-20010-222 (MILWB03S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/27 15:42:54 VCS INFO V-16-20010-223 (MILWB03S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/27 15:42:55 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 15:42:56 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/27 15:42:56 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/27 15:42:56 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/27 15:42:56 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 15:42:56 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 15:42:56 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/27 15:42:56 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/27 15:42:56 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/27 15:47:07 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 15:47:07 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 15:47:07 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 15:47:07 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 15:47:07 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 15:47:07 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 15:47:07 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 15:47:07 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 15:47:13 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 15:47:13 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 15:47:13 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 15:47:13 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 15:47:13 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 15:47:13 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 15:47:13 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 15:47:13 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 15:47:15 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 15:47:15 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:15 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 15:47:15 2015 2015/05/27 15:47:16 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 15:47:16 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:47:16 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:47:17 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:47:17 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 15:47:17 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 15:47:17 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 15:47:17 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 15:47:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 15:47:17 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 15:47:17 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 15:47:17 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 15:47:17 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 15:47:17 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/27 15:47:17 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/27 15:47:17 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/27 15:47:17 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 15:47:17 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/27 15:47:17 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/27 15:47:17 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 15:47:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 15:47:17 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 15:47:17 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 15:47:17 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 15:47:17 VCS NOTICE V-16-3-18321 (MILWB03S) All heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are down. will not accept connection from cluster (MILWB02SCluster) until at least one heartbeat to cluster (MILWB02SCluster) is alive 2015/05/27 15:47:18 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 15:47:18 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 15:47:18 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/27 15:47:18 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/27 15:47:18 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 15:47:18 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 15:47:18 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:47:19 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/27 15:47:19 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/27 15:47:20 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:47:20 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 15:47:20 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 15:47:23 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/27 15:47:23 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:47:23 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/27 15:47:23 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/27 15:55:11 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/27 15:55:11 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/27 15:55:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 15:55:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 15:55:11 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/27 15:55:11 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/27 15:55:13 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/27 15:55:13 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/27 15:55:13 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/27 15:55:13 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 15:55:13 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/27 15:55:13 VCS INFO V-16-20010-222 (MILWB03S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/27 15:55:13 VCS INFO V-16-20010-223 (MILWB03S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/27 15:55:14 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 15:55:15 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/27 15:55:15 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/27 15:55:15 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/27 15:55:15 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 15:55:15 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/27 15:55:15 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/27 15:55:15 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/27 15:55:15 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/27 15:59:24 VCS INFO V-16-1-10196 Cluster logger started 2015/05/27 15:59:24 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/27 15:59:24 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/27 15:59:24 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/27 15:59:24 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/27 15:59:24 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/27 15:59:24 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/27 15:59:24 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/27 15:59:29 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/27 15:59:29 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/27 15:59:29 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/27 15:59:30 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/27 15:59:30 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/27 15:59:30 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/27 15:59:30 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/27 15:59:30 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/27 15:59:32 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/27 15:59:32 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:32 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed May 27 15:59:32 2015 2015/05/27 15:59:33 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/27 15:59:33 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:59:33 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:59:33 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/27 15:59:33 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/27 15:59:33 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/27 15:59:33 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/27 15:59:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/27 15:59:33 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:59:33 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/27 15:59:33 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/27 15:59:34 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/27 15:59:34 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/27 15:59:34 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/27 15:59:34 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/27 15:59:34 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/27 15:59:34 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/27 15:59:34 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/27 15:59:34 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/27 15:59:34 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/27 15:59:34 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/27 15:59:34 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/27 15:59:34 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/27 15:59:34 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/27 15:59:35 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/27 15:59:35 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/27 15:59:35 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/27 15:59:35 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/27 15:59:35 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:59:35 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/27 15:59:35 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 15:59:35 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/27 15:59:35 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/27 15:59:36 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/27 15:59:36 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/27 15:59:36 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/27 15:59:36 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/27 15:59:37 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/27 15:59:37 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/27 15:59:37 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/27 17:42:48 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:43:56 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:43:56 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:58:14 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:58:16 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:58:18 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:58:19 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:58:24 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:58:24 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 17:59:10 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/27 17:59:31 VCS INFO V-16-1-50135 User admin fired command: hagrp -online AppService MILWB03S localclus from ::ffff:10.66.28.69 2015/05/27 17:59:31 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/27 17:59:31 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB03S 2015/05/27 17:59:31 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/27 17:59:31 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 17:59:31 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/05/27 17:59:31 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 17:59:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 17:59:32 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/27 17:59:41 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/27 17:59:43 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 17:59:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 17:59:44 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 17:59:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 17:59:46 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 17:59:46 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 17:59:46 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 17:59:46 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 17:59:46 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/27 17:59:47 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 18:00:11 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 18:00:12 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/22252/psinfo 2015/05/27 18:00:12 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 18:00:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:00:24 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/28693/psinfo 2015/05/27 18:00:24 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 18:00:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:00:35 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 18:00:35 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 25449 22252 1 18:00:04 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 22252 22249 2 17:59:46 ? 0:22 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 25355 22252 0 18:00:04 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 25402 22252 0 18:00:04 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 15-05-27 18:00:25 || haStart:sybase is starting, break! 15-05-27 18:00:25 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 18:00:35 || haStart:The imap is running, break! 15-05-27 18:00:35 || haStart:Success to start the nms! ============================================== 2015/05/27 18:00:40 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 18:00:40 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 18:01:08 VCS NOTICE V-16-1-50086 CPU usage on MILWB03S is 70% 2015/05/27 18:04:08 VCS INFO V-16-1-50086 CPU usage on MILWB03S is 63% 2015/05/27 18:50:03 VCS INFO V-16-1-50135 User admin fired command: hagrp -offline AppService MILWB03S localclus from ::ffff:10.66.28.69 2015/05/27 18:50:03 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/27 18:50:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:51:05 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 18:51:05 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopped LogService..........................stopped Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopping... TrapReceiver........................stopping... UniteUitlDM.. ============================================== 2015/05/27 18:51:07 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:51:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:51:07 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 18:51:08 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 18:51:08 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 18:51:08 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:51:08 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:51:10 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 18:51:10 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 18:51:10 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 18:51:10 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:51:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:51:12 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:51:12 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 18:51:14 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 18:51:16 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 18:51:16 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/27 18:51:46 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/27 18:51:46 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/27 18:51:46 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/27 18:51:46 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 18:51:46 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/27 18:51:46 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 18:51:58 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 18:51:59 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 18:52:24 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 18:52:35 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 18:53:00 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 18:53:00 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:09:25 VCS INFO V-16-1-50806 Received request to switch group AppService from remote system MILWB02S to best possible local system 2015/05/27 19:09:25 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB03S 2015/05/27 19:10:31 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:10:32 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:10:34 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:10:36 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:10:40 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:10:40 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/27 19:10:40 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/27 19:10:40 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/05/27 19:10:40 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/27 19:10:40 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 19:10:40 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/27 19:10:40 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 19:10:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:10:40 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/27 19:11:04 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/27 19:11:06 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 19:11:06 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:11:07 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 19:11:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:11:07 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/27 19:11:07 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/27 19:11:07 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/27 19:11:07 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/27 19:11:07 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/27 19:11:07 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/27 19:11:29 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/27 19:11:29 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/19359/psinfo 2015/05/27 19:11:29 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 19:11:29 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:11:41 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/23895/psinfo 2015/05/27 19:11:41 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 19:11:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:11:52 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/27 19:11:53 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 19359 19357 2 19:11:08 ? 0:18 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 21357 19359 0 19:11:23 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 21403 19359 1 19:11:23 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 21311 19359 0 19:11:23 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 15-05-27 19:11:42 || haStart:sybase is starting, break! 15-05-27 19:11:42 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-27 19:11:52 || haStart:The imap is running, break! 15-05-27 19:11:52 || haStart:Success to start the nms! ============================================== 2015/05/27 19:12:07 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/27 19:12:07 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/27 19:12:38 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 98% 2015/05/27 19:41:48 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:41:49 VCS ERROR V-16-1-10303 Resource appNIC (Owner: Unspecified, Group: AppService) is FAULTED (timed out) on sys MILWB03S 2015/05/27 19:41:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:42:51 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/27 19:42:51 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... U ============================================== 2015/05/27 19:42:53 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 19:42:53 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:42:54 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/27 19:42:54 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/27 19:42:55 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/27 19:42:55 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 19:42:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:42:57 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/27 19:42:58 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/27 19:42:58 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/27 19:42:58 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 19:42:58 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:42:59 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 19:42:59 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/27 19:43:01 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/27 19:43:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:43:03 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/27 19:43:03 VCS ERROR V-16-1-10205 Group AppService is faulted on system MILWB03S 2015/05/27 19:43:03 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/27 19:43:03 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/27 19:43:03 VCS INFO V-16-1-50010 Group AppService is online or faulted on system MILWB03S 2015/05/27 19:43:03 VCS NOTICE V-16-1-10235 Restart is set for group AppService. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2015/05/27 19:43:03 VCS WARNING V-16-1-50911 Unable to fail over global group AppService in local cluster. Attempting to fail group over to a remote cluster [ClusterFailoverPolicy = Auto] 2015/05/27 19:43:03 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB02SCluster as potential target for group AppService 2015/05/27 19:43:04 VCS INFO V-16-1-50907 Found a possible target remote system in cluster MILWB02SCluster. 2015/05/27 19:43:04 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB02SCluster and MILWB03SCluster 2015/05/27 19:43:04 VCS INFO V-16-1-50917 Initiating online of group AppService on cluster MILWB02SCluster 2015/05/27 19:43:04 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/27 19:43:04 VCS NOTICE V-16-1-10231 Clearing start attributes for resources of group AppService on node MILWB03S 2015/05/27 19:43:04 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/27 19:44:04 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:44:17 VCS ERROR V-16-3-18313 (MILWB03S) TCP/IP connection via connector from cluster MILWB03SCluster to cluster MILWB02SCluster is hung; intentionally disconnecting. Auto-reconnect will occur however you may wish to examine the wac resource on system 10.66.28.53 2015/05/27 19:44:17 VCS ERROR V-16-3-18311 (MILWB03S) MILWB03SCluster lost connection to cluster MILWB02SCluster 2015/05/27 19:44:17 VCS WARNING V-16-3-18150 Target disconnected before response returned; command may or may not have been delivered 2015/05/27 19:45:04 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:46:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:47:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:48:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:49:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:50:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:51:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:52:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:53:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:54:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:55:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:56:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:56:12 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/27 19:57:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:58:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 19:59:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:00:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:01:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:02:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:03:04 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:04:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:05:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:06:03 VCS WARNING V-16-10001-7506 (MILWB03S) NIC:appNIC:monitor:Resource is offline. No Network Host could be reached 2015/05/27 20:06:06 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/27 20:06:06 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:06:06 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/27 20:06:06 VCS INFO V-16-1-50918 Local cluster MILWB03SCluster and remote cluster MILWB02SCluster have Authority for group AppService. Giving up Authority since group is online on remote cluster and offline in local cluster 2015/05/27 20:06:06 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/27 20:06:06 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/27 20:06:06 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/27 20:06:50 VCS INFO V-16-1-10299 Resource appNIC (Owner: Unspecified, Group: AppService) is online on MILWB03S (Not initiated by VCS) 2015/05/27 20:09:29 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/27 20:13:11 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:13:13 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/27 20:13:31 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/27 20:13:31 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:13:31 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:13:31 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:13:32 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:13:33 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:13:47 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:14:08 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:14:08 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:14:08 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:14:09 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:14:09 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:14:13 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:14:28 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:14:29 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/27 20:14:47 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/27 20:14:48 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:14:48 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:14:48 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:14:49 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:14:50 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/27 20:15:04 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:15:31 VCS ERROR V-16-2-13067 (MILWB03S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/05/27 20:15:31 VCS WARNING V-16-20010-301 (MILWB03S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/05/27 20:15:31 VCS ERROR V-16-20010-302 (MILWB03S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/05/27 20:15:32 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/05/27 20:15:32 VCS ERROR V-16-2-13069 (MILWB03S) Resource(datarvg) - clean failed. 2015/05/27 20:16:31 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:16:32 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/27 20:16:32 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:16:37 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/27 20:16:39 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB02SCluster 0 MILWB03SCluster 1 from localhost 2015/05/27 20:16:39 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/27 20:16:42 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:16:42 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:16:42 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:16:42 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:16:42 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:16:42 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:16:42 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:17:04 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:17:04 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:17:04 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:17:05 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:17:05 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:17:09 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/27 20:22:25 VCS INFO V-16-1-50135 User admin fired command: haconf -dump from ::ffff:10.66.28.69 2015/05/27 23:56:13 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 03:56:14 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 07:56:15 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 08:38:04 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 08:50:46 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 08:50:46 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 08:59:25 VCS WARNING V-16-1-50132 User admin, logging in from ::ffff:10.66.28.69 has wrong password 2015/05/28 08:59:37 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/28 10:12:46 VCS INFO V-16-1-50135 User admin fired command: haconf -makerw from ::ffff:10.66.28.69 2015/05/28 10:12:46 VCS INFO V-16-1-50135 User admin fired command: hagrp -freeze -persistent AppService from ::ffff:10.66.28.69 2015/05/28 10:14:11 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/28 10:14:11 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/28 10:14:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 10:14:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 10:14:11 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/28 10:14:11 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/28 10:14:12 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/28 10:14:12 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/28 10:14:12 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/28 10:14:12 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 10:14:12 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/28 10:14:12 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 10:14:13 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/28 10:14:13 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/28 10:14:13 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/28 10:14:13 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 10:14:13 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 10:14:13 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/28 10:14:13 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/28 10:14:13 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/28 10:18:32 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/28 10:18:32 VCS INFO V-16-1-10196 Cluster logger started 2015/05/28 10:18:32 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/28 10:18:32 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/28 10:18:32 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/28 10:18:32 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/28 10:18:32 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/28 10:18:32 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/28 10:18:37 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/28 10:18:37 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/28 10:18:37 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/28 10:18:38 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/28 10:18:38 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/28 10:18:38 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/28 10:18:38 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/28 10:18:38 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/28 10:18:40 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/28 10:18:40 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu May 28 10:18:41 2015 2015/05/28 10:18:42 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/28 10:18:44 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 10:18:44 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 10:18:44 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 10:18:44 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/28 10:18:44 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/28 10:18:44 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/28 10:18:44 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/28 10:18:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 10:18:44 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 10:18:45 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/28 10:18:45 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/28 10:18:45 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/28 10:18:45 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 10:18:45 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/28 10:18:45 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/28 10:18:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/28 10:18:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 10:18:45 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/28 10:18:45 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/28 10:18:45 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/28 10:18:45 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/28 10:18:45 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/28 10:18:46 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/28 10:18:46 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/28 10:18:46 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/28 10:18:46 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/28 10:18:46 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/28 10:18:46 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/28 10:18:46 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/28 10:18:47 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/28 10:18:47 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/28 10:18:48 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 10:18:48 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/28 10:18:48 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:18:48 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/28 10:18:48 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/28 10:18:49 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 10:18:49 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/28 10:18:49 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/28 10:19:52 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/05/28 10:20:57 VCS INFO V-16-1-50806 Received request to switch group AppService from remote system MILWB02S to best possible local system 2015/05/28 10:20:57 VCS INFO V-16-1-50804 Group AppService is able to go online on local system MILWB03S 2015/05/28 10:22:02 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:22:23 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:22:26 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:22:27 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:22:31 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:22:31 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:22:31 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/28 10:22:31 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/05/28 10:22:31 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/28 10:22:31 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 10:22:31 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/28 10:22:31 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 10:22:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 10:22:32 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/28 10:23:17 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/28 10:23:19 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 10:23:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 10:23:20 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 10:23:20 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 10:23:22 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/28 10:23:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/28 10:23:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/28 10:23:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/28 10:23:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/28 10:23:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/28 10:23:55 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/28 10:23:56 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5690/psinfo 2015/05/28 10:23:56 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 10:23:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 10:24:07 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/15171/psinfo 2015/05/28 10:24:07 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 10:24:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 10:24:18 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 10:24:18 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 5690 5687 2 10:23:23 ? 0:27 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 11959 5690 0 10:23:48 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 12004 5690 1 10:23:48 ? 0:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 11913 5690 0 10:23:47 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 15-05-28 10:24:08 || haStart:sybase is starting, break! 15-05-28 10:24:08 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 10:24:18 || haStart:The imap is running, break! 15-05-28 10:24:18 || haStart:Success to start the nms! ============================================== 2015/05/28 10:24:49 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 10:24:49 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/28 10:25:17 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 91% 2015/05/28 10:29:51 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/05/28 10:29:51 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/05/28 10:31:53 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/28 10:34:21 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/28 10:34:21 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 10:34:21 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/28 10:34:21 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/28 11:40:30 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/05/28 11:40:30 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/05/28 11:40:30 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/05/28 11:40:30 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/28 11:40:30 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 11:41:32 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/28 11:41:33 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping... UniteUitlDM.. ============================================== 2015/05/28 11:41:35 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 11:41:35 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 11:41:35 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/28 11:41:36 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/28 11:41:36 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/28 11:41:36 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 11:41:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 11:41:37 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/28 11:41:37 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/28 11:41:37 VCS CRITICAL V-16-20018-4 (MILWB03S) Sybase:DatabaseServer:monitor:Process dataserver restarted 2015/05/28 11:41:37 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 11:41:37 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 11:41:39 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 11:41:39 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 11:41:41 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/28 11:41:43 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 11:41:43 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/28 11:41:43 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/28 11:41:43 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/28 11:41:43 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/28 11:41:43 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/28 11:41:43 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 11:41:43 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/28 11:41:43 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 11:42:10 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:42:11 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:42:37 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:42:49 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:43:36 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:43:36 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:46:37 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/28 11:46:37 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/28 11:46:37 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 11:46:37 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 11:46:38 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/28 11:46:38 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/28 11:46:39 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/28 11:46:39 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/28 11:46:39 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/28 11:46:39 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 11:46:39 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/28 11:46:39 VCS INFO V-16-20010-222 (MILWB03S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/28 11:46:40 VCS INFO V-16-20010-223 (MILWB03S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/28 11:46:40 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 11:46:41 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/28 11:46:41 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/28 11:46:41 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/28 11:46:41 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 11:46:41 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 11:46:41 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/28 11:46:41 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/28 11:46:41 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/28 11:54:16 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/28 11:54:16 VCS INFO V-16-1-10196 Cluster logger started 2015/05/28 11:54:16 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/28 11:54:16 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/28 11:54:16 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/28 11:54:16 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/28 11:54:16 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/28 11:54:16 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/28 11:54:21 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/28 11:54:21 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/28 11:54:21 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/28 11:54:22 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/28 11:54:22 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/28 11:54:22 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/28 11:54:22 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/28 11:54:22 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/28 11:54:23 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/28 11:54:23 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/28 11:54:23 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu May 28 11:54:23 2015 2015/05/28 11:54:24 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu May 28 11:54:24 2015 2015/05/28 11:54:24 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu May 28 11:54:24 2015 2015/05/28 11:54:24 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu May 28 11:54:24 2015 2015/05/28 11:54:24 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu May 28 11:54:24 2015 2015/05/28 11:54:24 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu May 28 11:54:24 2015 2015/05/28 11:54:24 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu May 28 11:54:24 2015 2015/05/28 11:54:24 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu May 28 11:54:24 2015 2015/05/28 11:54:25 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/28 11:54:25 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 11:54:25 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 11:54:25 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 11:54:25 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/28 11:54:25 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/28 11:54:25 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/28 11:54:25 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/28 11:54:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 11:54:26 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/28 11:54:26 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/28 11:54:26 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 11:54:26 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/28 11:54:26 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/28 11:54:26 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/28 11:54:26 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/28 11:54:26 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 11:54:26 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/28 11:54:26 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/28 11:54:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/28 11:54:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 11:54:26 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/28 11:54:26 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/28 11:54:26 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/28 11:54:27 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/28 11:54:27 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/28 11:54:27 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/28 11:54:27 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/28 11:54:27 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/28 11:54:27 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/28 11:54:28 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/28 11:54:28 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/28 11:54:28 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 11:54:29 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 11:54:29 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/28 11:54:29 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/28 11:54:32 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/28 11:54:32 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 11:54:32 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/28 11:54:32 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/28 12:03:07 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/28 12:03:07 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/28 12:03:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 12:03:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 12:03:07 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/28 12:03:07 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/28 12:03:08 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/28 12:03:08 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/28 12:03:08 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/28 12:03:08 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 12:03:08 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/28 12:03:08 VCS INFO V-16-20010-222 (MILWB03S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/28 12:03:09 VCS INFO V-16-20010-223 (MILWB03S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/28 12:03:09 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 12:03:10 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/28 12:03:10 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/28 12:03:10 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/28 12:03:10 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 12:03:10 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 12:03:10 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/28 12:03:10 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/28 12:03:10 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/28 12:07:30 VCS INFO V-16-1-10196 Cluster logger started 2015/05/28 12:07:30 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/28 12:07:30 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/28 12:07:30 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/28 12:07:30 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/28 12:07:30 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/28 12:07:30 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/28 12:07:30 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/28 12:07:35 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/28 12:07:35 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/28 12:07:35 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/28 12:07:36 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/28 12:07:37 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/28 12:07:37 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/28 12:07:37 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/28 12:07:37 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/28 12:07:38 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/28 12:07:38 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:38 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu May 28 12:07:38 2015 2015/05/28 12:07:39 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/28 12:07:39 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:07:39 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:07:40 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:07:40 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/28 12:07:40 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/28 12:07:40 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/28 12:07:40 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/28 12:07:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 12:07:40 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/28 12:07:40 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/28 12:07:40 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/28 12:07:40 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/28 12:07:40 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 12:07:40 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/28 12:07:40 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/28 12:07:40 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/28 12:07:40 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/28 12:07:40 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/28 12:07:40 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 12:07:40 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/28 12:07:40 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/28 12:07:40 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/28 12:07:40 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 12:07:40 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/28 12:07:41 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/28 12:07:41 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/28 12:07:41 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/28 12:07:41 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/28 12:07:41 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/28 12:07:41 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:07:43 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/28 12:07:43 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/28 12:07:45 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/28 12:07:45 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:07:45 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/28 12:07:45 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/28 12:07:46 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:07:46 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/28 12:07:46 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/28 12:10:12 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost 2015/05/28 12:10:12 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/05/28 12:10:12 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 12:10:12 VCS NOTICE V-16-1-10300 Initiating Offline of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 12:10:12 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/05/28 12:10:12 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/05/28 12:10:13 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/05/28 12:10:13 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/05/28 12:10:13 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group ClusterService 2015/05/28 12:10:13 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 12:10:13 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MILWB03S 2015/05/28 12:10:13 VCS INFO V-16-20010-222 (MILWB03S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon. 2015/05/28 12:10:14 VCS INFO V-16-20010-223 (MILWB03S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon. 2015/05/28 12:10:14 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 12:10:15 VCS INFO V-16-1-10305 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (VCS initiated) 2015/05/28 12:10:15 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/05/28 12:10:15 VCS NOTICE V-16-1-10010 Stopping all agents 2015/05/28 12:10:15 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 12:10:15 VCS INFO V-16-1-10494 System MILWB03S not in RUNNING state 2015/05/28 12:10:15 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group VVRService for node MILWB03S 2015/05/28 12:10:15 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/05/28 12:10:15 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/05/28 12:18:16 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/28 12:18:16 VCS INFO V-16-1-10196 Cluster logger started 2015/05/28 12:18:16 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/28 12:18:16 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/28 12:18:16 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/28 12:18:16 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/28 12:18:17 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/28 12:18:17 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/28 12:18:22 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/28 12:18:22 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/28 12:18:22 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/28 12:18:23 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/28 12:18:23 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/28 12:18:23 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/28 12:18:23 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/28 12:18:23 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/28 12:18:24 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/28 12:18:25 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:25 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu May 28 12:18:25 2015 2015/05/28 12:18:26 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/28 12:18:26 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:18:26 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:18:26 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:18:26 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/28 12:18:26 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/28 12:18:26 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/28 12:18:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/28 12:18:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/28 12:18:27 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/28 12:18:27 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/28 12:18:27 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/CLEAN 2015/05/28 12:18:27 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/28 12:18:27 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/28 12:18:27 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/28 12:18:27 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/28 12:18:27 VCS INFO V-16-1-10304 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (First probe) 2015/05/28 12:18:27 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/28 12:18:27 VCS INFO V-16-1-50007 Initiating auto-start online of group VVRService 2015/05/28 12:18:27 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group VVRService 2015/05/28 12:18:27 VCS INFO V-16-1-50023 Parent group AppService can not come online on system MILWB03S; hence child group VVRService may not come online on system MILWB03S 2015/05/28 12:18:27 VCS INFO V-16-1-50025 Searching for a feasible system. System MILWB03S will be considered for bringing the group VVRService online even if it is not suitable for a parent group to come online. 2015/05/28 12:18:27 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/28 12:18:27 VCS NOTICE V-16-1-10301 Initiating Online of Resource datarvg (Owner: Unspecified, Group: VVRService) on System MILWB03S 2015/05/28 12:18:28 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/28 12:18:28 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/28 12:18:28 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/28 12:18:28 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/28 12:18:29 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/28 12:18:29 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (online) ============================================== VxVM VVR vxrvg INFO V-5-1-3229 Rvg datarvg is already recovered VxVM VVR vxrlink INFO V-5-1-3534 Rlink datarlk is already recovered ============================================== 2015/05/28 12:18:29 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:18:30 VCS INFO V-16-1-10298 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (VCS initiated) 2015/05/28 12:18:30 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/28 12:18:30 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/28 12:18:30 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/28 12:18:30 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/28 12:18:30 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:18:30 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:18:30 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:18:30 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:18:30 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:18:30 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:18:31 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:18:31 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/28 12:18:31 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/28 12:18:32 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/28 12:18:32 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/28 12:18:32 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/28 12:56:40 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:56:41 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:56:43 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:56:44 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:56:48 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 12:56:48 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:01:17 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/05/28 13:01:17 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/05/28 13:10:22 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/28 13:10:22 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:10:22 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/28 13:10:22 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/28 13:11:00 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/05/28 13:11:00 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/05/28 13:21:15 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/28 13:21:15 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:21:15 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/28 13:21:15 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/28 13:23:34 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:23:36 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:24:03 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:24:14 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:24:30 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:24:30 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:24:30 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:24:32 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:24:32 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:25:00 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:25:00 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:25:12 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 13:25:12 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:47:26 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:47:47 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:47:49 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:47:50 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:47:55 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:47:55 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:48:25 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:48:26 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:48:48 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:49:00 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:49:23 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:49:23 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:54:54 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:56:12 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 14:56:12 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:27:33 VCS ERROR V-16-1-50981 Resource appNIC is faulted on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:32:43 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:33:44 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:33:45 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:33:46 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:33:51 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:33:51 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/28 15:33:51 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/05/28 15:33:51 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/28 15:33:51 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 15:33:51 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/28 15:33:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 15:33:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:33:51 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/28 15:34:00 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/28 15:34:02 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 15:34:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:34:04 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 15:34:04 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:34:06 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/28 15:34:06 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/28 15:34:06 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/28 15:34:06 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/28 15:34:06 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/28 15:34:06 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/28 15:34:34 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:34:34 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:34:37 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/28 15:34:38 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/3498/psinfo 2015/05/28 15:34:38 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 15:34:38 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:34:49 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/12568/psinfo 2015/05/28 15:34:49 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 15:34:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:35:00 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 15:35:01 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9733 3498 0 15:34:31 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 9684 3498 0 15:34:31 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 3498 3495 2 15:34:06 ? 0:27 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9779 3498 1 15:34:31 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 15-05-28 15:34:50 || haStart:sybase is starting, break! 15-05-28 15:34:50 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 15:35:00 || haStart:The imap is running, break! 15-05-28 15:35:00 || haStart:Success to start the nms! ============================================== 2015/05/28 15:35:30 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 15:35:30 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/28 15:35:59 VCS WARNING V-16-1-50086 CPU usage on MILWB03S is 89% 2015/05/28 15:43:21 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/05/28 15:43:21 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/05/28 15:43:21 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/05/28 15:43:21 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/28 15:43:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:44:23 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/28 15:44:24 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopping... EmfGnlDevDm.........................stopped EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopped TrapReceiver........................stopping... Unite ============================================== 2015/05/28 15:44:27 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 15:44:27 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:44:27 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/28 15:44:28 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/28 15:44:28 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/28 15:44:28 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 15:44:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:44:30 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/28 15:44:31 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/28 15:44:31 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/28 15:44:31 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 15:44:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:44:32 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 15:44:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 15:44:34 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/28 15:44:37 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/28 15:44:37 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/28 15:44:37 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/28 15:44:37 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/28 15:44:37 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/28 15:44:37 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/28 15:44:37 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 15:44:37 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/28 15:44:37 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 15:44:48 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:44:50 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:45:12 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:45:23 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:45:46 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 15:45:46 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:14:37 VCS ERROR V-16-1-50981 Resource appNIC is faulted on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:14:59 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/28 16:19:47 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:24:49 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:27:10 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:27:11 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:27:16 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:27:16 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/28 16:27:16 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/05/28 16:27:16 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/28 16:27:16 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/28 16:27:16 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/28 16:27:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/28 16:27:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 16:27:16 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/28 16:27:24 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/28 16:27:26 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 16:27:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 16:27:28 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 16:27:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 16:27:28 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/28 16:27:28 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/28 16:27:28 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/28 16:27:28 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/28 16:27:28 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/28 16:27:28 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/28 16:27:56 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/28 16:27:57 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/876/psinfo 2015/05/28 16:27:57 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 16:27:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 16:28:08 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/8345/psinfo 2015/05/28 16:28:08 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 16:28:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/28 16:28:19 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/28 16:28:19 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 5858 876 0 16:27:50 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 876 875 2 16:27:29 ? 0:26 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 5907 876 1 16:27:50 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 5811 876 0 16:27:50 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 15-05-28 16:28:08 || haStart:sybase is starting, break! 15-05-28 16:28:08 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-28 16:28:19 || haStart:The imap is running, break! 15-05-28 16:28:19 || haStart:Success to start the nms! ============================================== 2015/05/28 16:28:34 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/28 16:28:34 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/28 16:28:59 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 100% 2015/05/28 16:44:38 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/28 16:44:38 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/28 20:15:00 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 00:15:01 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 04:15:02 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 07:20:01 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/05/29 07:20:01 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/05/29 07:20:01 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/29 07:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 07:21:03 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 07:21:03 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopping... LogService..........................stopped Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopping... TrapReceiver........................stopping... UniteUitl ============================================== 2015/05/29 07:21:05 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 07:21:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 07:21:05 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 07:21:06 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 07:21:06 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 07:21:06 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 07:21:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 07:21:07 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 07:21:07 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 07:21:07 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 07:21:07 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 07:21:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 07:21:09 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 07:21:09 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 07:21:11 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 07:21:13 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 07:21:13 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 07:21:13 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/29 07:21:13 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 07:21:13 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 07:21:13 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 07:21:13 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 07:21:13 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 07:21:13 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 07:21:40 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 07:21:41 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 07:22:03 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 07:22:15 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 07:22:39 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 07:22:39 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 08:15:03 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 09:15:34 VCS ERROR V-16-3-18313 (MILWB03S) TCP/IP connection via connector from cluster MILWB03SCluster to cluster MILWB02SCluster is hung; intentionally disconnecting. Auto-reconnect will occur however you may wish to examine the wac resource on system 10.66.28.53 2015/05/29 09:15:34 VCS ERROR V-16-3-18311 (MILWB03S) MILWB03SCluster lost connection to cluster MILWB02SCluster 2015/05/29 09:22:48 VCS ERROR V-16-3-18211 (MILWB03S) Cluster MILWB03SCluster lost heartbeat Icmp to cluster MILWB02SCluster 2015/05/29 09:22:48 VCS NOTICE V-16-3-18213 (MILWB03S) Cluster MILWB03SCluster lost all heartbeats to cluster MILWB02SCluster: effecting inquiry 2015/05/29 09:22:48 VCS ERROR V-16-3-18314 (MILWB03S) Cluster MILWB03SCluster lost all heartbeats to cluster MILWB02SCluster: no inquiry peers 2015/05/29 09:22:48 VCS INFO V-16-1-50522 Comparing ClusterList values for group AppService between clusters MILWB02SCluster and MILWB03SCluster 2015/05/29 09:22:48 VCS ERROR V-16-1-50908 Remote cluster MILWB02SCluster has faulted. Determining if global group AppService should be failed over to local cluster 2015/05/29 09:22:48 VCS INFO V-16-1-50903 Evaluating remote cluster MILWB02SCluster as potential target for group AppService 2015/05/29 09:22:48 VCS INFO V-16-1-50904 Cluster MILWB02SCluster is not in RUNNING state. Skipping... 2015/05/29 09:22:48 VCS INFO V-16-1-50899 Evaluating local cluster MILWB03SCluster as potential target for group AppService 2015/05/29 09:22:48 VCS INFO V-16-1-50902 Found a possible target system in local cluster 2015/05/29 09:22:48 VCS ERROR V-16-1-50910 Local cluster was determined to be the next suitable failover target for Group AppService. Initiating online of group 2015/05/29 09:22:48 VCS CRITICAL V-16-1-50513 Remote cluster 'MILWB02SCluster' has faulted 2015/05/29 09:22:48 VCS ERROR V-16-5-51071 ALERT[CFAULT-MILWB02SCluster-MILWB02SCluster] Remote cluster MILWB02SCluster has faulted. Administrative action is required. 2015/05/29 09:22:48 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 09:22:48 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 09:22:48 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/29 09:22:48 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 09:22:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:22:48 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 09:22:48 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:_p_p_mode = AS 2015/05/29 09:22:48 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:It is a AS mode. 2015/05/29 09:22:48 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:touch /opt/HWENGR/NMSApp/flag/To_Stop. 2015/05/29 09:22:48 VCS INFO V-16-20010-80 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg takeover successful; use vxrvg resync command to resynchronize the original primary upon its restoration 2015/05/29 09:22:50 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 09:22:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:22:51 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 09:22:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:22:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 09:22:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 09:22:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 09:22:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 09:22:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/29 09:22:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 09:23:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 09:23:24 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/9273/psinfo 2015/05/29 09:23:24 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 09:23:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:23:35 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/17632/psinfo 2015/05/29 09:23:35 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 09:23:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:23:46 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 09:23:47 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 9273 9271 2 09:22:52 ? 0:28 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 14253 9273 1 09:23:14 ? 0:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 14158 9273 0 09:23:14 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 14207 9273 0 09:23:14 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 15-05-29 09:23:35 || haStart:sybase is starting, break! 15-05-29 09:23:35 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 09:23:45 || haStart:The imap is running, break! 15-05-29 09:23:46 || haStart:Success to start the nms! ============================================== 2015/05/29 09:24:05 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 09:24:05 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/29 09:24:29 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 99% 2015/05/29 09:34:09 VCS NOTICE V-16-3-18321 (MILWB03S) All heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are down. will not accept connection from cluster (MILWB02SCluster) until at least one heartbeat to cluster (MILWB02SCluster) is alive 2015/05/29 09:34:27 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/29 09:34:33 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/29 09:34:33 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 09:34:33 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/29 09:34:33 VCS NOTICE V-16-1-50840 ALERT[CFAULT-MILWB02SCluster-MILWB02SCluster] was deleted automatically by the engine because alert no longer exists on cluster MILWB02SCluster 2015/05/29 09:34:33 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 09:34:33 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/29 09:34:47 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MILWB03S localclus from localhost 2015/05/29 09:34:47 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/29 09:34:47 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 09:34:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:35:51 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 09:35:51 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopped LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping... UniteUitl ============================================== 2015/05/29 09:35:54 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 09:35:54 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 09:35:54 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:35:54 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 09:35:55 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 09:35:55 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 09:35:55 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 09:35:55 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:35:57 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 09:35:58 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 09:35:58 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 09:35:58 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 09:35:58 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:35:59 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 09:35:59 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 09:36:01 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 09:36:04 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 09:36:04 VCS INFO V-16-1-50918 Local cluster MILWB03SCluster and remote cluster MILWB02SCluster have Authority for group AppService. Giving up Authority since group is online on remote cluster and offline in local cluster 2015/05/29 09:36:04 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 09:36:04 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 09:36:04 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/29 10:08:57 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:08:58 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/29 10:09:17 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/29 10:09:17 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:09:17 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:09:17 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:09:18 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:09:19 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:09:33 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 10:09:53 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:09:53 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:09:53 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:09:54 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:09:54 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:09:57 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:10:11 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:10:12 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/29 10:10:30 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/29 10:10:30 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:10:30 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:10:30 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:10:32 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:10:32 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 10:10:46 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 10:11:14 VCS ERROR V-16-2-13067 (MILWB03S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/05/29 10:11:14 VCS WARNING V-16-20010-301 (MILWB03S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/05/29 10:11:14 VCS ERROR V-16-20010-302 (MILWB03S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/05/29 10:11:15 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/05/29 10:11:15 VCS ERROR V-16-2-13069 (MILWB03S) Resource(datarvg) - clean failed. 2015/05/29 10:12:10 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:12:11 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/29 10:12:11 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 10:12:15 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 10:12:17 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB02SCluster 0 MILWB03SCluster 1 from localhost 2015/05/29 10:12:17 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 10:12:20 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:20 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:20 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:20 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:20 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:20 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:20 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:41 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:41 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:41 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:42 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:42 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 10:12:45 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:16:42 VCS ERROR V-16-1-50981 Resource DatabaseServer is faulted on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:18:47 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:19:10 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:19:11 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:19:15 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:19:15 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/29 11:19:15 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/05/29 11:19:15 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 11:19:15 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:19:15 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/29 11:19:15 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:19:15 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:19:16 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 11:19:39 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/29 11:19:42 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:19:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:19:43 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:19:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:19:44 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 11:19:44 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 11:19:44 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 11:19:44 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 11:19:44 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/29 11:19:44 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 11:20:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 11:20:24 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/15407/psinfo 2015/05/29 11:20:24 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:20:24 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:20:35 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/27095/psinfo 2015/05/29 11:20:35 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:20:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:20:46 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 11:20:46 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 15407 15405 2 11:19:44 ? 0:28 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 20245 15407 0 11:20:06 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 20294 15407 1 11:20:06 ? 0:08 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 20194 15407 2 11:20:06 ? 0:14 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 15-05-29 11:20:35 || haStart:sybase is starting, break! 15-05-29 11:20:35 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 11:20:45 || haStart:The imap is running, break! 15-05-29 11:20:46 || haStart:Success to start the nms! ============================================== 2015/05/29 11:20:58 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:20:58 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/29 11:21:29 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 99% 2015/05/29 11:28:27 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:28:27 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:31:21 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/05/29 11:31:21 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/05/29 11:31:21 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/29 11:31:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:32:23 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 11:32:23 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopped Nml_sdh.............................stopping... Nml_vlan............................stopped NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopped PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... U ============================================== 2015/05/29 11:32:25 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:32:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:32:26 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 11:32:26 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 11:32:26 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 11:32:26 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:32:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:32:29 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 11:32:30 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 11:32:30 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 11:32:30 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:32:30 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:32:31 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:32:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:32:33 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 11:32:35 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:32:35 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 11:32:35 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/29 11:32:36 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 11:32:36 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 11:32:36 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 11:32:36 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:32:36 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 11:32:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:32:45 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:32:46 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:33:14 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:33:25 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:33:50 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:33:50 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:42:46 VCS ERROR V-16-1-50981 Resource NMSServer is faulted on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:42:47 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:42:50 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:42:51 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:42:55 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:42:55 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/29 11:42:55 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/05/29 11:42:55 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 11:42:55 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:42:55 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/29 11:42:55 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:42:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:42:56 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 11:43:19 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/29 11:43:21 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:43:21 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:43:22 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:43:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:43:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 11:43:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 11:43:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 11:43:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 11:43:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/29 11:43:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 11:43:50 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 11:43:52 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/12938/psinfo 2015/05/29 11:43:52 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:43:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:44:02 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/20101/psinfo 2015/05/29 11:44:02 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:44:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:44:14 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 11:44:14 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 17749 12938 1 11:43:45 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 17651 12938 0 11:43:45 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 17700 12938 0 11:43:45 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 12938 12936 2 11:43:23 ? 0:25 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-29 11:44:03 || haStart:sybase is starting, break! 15-05-29 11:44:03 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 11:44:13 || haStart:The imap is running, break! 15-05-29 11:44:13 || haStart:Success to start the nms! ============================================== 2015/05/29 11:44:29 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 11:44:29 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/29 11:44:59 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 99% 2015/05/29 11:51:11 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:51:11 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:54:15 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/05/29 11:54:15 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/05/29 11:54:15 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/29 11:54:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:55:20 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 11:55:20 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopped Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopped Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopped PMSDm...............................stopping... PMSWeb..............................stopped SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping... UniteUitlDM.. ============================================== 2015/05/29 11:55:22 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:55:22 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:55:23 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 11:55:24 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 11:55:24 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 11:55:24 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:55:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:55:26 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 11:55:27 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 11:55:27 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 11:55:27 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:55:27 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:55:28 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:55:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 11:55:30 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 11:55:33 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 11:55:33 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 11:55:33 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/29 11:55:33 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 11:55:33 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 11:55:33 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 11:55:33 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 11:55:33 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 11:55:33 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 11:55:42 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:55:44 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:56:06 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:56:16 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:56:39 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 11:56:40 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:15:04 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 12:45:18 VCS ERROR V-16-1-50981 Resource BackupServer is faulted on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:46:22 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:46:23 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:46:24 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:46:29 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:46:29 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/29 12:46:29 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/05/29 12:46:29 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 12:46:29 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 12:46:29 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/05/29 12:46:29 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 12:46:29 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 12:46:29 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 12:46:54 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/05/29 12:46:55 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 12:46:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 12:46:57 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 12:46:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 12:46:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 12:46:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 12:46:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 12:46:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 12:46:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/29 12:46:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 12:47:26 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 12:47:26 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/10750/psinfo 2015/05/29 12:47:26 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 12:47:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 12:47:37 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/18177/psinfo 2015/05/29 12:47:37 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 12:47:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 12:47:47 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 12:47:48 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 15790 10750 1 12:47:20 ? 0:04 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 15744 10750 0 12:47:20 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 10750 10748 2 12:46:57 ? 0:25 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 15695 10750 0 12:47:20 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 15-05-29 12:47:37 || haStart:sybase is starting, break! 15-05-29 12:47:37 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 12:47:47 || haStart:The imap is running, break! 15-05-29 12:47:47 || haStart:Success to start the nms! ============================================== 2015/05/29 12:48:01 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 12:48:01 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/29 12:48:29 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 100% 2015/05/29 12:53:00 VCS INFO V-16-1-50135 User root fired command: hares -clear BackupServer MILWB02S from localhost 2015/05/29 12:53:15 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:53:15 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 12:59:22 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/05/29 12:59:22 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/05/29 12:59:22 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/05/29 12:59:22 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/29 12:59:22 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 13:00:24 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/05/29 13:00:25 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopping... InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopped Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopped Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopping... TrapReceiver........................stopping... U ============================================== 2015/05/29 13:00:27 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 13:00:27 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 13:00:28 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 13:00:29 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 13:00:29 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 13:00:29 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 13:00:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 13:00:31 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 13:00:32 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 13:00:32 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 13:00:32 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 13:00:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 13:00:33 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 13:00:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 13:00:35 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 13:00:38 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 13:00:38 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/05/29 13:00:38 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/29 13:00:38 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 13:00:38 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/05/29 13:00:38 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 13:00:38 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 13:00:38 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/05/29 13:00:38 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 13:00:48 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:00:50 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:01:12 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:01:23 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:01:47 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:01:47 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:35 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/05/29 13:24:35 VCS INFO V-16-1-10196 Cluster logger started 2015/05/29 13:24:35 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/05/29 13:24:35 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/05/29 13:24:35 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/05/29 13:24:35 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/05/29 13:24:35 VCS NOTICE V-16-1-10115 Using GABSIM 2015/05/29 13:24:35 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/05/29 13:24:41 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/05/29 13:24:41 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/05/29 13:24:41 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/05/29 13:24:42 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/05/29 13:24:42 VCS INFO V-16-1-10077 Received new cluster membership 2015/05/29 13:24:42 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/05/29 13:24:42 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/05/29 13:24:42 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/05/29 13:24:43 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/05/29 13:24:44 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:44 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Fri May 29 13:24:44 2015 2015/05/29 13:24:47 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/05/29 13:24:47 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/29 13:24:47 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/29 13:24:47 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/29 13:24:47 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/05/29 13:24:47 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/05/29 13:24:47 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/05/29 13:24:47 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/05/29 13:24:47 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/05/29 13:24:49 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/05/29 13:24:49 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/05/29 13:24:49 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/05/29 13:24:49 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin PWD=/var/VRTSvcs/diag/had SMF_FMRI=svc:/system/vcs-onenode:default SMF_METHOD=/lib/svc/method/vcs start SMF_RESTARTER=svc:/system/svc/restarter:default TZ=Europe/Rome VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger 2015/05/29 13:24:49 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/05/29 13:24:50 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/05/29 13:24:50 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/05/29 13:24:50 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/05/29 13:24:50 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/05/29 13:24:50 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/05/29 13:24:50 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (First probe) 2015/05/29 13:24:50 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/05/29 13:24:50 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/05/29 13:24:50 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/05/29 13:24:51 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/29 13:24:51 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/29 13:24:52 VCS INFO V-16-1-10304 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (First probe) 2015/05/29 13:24:52 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/05/29 13:24:52 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/05/29 13:24:54 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/29 13:24:54 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 13:24:55 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/29 13:24:55 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/29 13:31:58 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/05/29 13:31:58 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/05/29 13:42:09 VCS ERROR V-16-3-18211 (MILWB03S) Cluster MILWB03SCluster lost heartbeat Icmp to cluster MILWB02SCluster 2015/05/29 14:20:50 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.70 2015/05/29 14:30:17 VCS NOTICE V-16-3-18321 (MILWB03S) All heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are down. will not accept connection from cluster (MILWB02SCluster) until at least one heartbeat to cluster (MILWB02SCluster) is alive 2015/05/29 14:30:48 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/29 14:30:53 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/29 14:30:53 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:30:53 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/29 14:30:53 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/29 14:33:31 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:33:31 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:33:31 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:33:32 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:33:33 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:33:33 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:33:33 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:34:11 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:34:21 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:35:11 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 14:35:11 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:34:28 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/05/29 15:34:28 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/05/29 15:36:26 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB03S from localhost 2015/05/29 15:36:45 VCS INFO V-16-1-50135 User root fired command: hagrp -online -force AppService MILWB03S from localhost 2015/05/29 15:36:45 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 15:36:45 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 15:36:45 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/05/29 15:36:45 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 15:36:45 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:36:45 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/05/29 15:36:45 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:_p_p_mode = AS 2015/05/29 15:36:45 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:It is a AS mode. 2015/05/29 15:36:45 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:touch /opt/HWENGR/NMSApp/flag/To_Stop. 2015/05/29 15:36:45 VCS INFO V-16-20010-80 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg takeover successful; use vxrvg resync command to resynchronize the original primary upon its restoration 2015/05/29 15:36:47 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 15:36:47 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:36:49 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 15:36:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:36:51 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 15:36:51 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 15:36:51 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 15:36:51 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 15:36:51 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/29 15:36:51 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 15:37:24 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 15:37:25 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/24587/psinfo 2015/05/29 15:37:25 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 15:37:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:37:35 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/4815/psinfo 2015/05/29 15:37:35 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 15:37:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:37:47 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 15:37:47 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 1127 24587 1 15:37:16 ? 0:07 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 1078 24587 0 15:37:16 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 1030 24587 0 15:37:16 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 24587 24584 2 15:36:51 ? 0:27 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-29 15:37:36 || haStart:sybase is starting, break! 15-05-29 15:37:37 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 15:37:47 || haStart:The imap is running, break! 15-05-29 15:37:47 || haStart:Success to start the nms! ============================================== 2015/05/29 15:38:17 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 15:38:17 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/29 15:38:50 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 92% 2015/05/29 15:44:09 VCS ERROR V-16-3-18211 (MILWB03S) Cluster MILWB03SCluster lost heartbeat Icmp to cluster MILWB02SCluster 2015/05/29 15:46:14 VCS NOTICE V-16-3-18321 (MILWB03S) All heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are down. will not accept connection from cluster (MILWB02SCluster) until at least one heartbeat to cluster (MILWB02SCluster) is alive 2015/05/29 15:46:48 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/05/29 15:46:50 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/05/29 15:46:50 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/05/29 15:46:50 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/05/29 15:46:50 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:46:50 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:46:50 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:46:51 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:46:51 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/05/29 15:46:51 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 15:46:51 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/05/29 15:49:30 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 15:49:30 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:49:30 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:49:30 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:49:31 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:49:31 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:49:31 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:49:32 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:50:00 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:50:11 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:50:58 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:50:58 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 15:50:58 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 15:51:49 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MILWB03S localclus from localhost 2015/05/29 15:51:49 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/05/29 15:51:49 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 15:51:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:56:51 VCS INFO V-16-2-13003 (MILWB03S) Resource(NMSServer): Output of the timed out operation (offline) Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopped Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopped TrapReceiver........................stopping... UniteUitlDM...... 2015/05/29 15:56:51 VCS WARNING V-16-2-13011 (MILWB03S) Resource(NMSServer): offline procedure did not complete within the expected time. 2015/05/29 15:56:51 VCS ERROR V-16-2-13063 (MILWB03S) Agent is calling clean for resource(NMSServer) because offline did not complete within the expected time. 2015/05/29 15:56:56 VCS INFO V-16-2-13068 (MILWB03S) Resource(NMSServer) - clean completed successfully. 2015/05/29 15:56:58 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 15:56:58 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/05/29 15:56:58 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:56:58 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/05/29 15:56:59 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/05/29 15:56:59 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/05/29 15:56:59 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 15:56:59 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:57:00 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/05/29 15:57:00 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/05/29 15:57:00 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/05/29 15:57:00 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 15:57:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:57:02 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 15:57:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 15:57:04 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/05/29 15:57:06 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/05/29 15:57:06 VCS INFO V-16-1-50918 Local cluster MILWB03SCluster and remote cluster MILWB02SCluster have Authority for group AppService. Giving up Authority since group is online on remote cluster and offline in local cluster 2015/05/29 15:57:06 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/05/29 15:57:06 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 15:57:06 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/05/29 16:22:04 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:22:04 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:22:04 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:22:05 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:22:05 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:22:08 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:23:25 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:23:26 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:23:28 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:23:29 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:23:33 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:23:33 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:23:43 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:23:44 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/29 16:24:02 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/29 16:24:02 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 16:24:02 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 16:24:02 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 16:24:03 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 16:24:04 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/05/29 16:24:18 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB03S localclus from localhost 2015/05/29 16:24:18 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/05/29 16:24:18 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB03S 2015/05/29 16:24:18 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/05/29 16:24:18 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/05/29 16:24:18 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/05/29 16:24:18 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/05/29 16:24:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 16:24:19 VCS INFO V-16-20010-72 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/05/29 16:24:19 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 16:24:21 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 16:24:21 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 16:24:22 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 16:24:22 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 16:24:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/05/29 16:24:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/05/29 16:24:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/05/29 16:24:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/05/29 16:24:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/05/29 16:24:23 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/05/29 16:24:51 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/05/29 16:24:51 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/28267/psinfo 2015/05/29 16:24:51 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 16:24:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 16:25:02 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/5783/psinfo 2015/05/29 16:25:02 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 16:25:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/05/29 16:25:13 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/05/29 16:25:13 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 3176 28267 0 16:24:45 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 3127 28267 0 16:24:45 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 3224 28267 1 16:24:45 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 28267 28265 3 16:24:23 ? 0:27 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-05-29 16:25:03 || haStart:sybase is starting, break! 15-05-29 16:25:03 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-05-29 16:25:13 || haStart:The imap is running, break! 15-05-29 16:25:13 || haStart:Success to start the nms! ============================================== 2015/05/29 16:25:15 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:25:15 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:25:15 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:25:16 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:25:16 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:25:17 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:25:25 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/05/29 16:25:25 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/05/29 16:25:50 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 100% 2015/05/29 16:26:58 VCS NOTICE V-16-1-50517 Resource NMSServer has been deleted from cluster MILWB02SCluster 2015/05/29 16:26:58 VCS NOTICE V-16-1-50517 Resource DataFilesystem has been deleted from cluster MILWB02SCluster 2015/05/29 16:26:58 VCS NOTICE V-16-1-50517 Resource appNIC has been deleted from cluster MILWB02SCluster 2015/05/29 16:26:58 VCS NOTICE V-16-1-50517 Resource RVGPrimary has been deleted from cluster MILWB02SCluster 2015/05/29 16:26:58 VCS NOTICE V-16-1-50517 Resource DatabaseServer has been deleted from cluster MILWB02SCluster 2015/05/29 16:26:58 VCS NOTICE V-16-1-50517 Resource BackupServer has been deleted from cluster MILWB02SCluster 2015/05/29 16:26:58 VCS NOTICE V-16-1-50518 Group AppService has been deleted from cluster MILWB02SCluster 2015/05/29 16:26:58 VCS INFO V-16-1-50957 Updating ClusterList for group AppService in local cluster configuration to reflect group deletion on cluster MILWB02SCluster 2015/05/29 16:26:58 VCS INFO V-16-1-50958 Local cluster dumping configuration to save ClusterList modification for group AppService 2015/05/29 16:26:59 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:27:00 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/05/29 16:27:00 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 16:27:04 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:27:04 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB03SCluster and remote cluster MILWB02SCluster 2015/05/29 16:27:04 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:27:04 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:27:04 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:27:04 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:27:04 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:27:04 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/05/29 16:27:04 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB03SCluster and remote cluster MILWB02SCluster 2015/05/29 16:27:06 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:27:07 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB02SCluster 0 MILWB03SCluster 1 from localhost 2015/05/29 16:27:07 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 16:27:08 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/05/29 16:27:09 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/05/29 16:27:28 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/05/29 16:27:44 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/05/29 17:21:17 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/29 21:21:18 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 01:21:19 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 05:21:20 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 09:21:21 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 13:21:22 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 17:21:23 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/30 21:21:24 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 01:21:25 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 05:21:26 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 09:21:26 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 13:21:27 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 17:21:28 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/05/31 21:21:29 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 01:21:30 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 05:21:31 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 08:39:34 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/06/01 08:39:34 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/06/01 08:39:34 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/06/01 08:39:34 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/01 08:40:39 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/01 08:40:39 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopping... InventoryDM.........................stopped ItmService..........................stopped LicenseService......................stopping... LogService..........................stopped Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopped Nml_vlan............................stopped NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver........................stopping... UniteUitlDM.. ============================================== 2015/06/01 08:40:41 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/01 08:40:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/01 08:40:41 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/01 08:40:42 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/01 08:40:42 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/01 08:40:42 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/01 08:40:42 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/01 08:40:43 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/01 08:40:43 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/01 08:40:44 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/01 08:40:44 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/01 08:40:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/01 08:40:45 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/01 08:40:45 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/01 08:40:47 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/01 08:40:49 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/01 08:40:49 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/01 08:40:49 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/06/01 08:40:49 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/01 08:40:49 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/01 08:40:49 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/01 08:40:49 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/01 08:40:49 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/01 08:40:49 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/01 08:41:00 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/01 08:41:01 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/01 08:41:25 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/01 08:41:37 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/01 08:42:00 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/01 08:42:00 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/06/01 09:21:32 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 13:21:33 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 17:21:34 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/01 21:21:35 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 01:21:36 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 05:21:37 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 09:21:38 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 13:21:39 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 17:21:40 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/02 21:21:41 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 01:21:42 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 05:21:43 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 09:21:44 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 10:03:31 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/06/03 10:03:31 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/06/03 10:13:07 VCS ERROR V-16-3-18211 (MILWB03S) Cluster MILWB03SCluster lost heartbeat Icmp to cluster MILWB02SCluster 2015/06/03 10:36:37 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB03S from localhost 2015/06/03 10:44:07 VCS INFO V-16-1-50135 User root fired command: hagrp -online -force AppService MILWB03S from localhost 2015/06/03 10:44:07 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/03 10:44:07 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/03 10:44:07 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/06/03 10:44:07 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/03 10:44:07 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 10:44:07 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/03 10:44:07 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:_p_p_mode = AS 2015/06/03 10:44:08 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:It is a AS mode. 2015/06/03 10:44:08 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:touch /opt/HWENGR/NMSApp/flag/To_Stop. 2015/06/03 10:44:08 VCS INFO V-16-20010-80 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg takeover successful; use vxrvg resync command to resynchronize the original primary upon its restoration 2015/06/03 10:44:10 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/03 10:44:10 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 10:44:11 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/03 10:44:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 10:44:12 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/03 10:44:12 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/03 10:44:12 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/03 10:44:12 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/03 10:44:12 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/06/03 10:44:12 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/03 10:44:45 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/03 10:44:46 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/28428/psinfo 2015/06/03 10:44:46 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/03 10:44:46 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 10:44:57 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/7925/psinfo 2015/06/03 10:44:57 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/03 10:44:57 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 10:45:08 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/03 10:45:08 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 3915 28428 0 10:44:35 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 28428 28427 2 10:44:12 ? 0:29 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 4019 28428 1 10:44:35 ? 0:06 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 3967 28428 1 10:44:35 ? 0:02 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 15-06-03 10:44:58 || haStart:sybase is starting, break! 15-06-03 10:44:58 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-03 10:45:08 || haStart:The imap is running, break! 15-06-03 10:45:08 || haStart:Success to start the nms! ============================================== 2015/06/03 10:45:22 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/03 10:45:22 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/06/03 10:45:49 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 100% 2015/06/03 10:54:03 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 10:54:12 VCS INFO V-16-1-50135 User root fired command: hagrp -freeze -persistent AppService from localhost 2015/06/03 10:54:21 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:06:16 VCS NOTICE V-16-3-18321 (MILWB03S) All heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are down. will not accept connection from cluster (MILWB02SCluster) until at least one heartbeat to cluster (MILWB02SCluster) is alive 2015/06/03 11:06:47 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/06/03 11:06:51 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/06/03 11:06:51 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/06/03 11:06:51 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/06/03 11:06:51 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:06:51 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:06:51 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:06:51 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:06:51 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:06:51 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:06:51 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/06/03 11:06:51 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/03 11:06:51 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/06/03 11:09:30 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/03 11:09:30 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:09:30 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:09:30 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:09:31 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:09:31 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:09:31 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:09:32 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:10:02 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:10:13 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:10:58 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:10:58 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/03 11:10:58 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:11:09 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MILWB03S localclus from localhost 2015/06/03 11:25:56 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:26:04 VCS INFO V-16-1-50135 User root fired command: hagrp -unfreeze -persistent AppService from localhost 2015/06/03 11:26:11 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:33:50 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MILWB02S from localhost 2015/06/03 11:34:55 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:34:56 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:34:58 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:34:59 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:35:03 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:35:03 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/03 11:35:03 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:43:24 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:43:24 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:43:24 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:43:25 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:43:25 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:43:26 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:43:40 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:43:42 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/06/03 11:44:00 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/06/03 11:44:17 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:44:38 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:44:38 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:44:38 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:44:39 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:44:39 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:44:40 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:45:45 VCS ERROR V-16-2-13067 (MILWB03S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/06/03 11:45:45 VCS WARNING V-16-20010-301 (MILWB03S) RVG:datarvg:clean:The command 'vxrvg -g datadg stop datarvg' failed. Doing a forced stop 2015/06/03 11:45:45 VCS ERROR V-16-20010-302 (MILWB03S) RVG:datarvg:clean:The forced stop of RVG datarvg using the command 'vxrvg -g datadg -f stop datarvg' failed 2015/06/03 11:45:46 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration VxVM VVR vxrvg ERROR V-5-1-639 Failed to obtain locks: datarvg: no such object in the configuration ============================================== 2015/06/03 11:45:46 VCS ERROR V-16-2-13069 (MILWB03S) Resource(datarvg) - clean failed. 2015/06/03 11:46:21 VCS NOTICE V-16-1-50517 Resource NMSServer has been deleted from cluster MILWB02SCluster 2015/06/03 11:46:21 VCS NOTICE V-16-1-50517 Resource DataFilesystem has been deleted from cluster MILWB02SCluster 2015/06/03 11:46:21 VCS NOTICE V-16-1-50517 Resource appNIC has been deleted from cluster MILWB02SCluster 2015/06/03 11:46:21 VCS NOTICE V-16-1-50517 Resource RVGPrimary has been deleted from cluster MILWB02SCluster 2015/06/03 11:46:21 VCS NOTICE V-16-1-50517 Resource DatabaseServer has been deleted from cluster MILWB02SCluster 2015/06/03 11:46:21 VCS NOTICE V-16-1-50517 Resource BackupServer has been deleted from cluster MILWB02SCluster 2015/06/03 11:46:21 VCS NOTICE V-16-1-50518 Group AppService has been deleted from cluster MILWB02SCluster 2015/06/03 11:46:21 VCS INFO V-16-1-50957 Updating ClusterList for group AppService in local cluster configuration to reflect group deletion on cluster MILWB02SCluster 2015/06/03 11:46:21 VCS INFO V-16-1-50958 Local cluster dumping configuration to save ClusterList modification for group AppService 2015/06/03 11:46:22 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:46:24 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/06/03 11:46:24 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:46:27 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:46:27 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB03SCluster and remote cluster MILWB02SCluster 2015/06/03 11:46:27 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:46:27 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:46:27 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:46:27 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:46:27 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:46:27 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/03 11:46:27 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB03SCluster and remote cluster MILWB02SCluster 2015/06/03 11:46:29 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:46:31 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB02SCluster 0 MILWB03SCluster 1 from localhost 2015/06/03 11:46:32 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 11:46:33 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/03 11:46:34 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/06/03 11:46:53 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/06/03 11:47:10 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/03 13:21:45 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 16:38:36 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/06/03 16:38:36 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/06/03 16:38:36 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/06/03 16:38:36 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/06/03 16:38:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 16:43:37 VCS INFO V-16-2-13003 (MILWB03S) Resource(NMSServer): Output of the timed out operation (offline) Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopped Nml_sdh.............................stopped Nml_vlan............................stopping... NmslogzipDm_398.....................stopped Notify_Service......................stopping... PMDataRefiner.......................stopped PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping... Unite 2015/06/03 16:43:37 VCS WARNING V-16-2-13011 (MILWB03S) Resource(NMSServer): offline procedure did not complete within the expected time. 2015/06/03 16:43:37 VCS ERROR V-16-2-13063 (MILWB03S) Agent is calling clean for resource(NMSServer) because offline did not complete within the expected time. 2015/06/03 16:43:42 VCS INFO V-16-2-13068 (MILWB03S) Resource(NMSServer) - clean completed successfully. 2015/06/03 16:43:45 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/03 16:43:45 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 16:43:45 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/03 16:43:46 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/03 16:43:46 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/03 16:43:46 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/03 16:43:46 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 16:43:47 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/03 16:43:47 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/03 16:43:47 VCS CRITICAL V-16-20018-4 (MILWB03S) Sybase:DatabaseServer:monitor:Process dataserver restarted 2015/06/03 16:43:47 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/03 16:43:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 16:43:48 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/03 16:43:48 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/03 16:43:51 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/03 16:43:53 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/03 16:43:53 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/03 16:43:53 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/06/03 16:43:53 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/03 16:43:53 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/03 16:43:53 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/03 16:43:53 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/03 16:43:53 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/03 16:43:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/03 16:44:07 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 16:44:09 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 16:44:34 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 16:44:45 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 16:45:10 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 16:45:10 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/06/03 17:21:45 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/03 21:21:46 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 01:21:47 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 05:21:48 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 09:21:49 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 10:23:24 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/06/04 10:23:24 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/06/04 10:25:52 VCS INFO V-16-1-50135 User root fired command: hagrp -online -force AppService MILWB03S from localhost 2015/06/04 10:25:52 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/04 10:25:52 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 10:25:52 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/06/04 10:25:52 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 10:25:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:25:53 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/04 10:25:53 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:_p_p_mode = AS 2015/06/04 10:25:53 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:It is a AS mode. 2015/06/04 10:25:53 VCS INFO V-16-20010-79 (MILWB03S) RVGPrimary:RVGPrimary:online:touch /opt/HWENGR/NMSApp/flag/To_Stop. 2015/06/04 10:25:53 VCS INFO V-16-20010-80 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg takeover successful; use vxrvg resync command to resynchronize the original primary upon its restoration 2015/06/04 10:25:55 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:25:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:25:56 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:25:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:25:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/04 10:25:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/04 10:25:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/04 10:25:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/04 10:25:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/06/04 10:25:57 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/04 10:26:38 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/04 10:26:39 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/26314/psinfo 2015/06/04 10:26:39 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:26:39 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:26:50 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/8706/psinfo 2015/06/04 10:26:50 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:26:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:27:01 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/04 10:27:01 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 1575 26314 1 10:26:20 ? 0:11 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 1525 26314 1 10:26:20 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 1473 26314 0 10:26:20 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 26314 26313 3 10:25:57 ? 0:40 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-06-04 10:26:50 || haStart:sybase is starting, break! 15-06-04 10:26:51 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-04 10:27:01 || haStart:The imap is running, break! 15-06-04 10:27:01 || haStart:Success to start the nms! ============================================== 2015/06/04 10:27:18 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:27:18 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/06/04 10:27:48 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 99% 2015/06/04 10:33:07 VCS ERROR V-16-3-18211 (MILWB03S) Cluster MILWB03SCluster lost heartbeat Icmp to cluster MILWB02SCluster 2015/06/04 10:41:33 VCS NOTICE V-16-3-18321 (MILWB03S) All heartbeats from cluster (MILWB03SCluster) to cluster (MILWB02SCluster) are down. will not accept connection from cluster (MILWB02SCluster) until at least one heartbeat to cluster (MILWB02SCluster) is alive 2015/06/04 10:41:47 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/06/04 10:41:51 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/06/04 10:41:51 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/06/04 10:41:51 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/06/04 10:41:51 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:41:51 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:41:51 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:41:51 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:41:51 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:41:51 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:41:51 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/06/04 10:41:51 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/04 10:41:51 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/06/04 10:44:48 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/04 10:44:48 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:44:48 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:44:48 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:44:49 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:44:49 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:44:49 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:44:50 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:45:23 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:45:35 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:46:24 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:46:24 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/04 10:46:24 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:47:01 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MILWB03S localclus from localhost 2015/06/04 10:47:01 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/06/04 10:47:01 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/04 10:47:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:52:04 VCS INFO V-16-2-13003 (MILWB03S) Resource(NMSServer): Output of the timed out operation (offline) Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopped NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopped PMSWeb..............................stopped SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopped SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping... UniteUitl 2015/06/04 10:52:04 VCS WARNING V-16-2-13011 (MILWB03S) Resource(NMSServer): offline procedure did not complete within the expected time. 2015/06/04 10:52:04 VCS ERROR V-16-2-13063 (MILWB03S) Agent is calling clean for resource(NMSServer) because offline did not complete within the expected time. 2015/06/04 10:52:09 VCS INFO V-16-2-13068 (MILWB03S) Resource(NMSServer) - clean completed successfully. 2015/06/04 10:52:11 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 10:52:11 VCS ERROR V-16-1-50921 CONCURRENCY VIOLATION:Group AppService is online on the following clusters [MILWB02SCluster, MILWB03SCluster] 2015/06/04 10:52:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:52:11 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/04 10:52:12 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/04 10:52:12 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/04 10:52:12 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 10:52:12 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:52:13 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/04 10:52:13 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/04 10:52:13 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/04 10:52:13 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 10:52:13 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:52:15 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 10:52:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:52:17 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/04 10:52:19 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 10:52:19 VCS INFO V-16-1-50918 Local cluster MILWB03SCluster and remote cluster MILWB02SCluster have Authority for group AppService. Giving up Authority since group is online on remote cluster and offline in local cluster 2015/06/04 10:52:19 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/04 10:52:19 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 10:52:19 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/06/04 10:57:00 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:57:00 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:57:00 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:57:01 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:57:01 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:57:04 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:58:21 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:58:23 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:58:25 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:58:26 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:58:31 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:58:31 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 10:58:39 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 10:58:41 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/06/04 10:58:59 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/06/04 10:58:59 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/06/04 10:58:59 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/06/04 10:58:59 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/06/04 10:59:01 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/06/04 10:59:01 VCS INFO V-16-1-10306 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (Previous State = OFFLINE) 2015/06/04 10:59:15 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MILWB03S localclus from localhost 2015/06/04 10:59:16 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/06/04 10:59:16 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB03S 2015/06/04 10:59:16 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/04 10:59:16 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 10:59:16 VCS NOTICE V-16-1-10166 Initiating manual online of group AppService on system MILWB03S 2015/06/04 10:59:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 10:59:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:59:16 VCS INFO V-16-20010-72 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/06/04 10:59:16 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 10:59:18 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:59:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:59:19 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:59:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 10:59:20 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/04 10:59:20 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/04 10:59:20 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/04 10:59:20 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/04 10:59:20 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/06/04 10:59:20 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/04 10:59:48 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/04 10:59:49 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/5226/psinfo 2015/06/04 10:59:49 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 10:59:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:00:00 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/12593/psinfo 2015/06/04 11:00:00 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 11:00:00 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:00:11 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/04 11:00:11 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 10005 5226 0 10:59:42 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 5226 5224 3 10:59:20 ? 0:27 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 9957 5226 0 10:59:42 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 10052 5226 1 10:59:42 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 15-06-04 11:00:00 || haStart:sybase is starting, break! 15-06-04 11:00:00 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-04 11:00:11 || haStart:The imap is running, break! 15-06-04 11:00:11 || haStart:Success to start the nms! ============================================== 2015/06/04 11:00:13 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:00:13 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:00:13 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:00:14 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:00:14 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:00:15 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:00:24 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 11:00:24 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/06/04 11:00:48 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 100% 2015/06/04 11:01:42 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/EMPTY 2015/06/04 11:01:43 VCS ERROR V-16-2-13067 (MILWB03S) Agent is calling clean for resource(datarvg) because the resource became OFFLINE unexpectedly, on its own. 2015/06/04 11:01:44 VCS INFO V-16-2-13716 (MILWB03S) Resource(datarvg): Output of the completed operation (clean) ============================================== VxVM VVR vxrvg WARNING V-5-1-3377 Can not stop rvg datarvg. Already disabled. ============================================== 2015/06/04 11:01:44 VCS INFO V-16-2-13068 (MILWB03S) Resource(datarvg) - clean completed successfully. 2015/06/04 11:01:44 VCS WARNING V-16-20010-224 (MILWB03S) RVG:datarvg:monitor:Rvg datarvg in diskgroup datadg is not in ENABLED/ACTIVE state, RVG state is DISABLED/EMPTY 2015/06/04 11:01:45 VCS INFO V-16-1-10307 Resource datarvg (Owner: Unspecified, Group: VVRService) is offline on MILWB03S (Not initiated by VCS) 2015/06/04 11:01:45 VCS NOTICE V-16-1-10446 Group VVRService is offline on system MILWB03S 2015/06/04 11:02:03 VCS NOTICE V-16-1-50517 Resource NMSServer has been deleted from cluster MILWB02SCluster 2015/06/04 11:02:03 VCS NOTICE V-16-1-50517 Resource DataFilesystem has been deleted from cluster MILWB02SCluster 2015/06/04 11:02:03 VCS NOTICE V-16-1-50517 Resource appNIC has been deleted from cluster MILWB02SCluster 2015/06/04 11:02:03 VCS NOTICE V-16-1-50517 Resource RVGPrimary has been deleted from cluster MILWB02SCluster 2015/06/04 11:02:03 VCS NOTICE V-16-1-50517 Resource DatabaseServer has been deleted from cluster MILWB02SCluster 2015/06/04 11:02:03 VCS NOTICE V-16-1-50517 Resource BackupServer has been deleted from cluster MILWB02SCluster 2015/06/04 11:02:03 VCS NOTICE V-16-1-50518 Group AppService has been deleted from cluster MILWB02SCluster 2015/06/04 11:02:03 VCS INFO V-16-1-50957 Updating ClusterList for group AppService in local cluster configuration to reflect group deletion on cluster MILWB02SCluster 2015/06/04 11:02:03 VCS INFO V-16-1-50958 Local cluster dumping configuration to save ClusterList modification for group AppService 2015/06/04 11:02:04 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 11:02:06 VCS INFO V-16-1-50135 User root fired command: hagrp -modify ... -delete -keys AppService ClusterList from localhost 2015/06/04 11:02:07 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 11:02:10 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:02:10 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB03SCluster and remote cluster MILWB02SCluster 2015/06/04 11:02:10 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:02:10 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:02:10 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:02:10 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:02:10 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:02:10 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:02:10 VCS ERROR V-16-1-50531 Number of clusters in the ClusterList attribute for group AppService is different between local cluster MILWB03SCluster and remote cluster MILWB02SCluster 2015/06/04 11:02:12 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 11:02:14 VCS INFO V-16-1-50135 User root fired command: hagrp -modify AppService ClusterList MILWB02SCluster 0 MILWB03SCluster 1 from localhost 2015/06/04 11:02:14 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 11:02:15 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/04 11:02:17 VCS INFO V-16-1-50135 User root fired command: hagrp -flush VVRService MILWB03S localclus 0 from localhost 2015/06/04 11:02:18 VCS INFO V-16-1-10299 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (Not initiated by VCS) 2015/06/04 11:02:18 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/06/04 11:02:18 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/06/04 11:02:36 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MILWB03S localclus 0 from localhost 2015/06/04 11:02:53 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/04 11:04:26 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/06/04 11:04:26 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/06/04 11:04:26 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/06/04 11:04:26 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/06/04 11:04:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:05:28 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/04 11:05:29 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopped Eml_PubSvr..........................stopping... Eml_mml.............................stopped Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopping... TrapReceiver........................stopping... UniteUitl ============================================== 2015/06/04 11:05:31 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 11:05:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:05:32 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/04 11:05:33 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/04 11:05:33 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/04 11:05:33 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 11:05:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:05:36 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/04 11:05:37 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/04 11:05:37 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/04 11:05:37 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 11:05:37 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:05:38 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 11:05:38 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:05:41 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/04 11:05:43 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/04 11:05:43 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/04 11:05:43 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/06/04 11:05:43 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/04 11:05:43 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/04 11:05:43 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/04 11:05:43 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 11:05:43 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/04 11:05:43 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 11:09:51 VCS ERROR V-16-1-50981 Resource RVGPrimary is faulted on system MILWB02S in cluster MILWB02SCluster 2015/06/04 11:09:51 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/06/04 11:09:51 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/06/04 11:09:51 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/04 11:09:51 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/04 11:09:51 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/06/04 11:09:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/04 11:09:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:09:51 VCS INFO V-16-20010-72 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required 2015/06/04 11:09:53 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 11:09:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:09:55 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 11:09:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:09:55 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/04 11:09:55 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/04 11:09:55 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/04 11:09:55 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/04 11:09:55 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/06/04 11:09:55 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/04 11:10:24 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/04 11:10:25 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/22038/psinfo 2015/06/04 11:10:25 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 11:10:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:10:36 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/29320/psinfo 2015/06/04 11:10:36 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 11:10:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/04 11:10:47 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/04 11:10:48 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 22038 22036 2 11:09:56 ? 0:26 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 26683 22038 0 11:10:17 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 26726 22038 0 11:10:18 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 26769 22038 1 11:10:18 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 15-06-04 11:10:36 || haStart:sybase is starting, break! 15-06-04 11:10:36 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-04 11:10:46 || haStart:The imap is running, break! 15-06-04 11:10:47 || haStart:Success to start the nms! ============================================== 2015/06/04 11:11:00 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/04 11:11:00 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/06/04 11:11:18 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 99% 2015/06/04 12:04:58 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 12:04:58 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/04 13:21:50 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 17:21:51 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/04 21:21:52 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 01:21:53 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 05:21:54 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 09:21:55 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 13:21:56 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 17:21:57 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/05 21:21:58 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 01:21:59 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 05:22:00 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 09:22:01 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 13:22:02 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 17:22:03 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/06 21:22:03 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 01:22:04 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 05:22:05 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 09:22:06 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 13:22:07 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 17:22:08 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/07 21:22:09 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 01:22:10 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 05:22:11 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 09:22:12 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 09:57:06 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/06/08 10:16:34 VCS INFO V-16-1-50135 User admin fired command: haconf -makerw from ::ffff:10.66.28.69 2015/06/08 10:16:35 VCS INFO V-16-1-50135 User admin fired command: hagrp -freeze -persistent AppService from ::ffff:10.66.28.69 2015/06/08 10:21:06 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:23:49 VCS INFO V-16-1-50135 User admin fired command: hares -add neNIC NIC AppService from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Critical 1 from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC PingOptimize 1 from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Protocol IPv4 from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC ExclusiveIPZone 0 from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Device from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify ... -delete -keys neNIC NetworkHosts from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC NetworkType from ::ffff:10.66.28.69 2015/06/08 10:23:50 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Enabled 0 from ::ffff:10.66.28.69 2015/06/08 10:24:24 VCS INFO V-16-1-50135 User admin fired command: hares -delete neNIC from ::ffff:10.66.28.69 2015/06/08 10:26:44 VCS INFO V-16-1-50135 User admin fired command: hares -add neNIC NIC AppService from ::ffff:10.66.28.69 2015/06/08 10:26:44 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Critical 1 from ::ffff:10.66.28.69 2015/06/08 10:26:45 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC PingOptimize 1 from ::ffff:10.66.28.69 2015/06/08 10:26:45 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Protocol IPv4 from ::ffff:10.66.28.69 2015/06/08 10:26:45 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC ExclusiveIPZone 0 from ::ffff:10.66.28.69 2015/06/08 10:26:45 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Device bge0 from ::ffff:10.66.28.69 2015/06/08 10:26:45 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC NetworkHosts 10.66.28.19 from ::ffff:10.66.28.69 2015/06/08 10:26:45 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC NetworkType from ::ffff:10.66.28.69 2015/06/08 10:26:45 VCS INFO V-16-1-50135 User admin fired command: hares -modify neNIC Enabled 0 from ::ffff:10.66.28.69 2015/06/08 10:26:53 VCS INFO V-16-1-50135 User admin fired command: hagrp -unfreeze -persistent AppService from ::ffff:10.66.28.69 2015/06/08 10:27:37 VCS NOTICE V-16-1-50514 Remote cluster 'MILWB02SCluster' has exited 2015/06/08 10:27:37 VCS INFO V-16-3-18309 (MILWB03S) Cluster MILWB02SCluster exited 2015/06/08 10:29:25 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_ALL_FORCE from localhost 2015/06/08 10:29:25 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from RUNNING to LEAVING 2015/06/08 10:29:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/06/08 10:29:25 VCS NOTICE V-16-1-50402 VCS engine on system MILWB03S has lost connection with wide area connector 2015/06/08 10:29:25 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully. 2015/06/08 10:29:26 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (VCS initiated) 2015/06/08 10:29:26 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MILWB03S 2015/06/08 10:29:26 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LEAVING to EXITING 2015/06/08 10:29:26 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from EXITING to EXITED 2015/06/08 10:29:28 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:29:28 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:29:28 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:29:28 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:29:28 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:29:28 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:29:28 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/06/08 10:29:28 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/06/08 10:29:28 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/06/08 10:29:28 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:29:28 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:29:28 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:29:40 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:29:40 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:29:40 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:29:40 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:29:40 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:29:40 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:29:40 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:29:40 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/06/08 10:29:40 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:29:40 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:29:40 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:30:05 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:30:05 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:30:05 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:30:05 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:30:05 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:30:05 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:30:05 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:30:05 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/06/08 10:30:05 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:30:05 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:30:05 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:30:40 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:30:40 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:30:40 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:30:40 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:30:40 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:30:40 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:30:40 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:30:40 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/06/08 10:30:40 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:30:40 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:30:40 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:31:25 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:31:25 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:31:25 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:31:25 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:31:25 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:31:25 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:31:25 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:31:25 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/06/08 10:31:25 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:31:25 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:31:25 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:32:20 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:32:20 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:32:20 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-restart 2015/06/08 10:32:20 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:32:20 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:32:20 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:32:20 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/08 10:32:20 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/06/08 10:32:20 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:32:20 VCS ERROR V-16-1-10116 GabHandle::open failed errno = 261 2015/06/08 10:32:20 VCS ERROR V-16-1-11033 GAB open failed. Exiting 2015/06/08 10:39:44 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/08 10:39:44 VCS INFO V-16-1-10196 Cluster logger started 2015/06/08 10:39:44 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode 2015/06/08 10:39:44 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/08 10:39:44 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0 2015/06/08 10:39:44 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00 2015/06/08 10:39:44 VCS NOTICE V-16-1-10115 Using GABSIM 2015/06/08 10:39:44 VCS NOTICE V-16-1-14032 Forming a one node cluster 2015/06/08 10:39:49 VCS NOTICE V-16-1-10619 'HAD' starting on: MILWB03S 2015/06/08 10:39:49 VCS INFO V-16-1-51138 Number of processors configured on this system are 16 2015/06/08 10:39:49 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide 2015/06/08 10:39:49 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/08 10:39:49 VCS INFO V-16-1-10077 Received new cluster membership 2015/06/08 10:39:49 VCS NOTICE V-16-1-10112 System (MILWB03S) - Membership: 0x1, DDNA: 0x0 2015/06/08 10:39:49 VCS NOTICE V-16-1-10086 System MILWB03S (Node '0') is in Regular Membership - Membership: 0x1 2015/06/08 10:39:49 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/06/08 10:39:49 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 10:39:50 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing 2015/06/08 10:39:50 VCS NOTICE V-16-1-10322 System MILWB03S (Node '0') changed state from LOCAL_BUILD to RUNNING 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Mon Jun 8 10:39:50 2015 2015/06/08 10:39:50 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables 2015/06/08 10:39:51 VCS INFO V-16-6-15023 (MILWB03S) dump_tunables: ########## VCS Environment Variables ########## VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG=/var/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger ########## Other Environment Variables ########## EAT_CATALOG_DIR=/opt/VRTSvcs EAT_HOME_DIR=/opt/VRTSvcs EDITOR=vi HOME=/ HWENGRROOT=/opt/HWENGR HWICMRROOT=/opt/HWICMR HWNMSJRE=/opt/HWNMSJRE IMAP_JAVA_HOME=/opt/HWNMSJRE/jre_sol LANG=C LD_LIBRARY_PATH=/opt/VRTSvcs/lib:/usr/openwin/lib LOGNAME=root MAIL=/var/mail/root MANPATH=/opt/VRTSob/man:/opt/VRTS/man:/opt/VRTSvxvm/man::/usr/share/man:/usr/sunvts/man:/opt/SUNWexplo/man OPENWINHOME=/usr/openwin PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise/sbin:/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin PS1=root@MILWB03S # PS2=root@MILWB03S > PWD=/var/VRTSvcs/diag/had SHELL=/bin/bash SHLVL=1 SSH_CLIENT=10.66.21.87 49256 22 SSH_CONNECTION=10.66.21.87 49256 10.66.28.70 22 SSH_TTY=/dev/pts/2 TERM=xterm TZ=Europe/Rome USER=root VCSONE_LOG_AGENT_NAME= VCSONE_LOG_CATEGORY=6 VCSONE_LOG_SCRIPT_NAME=hatrigger _=/opt/VRTS/bin/hastart 2015/06/08 10:39:51 VCS INFO V-16-6-15002 (MILWB03S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MILWB03S 1 successfully 2015/06/08 10:39:51 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/29320/psinfo 2015/06/08 10:39:51 VCS INFO V-16-1-10297 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (First probe) 2015/06/08 10:39:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/08 10:39:51 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/22038/psinfo 2015/06/08 10:39:51 VCS INFO V-16-1-10297 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (First probe) 2015/06/08 10:39:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/08 10:39:51 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (First probe) 2015/06/08 10:39:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/08 10:39:51 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MILWB03S (First probe) 2015/06/08 10:39:51 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MILWB03S 2015/06/08 10:39:51 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService 2015/06/08 10:39:51 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes 2015/06/08 10:39:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MILWB03S 2015/06/08 10:39:51 VCS INFO V-16-10001-88 (MILWB03S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully. 2015/06/08 10:39:51 VCS INFO V-16-1-51058 Received connection request from wide area connector 2015/06/08 10:39:51 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted 2015/06/08 10:39:51 VCS INFO V-16-1-50403 VCS engine on system MILWB03S has established connection with wide area connector 2015/06/08 10:39:52 VCS INFO V-16-3-18210 (MILWB03S) Cluster MILWB03SCluster heartbeat Icmp to cluster MILWB02SCluster is alive 2015/06/08 10:39:52 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MILWB03S (First probe) 2015/06/08 10:39:52 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes 2015/06/08 10:39:52 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MILWB03S 2015/06/08 10:39:52 VCS NOTICE V-16-1-10447 Group VVRService is online on system MILWB03S 2015/06/08 10:39:52 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MILWB03S (VCS initiated) 2015/06/08 10:39:52 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MILWB03S 2015/06/08 10:39:53 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MILWB03S 2015/06/08 10:39:53 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MILWB03S as the system is not a part of AutoStartList attribute of the group. 2015/06/08 10:39:53 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (First probe) 2015/06/08 10:39:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/08 10:39:56 VCS INFO V-16-1-10297 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (First probe) 2015/06/08 10:39:56 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/08 10:39:56 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MILWB03S 2015/06/08 10:39:56 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set. 2015/06/08 10:39:56 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/06/08 10:41:09 VCS INFO V-16-3-18302 (MILWB03S) Cluster MILWB03SCluster received connection from cluster MILWB02SCluster 2015/06/08 10:41:09 VCS INFO V-16-1-50515 Receiving snapshot from remote cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50991 System MILWB02S is running in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/08 10:41:09 VCS INFO V-16-1-50516 Received full snapshot from remote cluster MILWB02SCluster 2015/06/08 10:41:09 VCS NOTICE V-16-1-50512 Resetting DeclaredState for remote cluster MILWB02SCluster 2015/06/08 10:41:37 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.69 2015/06/08 14:39:50 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 18:39:51 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/08 22:39:52 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 02:39:53 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 06:39:54 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 08:24:20 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 08:41:04 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 09:24:09 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost 2015/06/09 09:24:19 VCS INFO V-16-1-50135 User root fired command: hares -add neNIC NIC AppService from localhost 2015/06/09 09:24:37 VCS INFO V-16-1-50135 User root fired command: hares -modify neNIC Device bge0 from localhost 2015/06/09 09:24:48 VCS INFO V-16-1-50135 User root fired command: hares -modify neNIC NetworkHosts 10.66.28.19 from localhost 2015/06/09 09:24:56 VCS INFO V-16-1-50135 User root fired command: hares -modify neNIC Enabled 1 from localhost 2015/06/09 09:25:04 VCS INFO V-16-1-50135 User root fired command: hares -link RVGPrimary neNIC 0 0 from localhost 2015/06/09 09:25:13 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost 2015/06/09 09:38:40 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/06/09 09:38:40 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/06/09 09:38:40 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/06/09 09:38:40 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/06/09 09:38:40 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 09:39:42 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/09 09:39:43 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopped EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopping... LicenseService......................stopping... LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopped PMSWeb..............................stopped SNMPCollectorDm_4402................stopping... SNMPServer..........................stopped SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopping... TrapReceiver.................. ============================================== 2015/06/09 09:39:45 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 09:39:45 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 09:39:46 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/09 09:39:46 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/09 09:39:47 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/09 09:39:47 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 09:39:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 09:39:47 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/09 09:39:48 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/09 09:39:48 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/09 09:39:48 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 09:39:48 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 09:39:49 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 09:39:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 09:39:51 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/09 09:39:54 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 09:39:54 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/09 09:39:54 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/06/09 09:39:54 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/09 09:39:54 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/09 09:39:54 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/09 09:39:54 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 09:39:54 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/09 09:39:54 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 09:40:06 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 09:40:07 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 09:40:31 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 09:40:41 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 09:41:07 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 09:41:07 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 10:39:55 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 10:40:06 VCS ERROR V-16-1-50981 Resource neNIC is faulted on system MILWB02S in cluster MILWB02SCluster 2015/06/09 10:41:11 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 10:41:32 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 10:41:33 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 10:41:35 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 10:41:39 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 10:41:39 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/06/09 10:41:39 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/06/09 10:41:39 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/09 10:41:39 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 10:41:39 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/06/09 10:41:39 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 10:41:39 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 10:41:39 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/09 10:41:48 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/06/09 10:41:49 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 10:41:49 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 10:41:51 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 10:41:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 10:41:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/09 10:41:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/09 10:41:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/09 10:41:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/09 10:41:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/06/09 10:41:52 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/09 10:42:21 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/09 10:42:21 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/27621/psinfo 2015/06/09 10:42:21 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 10:42:21 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 10:42:32 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/5205/psinfo 2015/06/09 10:42:32 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 10:42:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 10:42:44 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/09 10:42:44 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 27621 27619 2 10:41:52 ? 0:26 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt sybase 2665 27621 0 10:42:15 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 2711 27621 1 10:42:15 ? 0:03 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 2616 27621 0 10:42:14 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 15-06-09 10:42:33 || haStart:sybase is starting, break! 15-06-09 10:42:33 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-09 10:42:43 || haStart:The imap is running, break! 15-06-09 10:42:44 || haStart:Success to start the nms! ============================================== 2015/06/09 10:43:06 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 10:43:06 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/06/09 10:43:26 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 100% 2015/06/09 11:18:07 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 11:18:07 VCS NOTICE V-16-1-50982 Resource neNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 12:03:42 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/06/09 12:03:42 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/06/09 12:03:42 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/06/09 12:03:42 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/06/09 12:03:42 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 12:04:44 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/09 12:04:45 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopping... DCServer............................stopping... DmsBaseDm...........................stopping... EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopping... ItmService..........................stopped LicenseService......................stopped LogService..........................stopping... Naming_Service......................stopped Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopping... Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopping... Notify_Service......................stopped PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopped SyslogCollectorDM...................stopping... SystemService.......................stopped TXTNBIDm............................stopping... ToolkitService......................stopped TopoService.........................stopped TrapReceiver........................st ============================================== 2015/06/09 12:04:47 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 12:04:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 12:04:47 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/09 12:04:48 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/09 12:04:48 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/09 12:04:48 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 12:04:48 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 12:04:49 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/09 12:04:49 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/09 12:04:49 VCS WARNING V-16-20018-301 (MILWB03S) Sybase:DatabaseServer:monitor:Open for dataserver failed, setting cookie to NULL 2015/06/09 12:04:49 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 12:04:49 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 12:04:51 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 12:04:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 12:04:53 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/09 12:04:55 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/09 12:04:55 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/09 12:04:55 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/06/09 12:04:55 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/09 12:04:55 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/09 12:04:55 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/09 12:04:55 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 12:04:55 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/09 12:04:55 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 12:05:04 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 12:05:06 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 12:05:28 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 12:05:39 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 12:06:10 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 12:06:11 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:07:04 VCS ERROR V-16-1-50981 Resource appNIC is faulted on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:10:10 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:10.66.28.22 2015/06/09 14:12:14 VCS NOTICE V-16-1-50983 Resource NMSServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:17:17 VCS NOTICE V-16-1-50983 Resource BackupServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:19:38 VCS NOTICE V-16-1-50983 Resource DatabaseServer is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:19:39 VCS NOTICE V-16-1-50983 Resource DataFilesystem is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:19:44 VCS NOTICE V-16-1-50983 Resource RVGPrimary is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:19:44 VCS NOTICE V-16-1-50923 Local cluster MILWB03SCluster is assuming Authority for group AppService 2015/06/09 14:19:44 VCS INFO V-16-1-50925 Proceeding to online group AppService on the best possible system in the local cluster 2015/06/09 14:19:44 VCS INFO V-16-1-50810 Local cluster taking up Authority for group AppService 2015/06/09 14:19:44 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/09 14:19:44 VCS INFO V-16-1-10493 Evaluating MILWB03S as potential target node for group AppService 2015/06/09 14:19:44 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/09 14:19:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 14:19:44 VCS INFO V-16-20010-84 (MILWB03S) RVGPrimary:RVGPrimary:online:RLink datarlk will be used to decide for takeover or migrate. 2015/06/09 14:19:53 VCS INFO V-16-20010-76 (MILWB03S) RVGPrimary:RVGPrimary:online:RVG datarvg migrated successfully 2015/06/09 14:19:54 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 14:19:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 14:19:56 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 14:19:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 14:19:56 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1 2015/06/09 14:19:56 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY: 2015/06/09 14:19:56 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY 2015/06/09 14:19:56 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY 2015/06/09 14:19:56 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt GOJsOShSUsNSlSTsLQvMHmIMj 2> /dev/null 2015/06/09 14:19:56 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001 2015/06/09 14:20:25 VCS INFO V-16-20018-999 (MILWB03S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating 2015/06/09 14:20:26 VCS INFO V-16-20018-7 (MILWB03S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/11129/psinfo 2015/06/09 14:20:26 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 14:20:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 14:20:37 VCS INFO V-16-20018-7 (MILWB03S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/18235/psinfo 2015/06/09 14:20:37 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 14:20:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/09 14:20:48 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:online:Executed [/opt/HWENGR/NMSApp/haStart] successfully. 2015/06/09 14:20:49 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (online) ============================================== sybase 15812 11129 0 14:20:18 ? 0:00 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x655f0cf8, 10000000000, 0x1c39 sybase 15859 11129 0 14:20:18 ? 0:01 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x655f0cf8, 10000000000, 0x1c39 sybase 15904 11129 1 14:20:18 ? 0:05 /opt/sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x655f0cf8, 10000000000, 0x1c39 sybase 11129 11127 2 14:19:57 ? 0:24 /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt 15-06-09 14:20:38 || haStart:sybase is starting, break! 15-06-09 14:20:38 || haStart:The imap is not running, try to start it... Oracle Corporation SunOS 5.10 Generic Patch January 2005 15-06-09 14:20:48 || haStart:The imap is running, break! 15-06-09 14:20:48 || haStart:Success to start the nms! ============================================== 2015/06/09 14:21:07 VCS INFO V-16-1-10298 Resource NMSServer (Owner: Unspecified, Group: AppService) is online on MILWB03S (VCS initiated) 2015/06/09 14:21:07 VCS NOTICE V-16-1-10447 Group AppService is online on system MILWB03S 2015/06/09 14:21:26 VCS CRITICAL V-16-1-50086 CPU usage on MILWB03S is 99% 2015/06/09 14:39:06 VCS NOTICE V-16-1-50963 Global group AppService is offline on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:39:06 VCS NOTICE V-16-1-50982 Resource appNIC is online on system MILWB02S in cluster MILWB02SCluster 2015/06/09 14:39:56 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 18:39:57 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/09 22:39:57 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 02:39:58 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 06:39:59 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 10:40:00 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 14:40:01 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 18:40:02 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/10 22:40:03 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 02:40:04 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 06:40:05 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 10:40:06 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 14:40:07 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 18:40:08 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/11 22:40:09 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/12 02:40:10 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/12 06:40:11 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/06/12 08:11:00 VCS INFO V-16-1-50859 Attempting to switch group AppService from system MILWB03S to system MILWB02S 2015/06/12 08:11:00 VCS INFO V-16-1-50135 User root fired command: hagrp -switch AppService MILWB02S MILWB02SCluster from localhost 2015/06/12 08:11:00 VCS NOTICE V-16-1-50929 Initial tests indicate group AppService is able to switch to system MILWB02S. Initiating offline of group on system MILWB03S 2015/06/12 08:11:00 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MILWB03S 2015/06/12 08:11:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/12 08:12:02 VCS INFO V-16-10001-88 (MILWB03S) Application:NMSServer:offline:Executed [/opt/HWENGR/NMSApp/haStop] successfully. 2015/06/12 08:12:03 VCS INFO V-16-2-13716 (MILWB03S) Resource(NMSServer): Output of the completed operation (offline) ============================================== Oracle Corporation SunOS 5.10 Generic Patch January 2005 Agent_CORBA.........................stopping... BulkCollectorDm_4402................stopped DCServer............................stopping... DmsBaseDm...........................stopped EAMService..........................stopping... EmfGnlDevDm.........................stopping... EmfTrapReceiver_5754................stopping... Eml_PerfSvr.........................stopping... Eml_PubSvr..........................stopping... Eml_mml.............................stopping... Eml_monitomcat......................stopping... FaultService........................stopping... ForwardingService...................stopped InventoryDM.........................stopped ItmService..........................stopping... LicenseService......................stopped LogService..........................stopping... Naming_Service......................stopping... Nml_ason_otn........................stopping... Nml_ason_sdh........................stopping... Nml_common..........................stopped Nml_cps.............................stopping... Nml_eth.............................stopping... Nml_ip..............................stopping... Nml_otn.............................stopping... Nml_sdh.............................stopping... Nml_vlan............................stopping... NmslogzipDm_398.....................stopped Notify_Service......................stopping... PMDataRefiner.......................stopping... PMSDm...............................stopping... PMSWeb..............................stopping... SNMPCollectorDm_4402................stopping... SNMPServer..........................stopping... SecurityService.....................stopping... SyslogCollectorDM...................stopping... SystemService.......................stopping... TXTNBIDm............................stopping... ToolkitService......................stopping... TopoService.........................stopped TrapReceiver...................... ============================================== 2015/06/12 08:12:05 VCS INFO V-16-1-10305 Resource NMSServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/12 08:12:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/12 08:12:06 VCS NOTICE V-16-20018-26 (MILWB03S) SybaseBk:BackupServer:offline:Sybase Backup service DBSVR_back has been stopped 2015/06/12 08:12:06 VCS INFO V-16-2-13716 (MILWB03S) Resource(BackupServer): Output of the completed operation (offline) ============================================== Password: Backup Server: 3.48.1.1: The Backup Server will go down immediately. Terminating sessions. ============================================== 2015/06/12 08:12:07 VCS WARNING V-16-20018-301 (MILWB03S) SybaseBk:BackupServer:monitor:Open for backupserver failed, setting cookie to NULL 2015/06/12 08:12:07 VCS INFO V-16-1-10305 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/12 08:12:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/12 08:12:09 VCS NOTICE V-16-20018-18 (MILWB03S) Sybase:DatabaseServer:offline:Sybase service DBSVR has been stopped 2015/06/12 08:12:10 VCS INFO V-16-2-13716 (MILWB03S) Resource(DatabaseServer): Output of the completed operation (offline) ============================================== Password: Server SHUTDOWN by request. ASE is terminating this process. CT-LIBRARY error: ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect ============================================== 2015/06/12 08:12:10 VCS CRITICAL V-16-20018-4 (MILWB03S) Sybase:DatabaseServer:monitor:Process dataserver restarted 2015/06/12 08:12:10 VCS INFO V-16-1-10305 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/12 08:12:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/12 08:12:11 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/12 08:12:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MILWB03S 2015/06/12 08:12:13 VCS INFO V-16-20010-176 (MILWB03S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed 2015/06/12 08:12:16 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MILWB03S (VCS initiated) 2015/06/12 08:12:16 VCS INFO V-16-1-50924 Proceeding to online group AppService on system MILWB02S 2015/06/12 08:12:16 VCS NOTICE V-16-1-10446 Group AppService is offline on system MILWB03S 2015/06/12 08:12:16 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/12 08:12:16 VCS NOTICE V-16-1-51038 Received Authority request from cluster MILWB02SCluster to online group AppService on system MILWB02S 2015/06/12 08:12:16 VCS INFO V-16-1-50811 Local cluster giving up Authority for group AppService 2015/06/12 08:12:16 VCS INFO V-16-1-50814 Local cluster dumping configuration to save Authority information for group AppService 2015/06/12 08:12:16 VCS NOTICE V-16-1-50754 Clearing IntentOnline attribute for failover group AppService 2015/06/12 08:12:16 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes 2015/06/12 08:12:26 VCS NOTICE V-16-1-50982 Resource RVGPrimary is online on system MILWB02S in cluster MILWB02SCluster 2015/06/12 08:12:27 VCS NOTICE V-16-1-50982 Resource DataFilesystem is online on system MILWB02S in cluster MILWB02SCluster 2015/06/12 08:12:50 VCS NOTICE V-16-1-50982 Resource DatabaseServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/12 08:13:01 VCS NOTICE V-16-1-50982 Resource BackupServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/12 08:13:25 VCS NOTICE V-16-1-50982 Resource NMSServer is online on system MILWB02S in cluster MILWB02SCluster 2015/06/12 08:13:26 VCS NOTICE V-16-1-50962 Global group AppService is online on system MILWB02S in cluster MILWB02SCluster