04-30-2012 06:30 AM
Hi all.
I have a single node cluster, ready to become a part of a global cluster in the future.
I restarted the node (system) starting with this situation:
# hagrp -state
#Group Attribute System Value
AppService State MIVDB01S |ONLINE|
ClusterService State MIVDB01S |ONLINE|
VVRService State MIVDB01S |ONLINE|
after reboot I had this situation:
# hagrp -state
#Group Attribute System Value
AppService State MIVDB01S |PARTIAL|
ClusterService State MIVDB01S |ONLINE|
VVRService State MIVDB01S |ONLINE|
# hares -state
#Resource Attribute System Value
BackupServer State MIVDB01S OFFLINE
DataFilesystem State MIVDB01S OFFLINE
DatabaseServer State MIVDB01S OFFLINE
NMSServer State MIVDB01S OFFLINE|STATE UNKNOWN
RVGPrimary State MIVDB01S OFFLINE
appNIC State MIVDB01S ONLINE
datarvg State MIVDB01S ONLINE
ntfr State MIVDB01S ONLINE
wac State MIVDB01S ONLINE
I cannot escape from the 'STATE UNKNOWN' situation.
Can you help me ?
I try to reboot the server for the second time.
BR
Solved! Go to Solution.
05-02-2012 03:48 AM
Can you run:
/opt/HWENGR/NMSApp/haMonitor
echo $?
and post the output of results. I suspect the monitor program is not exiting with 100 or 110, which it needs to.
A few other things:
Mike
05-01-2012 02:45 AM
This means the monitor routine for resource NMSServer is not working. Could you post your main.cf so I can see types of resources and their attributes. I also noticed you don't have a resource that is obviously a diskgroup. Also look at /var/VRTSvcs/log/engine_A.log to see if it says why NMSServer monitor is failing.
Mike
05-02-2012 12:41 AM
Hi Mike.
You are right, but I don't understand why.
hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A MIVDB01S RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B AppService MIVDB01S N N PARTIAL
B ClusterService MIVDB01S Y N ONLINE
B VVRService MIVDB01S Y N ONLINE
-- RESOURCES NOT PROBED
-- Group Type Resource System
E AppService Application NMSServer MIVDB01S
I run the command:
hares -probe NMSServer -sys MIVDB01S
but with no effect
root@MIVDB01S # cat /etc/VRTSvcs/conf/config/main.cf
include "types.cf"
include "SybaseTypes.cf"
cluster MIVDB01SCluster (
UserNames = { admin = IpqIpkPmqLqqOyqKpn }
ClusterAddress = "172.22.8.132"
Administrators = { admin }
)
system MIVDB01S (
)
group AppService (
SystemList = { MIVDB01S = 0 }
AutoStart = 0
ClusterFailOverPolicy = Auto
)
Application NMSServer (
Critical = 0
StartProgram = "/opt/HWENGR/NMSApp/haStart"
StopProgram = "/opt/HWENGR/NMSApp/haStop"
CleanProgram = "/opt/HWENGR/NMSApp/haClean"
MonitorProgram = "/opt/HWENGR/NMSApp/haMonitor"
)
Mount DataFilesystem (
MountPoint = "/opt/sybase/data"
BlockDevice = "/dev/vx/dsk/datadg/lv_nms_data"
FSType = ufs
FsckOpt = "-y"
)
NIC appNIC (
Critical = 0
Device = bge0
NetworkHosts = { "172.22.8.115" }
)
RVGPrimary RVGPrimary (
RvgResourceName = datarvg
)
Sybase DatabaseServer (
Server = DBSVR
Owner = sybase
Home = "/opt/sybase"
Version = "15.0"
SA = sa
SApswd = dlgPlpEprPkpIpqPijEjfJg
User = NMSuser
UPword = NMSuser
)
SybaseBk BackupServer (
Critical = 0
Backupserver = DBSVR_back
Owner = sybase
Home = "/opt/sybase"
Version = "15.0"
Server = DBSVR
SA = sa
SApswd = dlgPlpEprPkpIpqPijEjfJg
)
requires group VVRService online local firm
BackupServer requires DatabaseServer
DataFilesystem requires RVGPrimary
DatabaseServer requires DataFilesystem
NMSServer requires BackupServer
RVGPrimary requires appNIC
// resource dependency tree
//
// group AppService
// {
// Application NMSServer
// {
// SybaseBk BackupServer
// {
// Sybase DatabaseServer
// {
// Mount DataFilesystem
// {
// RVGPrimary RVGPrimary
// {
// NIC appNIC
// }
// }
// }
// }
// }
// }
group ClusterService (
SystemList = { MIVDB01S = 0 }
AutoStartList = { MIVDB01S }
ClusterFailOverPolicy = Auto
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)
Application wac (
StartProgram = "/opt/VRTSvcs/bin/wacstart"
StopProgram = "/opt/VRTSvcs/bin/wacstop"
MonitorProcesses = { "/opt/VRTSvcs/bin/wac" }
RestartLimit = 3
)
NotifierMngr ntfr (
Critical = 0
SnmpConsoles = { MIVDB01S = Information }
)
// resource dependency tree
//
// group ClusterService
// {
// NotifierMngr ntfr
// Application wac
// }
group VVRService (
SystemList = { MIVDB01S = 0 }
AutoStartList = { MIVDB01S }
ClusterFailOverPolicy = Auto
)
RVG datarvg (
Critical = 0
RVG = datarvg
DiskGroup = datadg
)
// resource dependency tree
//
// group VVRService
// {
// RVG datarvg
// }
After the situation was correct, I rebooted the server with the command: 'init 6' without stop the VCS.
This is an intermediate situation with one local cluster with one node only. In the futre we have to add a second local cluster to obtail a global cluster.
root@MIVDB01S # cat /var/VRTSvcs/log/engine_A.log | grep "2012/04/30"
2012/04/30 00:13:35 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/04/30 04:13:36 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/04/30 08:02:51 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/04/30 08:02:51 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from RUNNING to LEAVING
2012/04/30 08:02:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource NMSServer (Owner: Unspecified, Group: AppService) on System MIVDB01S
2012/04/30 08:02:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 08:02:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ntfr (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 08:02:53 VCS ERROR V-16-10001-87 (MIVDB01S) Application:NMSServer:offline:Failed to execute [/opt/HWENGR/NMSApp/haStop].
2012/04/30 08:02:53 VCS INFO V-16-20010-222 (MIVDB01S) RVG:datarvg:monitor:vradmind not running. Starting the vradmind deamon.
2012/04/30 08:02:53 VCS INFO V-16-2-13716 (MIVDB01S) Resource(NMSServer): Output of the completed operation (offline)
2012/04/30 08:02:54 VCS INFO V-16-1-10305 Resource ntfr (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (VCS initiated)
2012/04/30 08:02:54 VCS NOTICE V-16-1-50402 VCS engine on system MIVDB01S has lost connection with wide area connector
2012/04/30 08:02:54 VCS INFO V-16-10001-88 (MIVDB01S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully.
2012/04/30 08:02:54 VCS INFO V-16-20010-223 (MIVDB01S) RVG:datarvg:monitor:in.vxrsyncd not running. Starting the in.vxrsyncd deamon.
2012/04/30 08:02:55 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (VCS initiated)
2012/04/30 08:02:55 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MIVDB01S
2012/04/30 08:02:55 VCS INFO V-16-1-10493 Evaluating MIVDB01S as potential target node for group ClusterService
2012/04/30 08:02:55 VCS INFO V-16-1-10494 System MIVDB01S not in RUNNING state
2012/04/30 08:02:55 VCS NOTICE V-16-1-10490 Resetting Evacuating (lastval=0) for group ClusterService for node MIVDB01S
2012/04/30 08:02:56 VCS ERROR V-16-2-13064 (MIVDB01S) Agent is calling clean for resource(NMSServer) because the resource is up even after offline completed.
2012/04/30 08:02:58 VCS INFO V-16-2-13068 (MIVDB01S) Resource(NMSServer) - clean completed successfully.
2012/04/30 08:03:00 VCS ERROR V-16-2-13077 (MIVDB01S) Agent is unable to offline resource(NMSServer). Administrative intervention may be required.
2012/04/30 08:03:52 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/04/30 08:04:53 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS from localhost
2012/04/30 08:11:55 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/04/30 08:11:55 VCS INFO V-16-1-10196 Cluster logger started
2012/04/30 08:11:55 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode
2012/04/30 08:11:55 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/04/30 08:11:55 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0
2012/04/30 08:11:55 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00
2012/04/30 08:11:55 VCS NOTICE V-16-1-10115 Using GABSIM
2012/04/30 08:11:55 VCS NOTICE V-16-1-14032 Forming a one node cluster
2012/04/30 08:12:00 VCS NOTICE V-16-1-10619 'HAD' starting on: MIVDB01S
2012/04/30 08:12:00 VCS INFO V-16-1-51138 Number of processors configured on this system are 32
2012/04/30 08:12:00 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/04/30 08:12:01 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/04/30 08:12:02 VCS INFO V-16-1-10077 Received new cluster membership
2012/04/30 08:12:02 VCS NOTICE V-16-1-10112 System (MIVDB01S) - Membership: 0x1, DDNA: 0x0
2012/04/30 08:12:02 VCS NOTICE V-16-1-10086 System MIVDB01S (Node '0') is in Regular Membership - Membership: 0x1
2012/04/30 08:12:02 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2012/04/30 08:12:03 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/04/30 08:12:03 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from LOCAL_BUILD to RUNNING
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Mon Apr 30 08:12:04 2012
2012/04/30 08:12:04 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/04/30 08:12:04 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (First probe)
2012/04/30 08:12:04 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (First probe)
2012/04/30 08:12:04 VCS INFO V-16-6-15023 (MIVDB01S) dump_tunables:
2012/04/30 08:12:04 VCS INFO V-16-1-10304 Resource ntfr (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (First probe)
2012/04/30 08:12:04 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (First probe)
2012/04/30 08:12:04 VCS INFO V-16-6-15002 (MIVDB01S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MIVDB01S 1 successfully
2012/04/30 08:12:05 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (First probe)
2012/04/30 08:12:05 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MIVDB01S
2012/04/30 08:12:05 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService
2012/04/30 08:12:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes
2012/04/30 08:12:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 08:12:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource ntfr (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 08:12:05 VCS INFO V-16-10001-88 (MIVDB01S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully.
2012/04/30 08:12:05 VCS INFO V-16-1-51058 Received connection request from wide area connector
2012/04/30 08:12:05 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted
2012/04/30 08:12:05 VCS INFO V-16-1-50403 VCS engine on system MIVDB01S has established connection with wide area connector
2012/04/30 08:12:05 VCS INFO V-16-1-10298 Resource ntfr (Owner: Unspecified, Group: ClusterService) is online on MIVDB01S (VCS initiated)
2012/04/30 08:12:05 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MIVDB01S (First probe)
2012/04/30 08:12:05 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes
2012/04/30 08:12:05 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MIVDB01S
2012/04/30 08:12:05 VCS NOTICE V-16-1-10447 Group VVRService is online on system MIVDB01S
2012/04/30 08:12:06 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MIVDB01S (VCS initiated)
2012/04/30 08:12:06 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MIVDB01S
2012/04/30 08:12:06 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MIVDB01S
2012/04/30 08:12:06 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MIVDB01S as the system is not a part of AutoStartList attribute of the group.
2012/04/30 08:12:07 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MIVDB01S (First probe)
2012/04/30 08:12:07 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 08:28:17 VCS INFO V-16-1-50135 User root fired command: hares -online DataFilesystem MIVDB01S from localhost
2012/04/30 09:11:44 VCS INFO V-16-1-50135 User root fired command: hares -online DataFilesystem MIVDB01S from localhost
2012/04/30 09:12:20 VCS INFO V-16-1-50135 User root fired command: hares -online BackupServer MIVDB01S from localhost
2012/04/30 09:14:03 VCS INFO V-16-1-50135 User root fired command: hares -clear NMSServer from localhost
2012/04/30 09:45:15 VCS INFO V-16-1-50135 User root fired command: hares -online DataFilesystem MIVDB01S from localhost
2012/04/30 09:47:04 VCS INFO V-16-1-10299 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MIVDB01S (Not initiated by VCS)
2012/04/30 09:47:04 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 09:47:46 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 10:36:04 VCS INFO V-16-1-50135 User root fired command: hagrp -offline AppService MIVDB01S from localhost
2012/04/30 10:36:04 VCS NOTICE V-16-1-10167 Initiating manual offline of group AppService on system MIVDB01S
2012/04/30 10:36:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MIVDB01S
2012/04/30 10:36:05 VCS INFO V-16-1-10305 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (VCS initiated)
2012/04/30 10:36:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MIVDB01S
2012/04/30 10:36:07 VCS INFO V-16-20010-176 (MIVDB01S) RVGPrimary:RVGPrimary:offline:Lockfile was successfully removed
2012/04/30 10:36:10 VCS INFO V-16-1-10305 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (VCS initiated)
2012/04/30 10:36:10 VCS NOTICE V-16-1-10446 Group AppService is offline on system MIVDB01S
2012/04/30 10:38:16 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 10:56:59 VCS INFO V-16-1-50135 User root fired command: hares -clear NMSServer MIVDB01S from localhost
2012/04/30 11:03:17 VCS INFO V-16-1-50135 User root fired command: hares -offline NMSServer MIVDB01S from localhost
2012/04/30 11:44:57 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 11:54:48 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 11:54:50 VCS INFO V-16-1-50135 User root fired command: hagrp -enable AppService from localhost
2012/04/30 11:55:20 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2012/04/30 11:55:26 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 11:59:26 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 12:00:15 VCS INFO V-16-1-50135 User root fired command: hagrp -enable VVRService from localhost
2012/04/30 12:00:26 VCS INFO V-16-1-50135 User root fired command: hagrp -enable ClusterService from localhost
2012/04/30 12:00:35 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2012/04/30 12:00:40 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 12:01:40 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 12:02:30 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Enabled 1 from localhost
2012/04/30 12:02:44 VCS INFO V-16-1-50135 User root fired command: hares -modify ntfr Enabled 1 from localhost
2012/04/30 12:02:55 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 1 from localhost
2012/04/30 12:03:08 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Enabled 1 from localhost
2012/04/30 12:03:27 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost
2012/04/30 12:03:41 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Enabled 1 from localhost
2012/04/30 12:03:55 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Enabled 1 from localhost
2012/04/30 12:04:14 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Enabled 1 from localhost
2012/04/30 12:04:30 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Enabled 1 from localhost
2012/04/30 12:04:50 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2012/04/30 12:04:58 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 12:06:23 VCS INFO V-16-1-50135 User root fired command: hares -online RVGPrimary MIVDB01S from localhost
2012/04/30 12:06:31 VCS INFO V-16-1-50135 User root fired command: hares -clear NMSServer from localhost
2012/04/30 12:07:13 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/04/30 12:08:26 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 12:08:39 VCS INFO V-16-1-50135 User root fired command: hagrp -enable AppService from localhost
2012/04/30 12:08:56 VCS INFO V-16-1-50135 User root fired command: hagrp -enable VVRService from localhost
2012/04/30 12:09:10 VCS INFO V-16-1-50135 User root fired command: hagrp -enable ClusterService from localhost
2012/04/30 12:09:16 VCS INFO V-16-1-50135 User root fired command: hagrp -enable AppService from localhost
2012/04/30 12:09:34 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Enabled 1 from localhost
2012/04/30 12:09:48 VCS INFO V-16-1-50135 User root fired command: hares -modify ntfr Enabled 1 from localhost
2012/04/30 12:09:57 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 1 from localhost
2012/04/30 12:10:03 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Enabled 1 from localhost
2012/04/30 12:10:11 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost
2012/04/30 12:10:25 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost
2012/04/30 12:10:26 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Enabled 1 from localhost
2012/04/30 12:10:34 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Enabled 1 from localhost
2012/04/30 12:10:44 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Enabled 1 from localhost
2012/04/30 12:10:50 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Enabled 1 from localhost
2012/04/30 12:11:02 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2012/04/30 12:13:17 VCS INFO V-16-1-50135 User root fired command: hares -online DataFilesystem MIVDB01S from localhost
2012/04/30 12:18:57 VCS INFO V-16-1-50135 User root fired command: hares -online RVGPrimary MIVDB01S from localhost
2012/04/30 12:35:06 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 12:35:54 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Enabled 1 from localhost
2012/04/30 12:36:18 VCS INFO V-16-1-50135 User root fired command: hares -modify ntfr Enabled 1 from localhost
2012/04/30 12:36:29 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 1 from localhost
2012/04/30 12:37:26 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Enabled 1 from localhost
2012/04/30 12:37:45 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost
2012/04/30 12:37:59 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Enabled 1 from localhost
2012/04/30 12:38:20 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Enabled 1 from localhost
2012/04/30 12:38:35 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Enabled 1 from localhost
2012/04/30 12:38:50 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Enabled 1 from localhost
2012/04/30 12:40:18 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2012/04/30 12:42:39 VCS INFO V-16-1-50135 User root fired command: hares -online RVGPrimary MIVDB01S from localhost
2012/04/30 12:44:52 VCS INFO V-16-1-50135 User root fired command: hares -online DataFilesystem MIVDB01S from localhost
2012/04/30 12:52:54 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE RVGPrimary MIVDB01S from localhost
2012/04/30 12:52:55 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 12:53:54 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE DataFilesystem MIVDB01S from localhost
2012/04/30 12:53:54 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 12:55:57 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MIVDB01S from localhost
2012/04/30 12:56:18 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MIVDB01S from localhost
2012/04/30 12:56:57 VCS INFO V-16-1-50135 User root fired command: hares -online RVGPrimary MIVDB01S from localhost
2012/04/30 13:04:49 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 13:04:57 VCS INFO V-16-1-50135 User root fired command: hares -modify wac Enabled 1 from localhost
2012/04/30 13:05:05 VCS INFO V-16-1-50135 User root fired command: hares -modify ntfr Enabled 1 from localhost
2012/04/30 13:05:15 VCS INFO V-16-1-50135 User root fired command: hares -modify datarvg Enabled 1 from localhost
2012/04/30 13:05:22 VCS INFO V-16-1-50135 User root fired command: hares -modify appNIC Enabled 1 from localhost
2012/04/30 13:05:29 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost
2012/04/30 13:05:36 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Enabled 1 from localhost
2012/04/30 13:05:43 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Enabled 1 from localhost
2012/04/30 13:05:49 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Enabled 1 from localhost
2012/04/30 13:05:55 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Enabled 1 from localhost
2012/04/30 13:06:02 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2012/04/30 13:06:46 VCS INFO V-16-1-50135 User root fired command: hares -online RVGPrimary MIVDB01S from localhost
2012/04/30 13:21:47 VCS INFO V-16-1-50135 User root fired command: hares -online BackupServer MIVDB01S from localhost
2012/04/30 13:22:06 VCS INFO V-16-1-50135 User root fired command: hares -online DatabaseServer MIVDB01S from localhost
2012/04/30 13:22:25 VCS INFO V-16-1-50135 User root fired command: hares -online DataFilesystem MIVDB01S from localhost
2012/04/30 13:30:03 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 13:30:04 VCS INFO V-16-1-50135 User root fired command: hagrp -enable AppService from localhost
2012/04/30 13:34:47 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MIVDB01S from localhost
2012/04/30 13:41:03 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE DataFilesystem MIVDB01S from localhost
2012/04/30 13:41:03 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 13:41:25 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE DatabaseServer MIVDB01S from localhost
2012/04/30 13:41:25 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 13:41:36 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE BackupServer MIVDB01S from localhost
2012/04/30 13:41:36 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 13:42:20 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MIVDB01S from localhost
2012/04/30 13:42:51 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE RVGPrimary MIVDB01S from localhost
2012/04/30 13:42:52 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 13:44:37 VCS INFO V-16-1-50135 User root fired command: hares -modify RVGPrimary Enabled 1 from localhost
2012/04/30 13:44:54 VCS INFO V-16-1-50135 User root fired command: hares -modify NMSServer Enabled 1 from localhost
2012/04/30 13:45:02 VCS INFO V-16-1-50135 User root fired command: hares -modify DatabaseServer Enabled 1 from localhost
2012/04/30 13:45:08 VCS INFO V-16-1-50135 User root fired command: hares -modify DataFilesystem Enabled 1 from localhost
2012/04/30 13:45:15 VCS INFO V-16-1-50135 User root fired command: hares -modify BackupServer Enabled 1 from localhost
2012/04/30 13:46:27 VCS INFO V-16-1-50135 User root fired command: hares -online RVGPrimary MIVDB01S from localhost
2012/04/30 13:46:45 VCS INFO V-16-1-50135 User root fired command: hares -online DataFilesystem MIVDB01S from localhost
2012/04/30 14:26:04 VCS INFO V-16-1-50135 User root fired command: hagrp -flush AppService MIVDB01S 0 from localhost
2012/04/30 14:26:04 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:26:04 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:26:04 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:26:05 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:29:39 VCS INFO V-16-1-50135 User root fired command: hagrp -clearadminwait AppService MIVDB01S from localhost
2012/04/30 14:33:02 VCS INFO V-16-1-50135 User root fired command: hagrp -disableresources AppService from localhost
2012/04/30 14:33:02 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MIVDB01S
2012/04/30 14:33:02 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set.
2012/04/30 14:33:47 VCS INFO V-16-1-50135 User root fired command: hares -clear NMSServer MIVDB01S from localhost
2012/04/30 14:34:07 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost
2012/04/30 14:34:07 VCS INFO V-16-1-10306 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:34:07 VCS INFO V-16-1-10306 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:34:07 VCS INFO V-16-1-10306 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:34:09 VCS INFO V-16-1-10306 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (Previous State = OFFLINE)
2012/04/30 14:35:43 VCS INFO V-16-1-50135 User root fired command: hagrp -offline -force AppService MIVDB01S from localhost
2012/04/30 15:34:01 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_ALL_FORCE from localhost
2012/04/30 15:34:01 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from RUNNING to LEAVING
2012/04/30 15:34:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 15:34:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ntfr (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 15:34:01 VCS NOTICE V-16-1-50402 VCS engine on system MIVDB01S has lost connection with wide area connector
2012/04/30 15:34:01 VCS INFO V-16-10001-88 (MIVDB01S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully.
2012/04/30 15:34:02 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (VCS initiated)
2012/04/30 15:34:03 VCS INFO V-16-1-10305 Resource ntfr (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (VCS initiated)
2012/04/30 15:34:03 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MIVDB01S
2012/04/30 15:34:03 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from LEAVING to EXITING
2012/04/30 15:34:03 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from EXITING to EXITED
2012/04/30 15:41:58 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/04/30 15:41:58 VCS INFO V-16-1-10196 Cluster logger started
2012/04/30 15:41:58 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode
2012/04/30 15:41:58 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/04/30 15:41:58 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0
2012/04/30 15:41:58 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00
2012/04/30 15:41:58 VCS NOTICE V-16-1-10115 Using GABSIM
2012/04/30 15:41:58 VCS NOTICE V-16-1-14032 Forming a one node cluster
2012/04/30 15:42:04 VCS NOTICE V-16-1-10619 'HAD' starting on: MIVDB01S
2012/04/30 15:42:04 VCS INFO V-16-1-51138 Number of processors configured on this system are 32
2012/04/30 15:42:04 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/04/30 15:42:06 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/04/30 15:42:06 VCS INFO V-16-1-10077 Received new cluster membership
2012/04/30 15:42:06 VCS NOTICE V-16-1-10112 System (MIVDB01S) - Membership: 0x1, DDNA: 0x0
2012/04/30 15:42:06 VCS NOTICE V-16-1-10086 System MIVDB01S (Node '0') is in Regular Membership - Membership: 0x1
2012/04/30 15:42:06 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2012/04/30 15:42:07 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/04/30 15:42:07 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from LOCAL_BUILD to RUNNING
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Mon Apr 30 15:42:07 2012
2012/04/30 15:42:07 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/04/30 15:42:08 VCS INFO V-16-1-10304 Resource BackupServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (First probe)
2012/04/30 15:42:08 VCS INFO V-16-1-10304 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (First probe)
2012/04/30 15:42:08 VCS INFO V-16-1-10304 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (First probe)
2012/04/30 15:42:08 VCS INFO V-16-1-10304 Resource ntfr (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (First probe)
2012/04/30 15:42:08 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (First probe)
2012/04/30 15:42:08 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MIVDB01S
2012/04/30 15:42:08 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService
2012/04/30 15:42:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes
2012/04/30 15:42:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 15:42:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource ntfr (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 15:42:08 VCS INFO V-16-6-15023 (MIVDB01S) dump_tunables:
2012/04/30 15:42:08 VCS INFO V-16-6-15002 (MIVDB01S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MIVDB01S 1 successfully
2012/04/30 15:42:08 VCS INFO V-16-10001-88 (MIVDB01S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully.
2012/04/30 15:42:08 VCS INFO V-16-1-10298 Resource ntfr (Owner: Unspecified, Group: ClusterService) is online on MIVDB01S (VCS initiated)
2012/04/30 15:42:08 VCS INFO V-16-1-51058 Received connection request from wide area connector
2012/04/30 15:42:08 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted
2012/04/30 15:42:08 VCS INFO V-16-1-50403 VCS engine on system MIVDB01S has established connection with wide area connector
2012/04/30 15:42:09 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MIVDB01S (First probe)
2012/04/30 15:42:09 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes
2012/04/30 15:42:09 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MIVDB01S
2012/04/30 15:42:09 VCS NOTICE V-16-1-10447 Group VVRService is online on system MIVDB01S
2012/04/30 15:42:09 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MIVDB01S (VCS initiated)
2012/04/30 15:42:09 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MIVDB01S
2012/04/30 15:42:10 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MIVDB01S
2012/04/30 15:42:10 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MIVDB01S as the system is not a part of AutoStartList attribute of the group.
2012/04/30 15:42:10 VCS INFO V-16-1-10304 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is offline on MIVDB01S (First probe)
2012/04/30 15:49:08 VCS INFO V-16-1-50133 User admin has logged in from ::ffff:172.22.8.132
2012/04/30 15:49:28 VCS INFO V-16-1-50135 User admin fired command: haconf -makerw from ::ffff:172.22.8.132
2012/04/30 15:49:28 VCS INFO V-16-1-50135 User admin fired command: hagrp -enableresources AppService from ::ffff:172.22.8.132
2012/04/30 15:49:51 VCS INFO V-16-1-50135 User admin fired command: hares -online RVGPrimary MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:50:05 VCS INFO V-16-1-50135 User admin fired command: hagrp -enableresources VVRService from ::ffff:172.22.8.132
2012/04/30 15:50:16 VCS INFO V-16-1-50135 User admin fired command: hagrp -enableresources VVRService from ::ffff:172.22.8.132
2012/04/30 15:50:32 VCS INFO V-16-1-50135 User admin fired command: hagrp -enableresources ClusterService from ::ffff:172.22.8.132
2012/04/30 15:51:05 VCS INFO V-16-1-50135 User admin fired command: hares -online RVGPrimary MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:51:49 VCS INFO V-16-1-50135 User admin fired command: hares -online DataFilesystem MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:52:04 VCS INFO V-16-1-50135 User admin fired command: hagrp -autoenable AppService MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:52:42 VCS INFO V-16-1-50135 User admin fired command: hares -online RVGPrimary MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:52:42 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 15:52:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource RVGPrimary (Owner: Unspecified, Group: AppService) on System MIVDB01S
2012/04/30 15:52:42 VCS INFO V-16-20010-72 (MIVDB01S) RVGPrimary:RVGPrimary:online:RVG datarvg is the primary; no action required
2012/04/30 15:52:44 VCS INFO V-16-1-10298 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MIVDB01S (VCS initiated)
2012/04/30 15:54:54 VCS INFO V-16-1-50135 User admin fired command: hares -online DataFilesystem MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:54:54 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 15:54:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource DataFilesystem (Owner: Unspecified, Group: AppService) on System MIVDB01S
2012/04/30 15:54:55 VCS INFO V-16-1-10298 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MIVDB01S (VCS initiated)
2012/04/30 15:55:08 VCS INFO V-16-1-50135 User admin fired command: hares -online DatabaseServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:55:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 15:55:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource DatabaseServer (Owner: Unspecified, Group: AppService) on System MIVDB01S
2012/04/30 15:55:09 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:su - sybase -c '. /opt/sybase/SYBASE.sh; /opt/sybase//install/startserver -f /opt/sybase//install/RUN_DBSVR' > /var/tmp/CheckStartResult.out 2>&1
2012/04/30 15:55:10 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY = CMDSY:
2012/04/30 15:55:10 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx CMD_KEY
2012/04/30 15:55:10 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx not VCS_KEY
2012/04/30 15:55:10 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx password_decrypt_cmd = /opt/VRTSvcs/sbin/vcsdecrypt dlgPlpEprPkpIpqPijEjfJg 2> /dev/null
2012/04/30 15:55:10 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:xxxxxxxxxxxxx Using password level EP001
2012/04/30 15:55:10 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:actual_passwd = Changeme123
2012/04/30 15:55:10 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:Sapswd = dlgPlpEprPkpIpqPijEjfJg
2012/04/30 15:55:20 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:ISQL_CMD: su - sybase -c ' . /opt/sybase/SYBASE.sh; isql -SDBSVR -Usa -PChangeme123 '
2012/04/30 15:56:13 VCS INFO V-16-20018-999 (MIVDB01S) Sybase:DatabaseServer:online:Recovery state is NOT_IN_RECOVERY - online script terminating
2012/04/30 15:56:14 VCS INFO V-16-20018-7 (MIVDB01S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/7599/psinfo
2012/04/30 15:56:14 VCS INFO V-16-1-10298 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MIVDB01S (VCS initiated)
2012/04/30 15:56:28 VCS INFO V-16-1-50135 User admin fired command: hares -online BackupServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:56:28 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 15:56:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource BackupServer (Owner: Unspecified, Group: AppService) on System MIVDB01S
2012/04/30 15:56:40 VCS INFO V-16-20018-7 (MIVDB01S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/942/psinfo
2012/04/30 15:56:40 VCS INFO V-16-1-10298 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MIVDB01S (VCS initiated)
2012/04/30 15:56:49 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:56:59 VCS INFO V-16-1-50135 User admin fired command: MSG_RES_PROBE NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:57:44 VCS INFO V-16-1-50135 User admin fired command: MSG_RES_PROBE NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:58:26 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:58:53 VCS INFO V-16-1-50135 User admin fired command: hares -action NMSServer program.vfd 0 MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:59:19 VCS INFO V-16-1-50135 User admin fired command: MSG_RES_PROBE NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 15:59:52 VCS INFO V-16-1-50135 User root fired command: hares -offline NMSServer MUIVDB01S from localhost
2012/04/30 16:00:02 VCS INFO V-16-1-50135 User root fired command: hares -offline NMSServer MIVDB01S from localhost
2012/04/30 16:02:53 VCS INFO V-16-1-50135 User admin fired command: MSG_RES_PROBE NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 16:03:17 VCS INFO V-16-1-50135 User admin fired command: haconf -dump from ::ffff:172.22.8.132
2012/04/30 16:05:03 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 16:05:19 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 16:05:25 VCS INFO V-16-1-50135 User admin fired command: MSG_RES_PROBE NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 16:07:18 VCS INFO V-16-1-50135 User admin fired command: hares -modify NMSServer Enabled 0 from ::ffff:172.22.8.132
2012/04/30 16:07:18 VCS NOTICE V-16-1-10438 Group AppService has been probed on system MIVDB01S
2012/04/30 16:07:18 VCS NOTICE V-16-1-10434 Group AppService will not start automatically as AutoStart attribute of the group is not set.
2012/04/30 16:07:26 VCS INFO V-16-1-50135 User admin fired command: hares -modify NMSServer Critical 0 from ::ffff:172.22.8.132
2012/04/30 16:07:29 VCS INFO V-16-1-50135 User admin fired command: hares -modify NMSServer Enabled 1 from ::ffff:172.22.8.132
2012/04/30 16:07:34 VCS INFO V-16-1-50135 User admin fired command: MSG_RES_PROBE NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 16:08:04 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 16:10:07 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MIVDB01S from localhost
2012/04/30 16:10:22 VCS INFO V-16-1-50135 User admin fired command: hares -online NMSServer MIVDB01S from ::ffff:172.22.8.132
2012/04/30 16:10:32 VCS INFO V-16-1-50135 User admin fired command: haconf -dump from ::ffff:172.22.8.132
2012/04/30 16:11:31 VCS INFO V-16-1-50135 User root fired command: hares -online NMSServer MIVDB01S from localhost
2012/04/30 16:32:37 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MIVDB01S from localhost
2012/04/30 16:34:03 VCS INFO V-16-1-50135 User root fired command: hagrp -enableresources AppService from localhost
2012/04/30 17:44:38 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_ALL_FORCE from localhost
2012/04/30 17:44:38 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from RUNNING to LEAVING
2012/04/30 17:44:38 VCS NOTICE V-16-1-10300 Initiating Offline of Resource wac (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 17:44:38 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ntfr (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 17:44:38 VCS NOTICE V-16-1-50402 VCS engine on system MIVDB01S has lost connection with wide area connector
2012/04/30 17:44:38 VCS INFO V-16-10001-88 (MIVDB01S) Application:wac:offline:Executed [/opt/VRTSvcs/bin/wacstop] successfully.
2012/04/30 17:44:39 VCS INFO V-16-1-10305 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (VCS initiated)
2012/04/30 17:44:40 VCS INFO V-16-1-10305 Resource ntfr (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (VCS initiated)
2012/04/30 17:44:40 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system MIVDB01S
2012/04/30 17:44:40 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from LEAVING to EXITING
2012/04/30 17:44:40 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from EXITING to EXITED
2012/04/30 17:45:35 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/04/30 17:45:35 VCS INFO V-16-1-10196 Cluster logger started
2012/04/30 17:45:35 VCS NOTICE V-16-1-11027 VCS engine startup arguments=-onenode
2012/04/30 17:45:35 VCS NOTICE V-16-1-11050 VCS engine version=5.1
2012/04/30 17:45:35 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.10.0
2012/04/30 17:45:35 VCS NOTICE V-16-1-11052 VCS engine pstamp=5.1.101.000-5.1SP1RP1-2011-01-20_18.30.00
2012/04/30 17:45:35 VCS NOTICE V-16-1-10115 Using GABSIM
2012/04/30 17:45:35 VCS NOTICE V-16-1-14032 Forming a one node cluster
2012/04/30 17:45:40 VCS NOTICE V-16-1-10619 'HAD' starting on: MIVDB01S
2012/04/30 17:45:40 VCS INFO V-16-1-51138 Number of processors configured on this system are 32
2012/04/30 17:45:40 VCS WARNING V-16-1-51140 In a multi-CPU system, configure an adequately high value for the ShutdownTimeout attribute. This ensures that when a system panics, its service groups successfully fail over to other systems. For more information, refer to the VCS Administrator's Guide
2012/04/30 17:45:40 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
2012/04/30 17:45:40 VCS INFO V-16-1-10077 Received new cluster membership
2012/04/30 17:45:40 VCS NOTICE V-16-1-10112 System (MIVDB01S) - Membership: 0x1, DDNA: 0x0
2012/04/30 17:45:40 VCS NOTICE V-16-1-10086 System MIVDB01S (Node '0') is in Regular Membership - Membership: 0x1
2012/04/30 17:45:40 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2012/04/30 17:45:41 VCS WARNING V-16-1-10030 UseFence=NONE. Hence do not need fencing
2012/04/30 17:45:41 VCS NOTICE V-16-1-10322 System MIVDB01S (Node '0') changed state from LOCAL_BUILD to RUNNING
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVG/RVGAgent for resource type RVG successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/RVGPrimary/RVGPrimaryAgent for resource type RVGPrimary successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Sybase/SybaseAgent for resource type Sybase successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent for resource type SybaseBk successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Mon Apr 30 17:45:41 2012
2012/04/30 17:45:41 VCS INFO V-16-1-10151 Invoking trigger to dump environment variables
2012/04/30 17:45:41 VCS INFO V-16-6-15023 (MIVDB01S) dump_tunables:
2012/04/30 17:45:41 VCS INFO V-16-6-15002 (MIVDB01S) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/dump_tunables MIVDB01S 1 successfully
2012/04/30 17:45:41 VCS INFO V-16-20018-7 (MIVDB01S) SybaseBk:BackupServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/backupserver -SDBSVR_back -e/opt/sybase/ASE-15_0/insta, PID = /proc/942/psinfo
2012/04/30 17:45:41 VCS INFO V-16-1-10297 Resource BackupServer (Owner: Unspecified, Group: AppService) is online on MIVDB01S (First probe)
2012/04/30 17:45:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 17:45:41 VCS INFO V-16-1-10297 Resource DataFilesystem (Owner: Unspecified, Group: AppService) is online on MIVDB01S (First probe)
2012/04/30 17:45:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 17:45:41 VCS INFO V-16-20018-7 (MIVDB01S) Sybase:DatabaseServer:monitor:Setting cookie for proc = /opt/sybase/ASE-15_0/bin/dataserver -sDBSVR -d/opt/sybase/data/lv_master -e/opt, PID = /proc/7599/psinfo
2012/04/30 17:45:41 VCS INFO V-16-1-10297 Resource DatabaseServer (Owner: Unspecified, Group: AppService) is online on MIVDB01S (First probe)
2012/04/30 17:45:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 17:45:41 VCS INFO V-16-1-10304 Resource ntfr (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (First probe)
2012/04/30 17:45:41 VCS INFO V-16-1-10304 Resource wac (Owner: Unspecified, Group: ClusterService) is offline on MIVDB01S (First probe)
2012/04/30 17:45:41 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system MIVDB01S
2012/04/30 17:45:41 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService
2012/04/30 17:45:41 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes
2012/04/30 17:45:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource wac (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 17:45:41 VCS NOTICE V-16-1-10301 Initiating Online of Resource ntfr (Owner: Unspecified, Group: ClusterService) on System MIVDB01S
2012/04/30 17:45:42 VCS INFO V-16-1-51058 Received connection request from wide area connector
2012/04/30 17:45:42 VCS INFO V-16-1-51059 Connection request from wide area connector has been accepted
2012/04/30 17:45:42 VCS INFO V-16-1-50403 VCS engine on system MIVDB01S has established connection with wide area connector
2012/04/30 17:45:42 VCS INFO V-16-10001-88 (MIVDB01S) Application:wac:online:Executed [/opt/VRTSvcs/bin/wacstart] successfully.
2012/04/30 17:45:42 VCS INFO V-16-1-10298 Resource ntfr (Owner: Unspecified, Group: ClusterService) is online on MIVDB01S (VCS initiated)
2012/04/30 17:45:42 VCS INFO V-16-1-10297 Resource datarvg (Owner: Unspecified, Group: VVRService) is online on MIVDB01S (First probe)
2012/04/30 17:45:42 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VVRService on all nodes
2012/04/30 17:45:42 VCS NOTICE V-16-1-10438 Group VVRService has been probed on system MIVDB01S
2012/04/30 17:45:42 VCS NOTICE V-16-1-10447 Group VVRService is online on system MIVDB01S
2012/04/30 17:45:43 VCS INFO V-16-1-10298 Resource wac (Owner: Unspecified, Group: ClusterService) is online on MIVDB01S (VCS initiated)
2012/04/30 17:45:43 VCS NOTICE V-16-1-10447 Group ClusterService is online on system MIVDB01S
2012/04/30 17:45:43 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MIVDB01S
2012/04/30 17:45:43 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MIVDB01S as the system is not a part of AutoStartList attribute of the group.
2012/04/30 17:45:43 VCS INFO V-16-1-10297 Resource RVGPrimary (Owner: Unspecified, Group: AppService) is online on MIVDB01S (First probe)
2012/04/30 17:45:43 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group AppService on all nodes
2012/04/30 17:48:26 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE NMSServer MIVDB01S from localhost
2012/04/30 17:49:00 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 17:52:25 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2012/04/30 17:52:26 VCS INFO V-16-1-50135 User root fired command: hagrp -enable AppService from localhost
2012/04/30 17:52:50 VCS INFO V-16-1-50135 User root fired command: hagrp -online AppService MIVDB01S from localhost
2012/04/30 19:37:38 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2012/04/30 23:37:39 VCS INFO V-16-1-53504 VCS Engine Alive message!!
---
I'm not a specialist about VCS, I'm starting to learn something about VCS, so I could not be enough accurate in my opinion.
bye
Tiziano
05-02-2012 03:48 AM
Can you run:
/opt/HWENGR/NMSApp/haMonitor
echo $?
and post the output of results. I suspect the monitor program is not exiting with 100 or 110, which it needs to.
A few other things:
Mike
05-02-2012 04:04 AM
Hello,
In addition to the information Mike suggested can you also check the permissions on the online,offline, clean and monitor
2012/04/30 08:02:53 VCS ERROR V-16-10001-87 (MIVDB01S) Application:NMSServer:offline:Failed to execute [/opt/HWENGR/NMSApp/haStop].
Regards
Mark
05-02-2012 04:19 AM
Hi Mike.
Thanks for your hint.
I investigate better and I understood that tha application NMSServer failed because an error in profile file of the user owner of the application
I fixed the profile file, and with the command /opt/HWENGR/NMSApp/haMonitor the Probe status changed. Then I started the application when the status changed from OFFLINE|UNKNOW to OFFLINE
Only a last question. Why it didn't work the command:
hares -probe NMSServer -sys MIVDB01S
and it worked correctly the command:
/opt/HWENGR/NMSApp/haMonitor
?
BR
Tiziano
05-02-2012 06:34 AM
Firstly, as already mentioned, and in response to the error message:
2012/04/30 08:02:53 VCS ERROR V-16-10001-87 (MIVDB01S) Application:NMSServer:offline:Failed to execute [/opt/HWENGR/NMSApp/haStop].
…you need to verify that you have correct permissions set for the various executable that you have defined in your NMSServer resource, IE:
$ ls -l /opt/HWENGR/NMSApp/haStart /opt/HWENGR/NMSApp/haStop /opt/HWENGR/NMSApp/haClean /opt/HWENGR/NMSApp/haMonitor
Make sure they are all executable.
On to your second question… There is a very big difference between running:
$ hares -probe NMSServer -sys MIVDB01S
and:
$ /opt/HWENGR/NMSApp/haMonitor
Running the "hares -probe NMSServer -sys MIVDB01S" command signals VCS to execute the program defined by the MonitorProgram attribute of the NMSServer resource, which I am sure you understand to be so. What you probably do not appreciate is that this causes the defined executable (in your case "/opt/HWENGR/NMSApp/haMonitor") to be run from within the root user's context (as all internal VCS processes are run from within the root user's context), AND from within an Environment that is almost certainly quite different to the Environment that is set when you run the "/opt/HWENGR/NMSApp/haMonitor" command interactively.
The short answer here is that you need to identify what Environment variables the "/opt/HWENGR/NMSApp/haMonitor" command relies upon in order to work correctly, and then you need to decide upon the best way to instantiate these Environment variables into the execution environment that will exist when VCS wants to probe the resource.
As the NMSServer resource is a VCS Type Application resource, you fortunately can use the EnvFile attribute to fulfil this requirement. I believe it is described well enough in the manual, but let us know if you can't figure it out.
--Kevin
05-02-2012 07:09 AM
Just to clarify, whether the online routine sources the profile depends on version of VCS and attributes used (from 5.1SP1 onwards).
In 5.1 (no SP1) , acording to 5.1 bundled agents guide, "su" is used for online and "su -" for offline, monitor and clean, but I would guess this is a typo and "su -" is used for online as well as I can't think of any reason why online would be different and looking at code on Linux, this is indeed the case - "su -" is used so profile is sourced.
Just realised, looking at logs you provided you are using 5.1SP1 and since attributes EnvFile and UseSUDash (new atributes for 5.1SP1) are not present in you extract from main.cf this means they are set as default which is:
Envfile = "" (Blank)
UseSUDash = 0
I would recommend you use the EnvFile to set your enviroment and leave UseSUDash set to 0. The reasons for this are in post https://www-secure.symantec.com/connect/ideas/vcs-agents-should-use-su-user-instead-su-user
Mike
05-03-2012 04:46 AM
Hi Mike,Kevin,Mark
Thanks a lot for your hints.
I cannot appreciate totally your considerations because I'm a dummy of VCS, unlikely.
In any case I can summarize my problem as follow:
I changed a parameter in .profile file of nmsuser owner of the resource NMSService
After that and after a reboot the resource was not probed and with the command:
/opt/HWENGR/NMSApp/haMonitor
I could understand that.
I fixed the problem i .profile file and with the command:
/opt/HWENGR/NMSApp/haMonitor
I had a correct answer:
12-05-02 13:11:01 || svc_adm query retrytimes start at 2012-05-02_13:11:01
12-05-02 13:11:01 || haMonitor:Success to get the process online retry count at 2012-05-02_13:11:01.
12-05-02 13:11:01 || svc_deploy query svc start at 2012-05-02_13:11:01
12-05-02 13:11:02 || haMonitor:Success to get the process start mode at 2012-05-02_13:11:02.
12-05-02 13:11:02 || svc_adm query status start at 2012-05-02_13:11:02
12-05-02 13:11:03 || haMonitor:Success to get the process running status at 2012-05-02_13:11:03.
12-05-02 13:11:03 || haMonitor:monitor finish, HA system set normal.
after that the flag Probed was changed from N to Y in 'hastatus -sum' command
Probably I could used also the command:
hares -probe NMSService -sys MIVDB01S
but only after fixed the .profile file
Also this command could change the flag Probed. Is it correct ? (please be patient if not)
I didn't touch any other parameters of VCS
BR
Tiziano
05-03-2012 05:19 AM
The flag Probed is an internal attibute that you cannot change manually and denotes if resource has been probed successfully (not just the probe has been run). Therefore, for the Probed flag to go from N to Y, the /opt/HWENGR/NMSApp/haMonitor has to return 100 for offline or 110 for online. This script is run by root and if you have "User" attribute set for the resource then it runs "su user -c script". As you have not set user attribute, script will run as root, which maybe ok if /opt/HWENGR/NMSApp/haStart does an su to nmsuser, or process should run as root.
Mike
05-03-2012 06:06 AM
Glad it is working again and you have determined what caused the issue ("changed a parameter in .profile file of nmsuser").
Because the NMSServer resource does not have the 'User' attribute set, we can conclude that the /opt/HWENGR/NMSApp/haMonitor command is reading nmsuser's .profile file itself (in one way or another).
Is the /opt/HWENGR/NMSApp/haMonitor command a binary or a scipt? If it is a script then you can easy view it to see how it reads the nmsserver user's .profile file contents.
Candidates are lines like:
source ~/nmsuser/.profile
--or--
. ~/nmsuser/.profile
--or-- it indirectly causes the ' ~/nmsuser/.profile' file to be read by doing:
su - user -c /path/to_some_NMS_server_status_command
It is a little harder to see if it reads the .profile file when it is a binary, but isn't really too hard (IE: man truss, dtrace, lsof, etc…)
Of course, either way, the /opt/HWENGR/NMSApp/haMonitor command could actually call or exec another command which actually reads in the .profile file.
No matter, it works now and you know what caused the problem.
Now, on the the point of the subject line...:
There is an important distinction that I do not think that you understand on how the resources state (and Probed flag) gets changed:
Running the /opt/HWENGR/NMSApp/haMonitor command manually did not cause the Probed state to go to 'Y'. The *only* thing that causes the Probed state to go to 'Y' is when the monitor entry point is run successfully (that is, from within the VCS framework). Running the /opt/HWENGR/NMSApp/haMonitor command outside of the VCS agent framework doesn't change anything within VCS's internal state tables.
What *had* to happen is that after you fixed the .profile file, at some point *before* you next looked at the Probe state of that resource, the VCS agent framework MonitorInterval timer was hit and the monitor entry point was run by VCS, as per normal.
As you may well guess, running the monitor entry point by manually executing the "hares -probe NMSService -sys MIVDB01S" command DOES update VCS internal states (as you are then using the VCS CLI to interface with the VCS agent framework).
-Kevin