Need help in Nic resource fault
We are experiencing below mentioned faults every now and then that affects our backup. setup is like we have BkupLan service group in which we have NIC resource (bkup_nic) and Phantom resource . Two service groups OSSfs and sybase1 have proxy resource monitoring this NIC resource. and both are configured with Ip resources Ossbak_ip1 and sybbak_ip1. The issue is Nic resource failed with below message and in turn proxy and ip resources failed. Then NIC comes back and proxy and ip come back too. Below is the configuration too Please advice why it goen offline/online frequently 2013/10/24 22:32:03 VCS WARNING V-16-10001-7506 (ossadm2) NIC:bkup_nic:monitor:Resource is offline. No Network Host could be reached 2013/10/24 22:32:03 VCS INFO V-16-2-13716 (ossadm2) Resource(bkup_nic): Output of the completed operation (monitor) ============================================== Broken Pipe ============================================== 2013/10/24 22:32:03 VCS ERROR V-16-1-10303 Resource bkup_nic (Owner: Unspecified, Group: BkupLan) is FAULTED (timed out) on sys ossadm2 2013/10/24 22:32:03 VCS INFO V-16-6-0 (ossadm2) resfault:(resfault) Invoked with arg0=ossadm2, arg1=bkup_nic, arg2=ONLINE 2013/10/24 22:32:03 VCS INFO V-16-0 (ossadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=ossadm2 ,arg2=bkup_nic2013/10/24 22:32:03 VCS INFO V-16-6-15002 (ossadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault ossadm2 bkup_nic ONLINE successfully 2013/10/24 22:32:20 VCS ERROR V-16-1-10303 Resource ossbak_p1 (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sys ossadm2 I chnage the Tolerance limit of NIC na dproxy agent to 2 root@ossadm1> hatype -display NIC NIC ToleranceLimit 2 oot@ossadm1> hatype -display Proxy Proxy ToleranceLimit 2 But still it failed like below :57:28 ossadm1 Had[18697]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource bkup_nic (Owner: Unspecified, Group: BkupLan) is FAULTED (timed out) on sys ossadm2 Nov 6 00:58:01 ossadm1 Had[18697]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource syb1bak_p1 (Owner: Unspecified, Group: Sybase1) is FAULTED (timed out) on sys ossadm2 Nov 6 00:58:01 ossadm1 Had[18697]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource ossbak_p1 (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sys ossadm2 It didnot say That syb1bak_p1 fauled and tolerance limit 2 not reached something like this to stay up for another monitor interval Do we need to set anything else so that proxy resoources dnt fault immediately and wait for sometime.Solved1.7KViews0likes2CommentsNic resource going down frequently
We are experiencing below mentioned faults every now and then that affects our backup. setup is like we have BkupLan service group in which we have NIC resource (bkup_nic) and Phantom resource . Two service groups OSSfs and sybase1 have proxy resource monitoring this NIC resource. and both are configured with Ip resources Ossbak_ip1 and sybbak_ip1. The issue is Nic resource failed with below message and in turn proxy and ip resources failed. Then NIC comes back and proxy and ip come back too. Below is the configuration too Please advice why it goen offline/online frequently 2013/10/24 22:32:03 VCS WARNING V-16-10001-7506 (ossadm2) NIC:bkup_nic:monitor:Resource is offline. No Network Host could be reached 2013/10/24 22:32:03 VCS INFO V-16-2-13716 (ossadm2) Resource(bkup_nic): Output of the completed operation (monitor) ============================================== Broken Pipe ============================================== 2013/10/24 22:32:03 VCS ERROR V-16-1-10303 Resource bkup_nic (Owner: Unspecified, Group: BkupLan) is FAULTED (timed out) on sys ossadm2 2013/10/24 22:32:03 VCS INFO V-16-6-0 (ossadm2) resfault:(resfault) Invoked with arg0=ossadm2, arg1=bkup_nic, arg2=ONLINE 2013/10/24 22:32:03 VCS INFO V-16-0 (ossadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=ossadm2 ,arg2=bkup_nic2013/10/24 22:32:03 VCS INFO V-16-6-15002 (ossadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault ossadm2 bkup_nic ONLINE successfully 2013/10/24 22:32:20 VCS ERROR V-16-1-10303 Resource ossbak_p1 (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sys ossadm2 2013/10/24 22:32:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ossbak_ip (Owner: Unspecified, Group: Ossfs) on System ossadm2 2013/10/24 22:32:20 VCS ERROR V-16-1-10303 Resource syb1bak_p1 (Owner: Unspecified, Group: Sybase1) is FAULTED (timed out) on sys ossadm2 2013/10/24 22:32:20 VCS INFO V-16-6-0 (ossadm2) resfault:(resfault) Invoked with arg0=ossadm2, arg1=ossbak_p1, arg2=ONLINE 2013/10/24 22:32:20 VCS INFO V-16-0 (ossadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=ossadm2 ,arg2=ossbak_p1 2013/10/24 22:32:20 VCS INFO V-16-6-0 (ossadm2) resfault:(resfault) Invoked with arg0=ossadm2, arg1=syb1bak_p1, arg2=ONLINE 2013/10/24 22:32:20 VCS INFO V-16-0 (ossadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=ossadm2 ,arg2=syb1bak_p1 2013/10/24 22:32:20 VCS INFO V-16-6-15002 (ossadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault ossadm2 ossbak_p1 ONLINE successfully 2013/10/24 22:32:20 VCS INFO V-16-6-15002 (ossadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault ossadm2 syb1bak_p1 ONLINE successfully 2013/10/24 22:32:22 VCS INFO V-16-1-10305 Resource ossbak_ip (Owner: Unspecified, Group: Ossfs) is offline on ossadm2 (VCS initiated) 2013/10/24 22:33:04 VCS INFO V-16-1-10299 Resource bkup_nic (Owner: Unspecified, Group: BkupLan) is online on ossadm2 (Not initiated by VCS) 2013/10/24 22:33:04 VCS NOTICE V-16-1-10447 Group BkupLan is online on system ossadm2 2013/10/24 22:33:20 VCS INFO V-16-1-10299 Resource syb1bak_p1 (Owner: Unspecified, Group: Sybase1) is online on ossadm2 (Not initiated by VCS) 2013/10/24 22:33:20 VCS INFO V-16-1-10299 Resource ossbak_p1 (Owner: Unspecified, Group: Ossfs) is online on ossadm2 (Not initiated by VCS) ==================================================================================== ossadm2{root} # hastatus -summ -- SYSTEM STATE -- System State Frozen A ossadm1 RUNNING 0 A ossadm2 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B BkupLan ossadm1 Y N ONLINE B BkupLan ossadm2 Y N ONLINE B DDCMon ossadm1 Y N ONLINE B DDCMon ossadm2 Y N ONLINE B Oss ossadm1 Y N OFFLINE B Oss ossadm2 Y N ONLINE B Ossfs ossadm1 Y N OFFLINE B Ossfs ossadm2 Y N ONLINE B PrivLan ossadm1 Y N ONLINE B PrivLan ossadm2 Y N ONLINE B PubLan ossadm1 Y N ONLINE B PubLan ossadm2 Y N ONLINE B StorLan ossadm1 Y N ONLINE B StorLan ossadm2 Y N ONLINE B Sybase1 ossadm1 Y N ONLINE B Sybase1 ossadm2 Y N OFFLINE you have mail ossadm2{root} # hagrp -resources BkupLan bkup_nic bkup_p ossadm2{root} # hagrp -resources bkup_nic VCS WARNING V-16-1-12130 Group bkup_nic does not exist ossadm2{root} # hagrp -resources BkupLan bkup_nic bkup_p ossadm2{root} # hares -display bkup_nic #Resource Attribute System Value bkup_nic Group global BkupLan bkup_nic Type global NIC bkup_nic AutoStart global 1 bkup_nic Critical global 0 bkup_nic Enabled global 1 bkup_nic LastOnline global ossadm2 bkup_nic MonitorOnly global 0 bkup_nic ResourceOwner global bkup_nic TriggerEvent global 0 bkup_nic ArgListValues ossadm1 Device 1 oce12 PingOptimize 1 1 NetworkHosts 1 127.0.0.1 Protocol 1 IPv4 NetworkType 1 ether ExclusiveIPZone 1 0 bkup_nic ArgListValues ossadm2 Device 1 oce12 PingOptimize 1 1 NetworkHosts 1 127.0.0.1 Protocol 1 IPv4 NetworkType 1 ether ExclusiveIPZone 1 0 bkup_nic ConfidenceLevel ossadm1 100 bkup_nic ConfidenceLevel ossadm2 100 bkup_nic ConfidenceMsg ossadm1 bkup_nic ConfidenceMsg ossadm2 bkup_nic Flags ossadm1 bkup_nic Flags ossadm2 bkup_nic IState ossadm1 not waiting bkup_nic IState ossadm2 not waiting bkup_nic MonitorMethod ossadm1 Traditional bkup_nic MonitorMethod ossadm2 Traditional bkup_nic Probed ossadm1 1 bkup_nic Probed ossadm2 1 bkup_nic Start ossadm1 0 bkup_nic Start ossadm2 0 bkup_nic State ossadm1 ONLINE bkup_nic State ossadm2 ONLINE bkup_nic ComputeStats global 0 bkup_nic ExclusiveIPZone global 0 bkup_nic NetworkHosts global 127.0.0.1 bkup_nic NetworkType global ether bkup_nic PingOptimize global 1 bkup_nic Protocol global IPv4 bkup_nic TriggerResStateChange global 0 bkup_nic ContainerInfo ossadm1 Type Name Enabled bkup_nic ContainerInfo ossadm2 Type Name Enabled bkup_nic Device ossadm1 oce12 bkup_nic Device ossadm2 oce12 bkup_nic MonitorTimeStats ossadm1 Avg 0 TS bkup_nic MonitorTimeStats ossadm2 Avg 0 TS bkup_nic ResourceInfo ossadm1 State Stale Msg TS bkup_nic ResourceInfo ossadm2 State Stale Msg TS ossadm2{root} # hares -display bkup_p #Resource Attribute System Value bkup_p Group global BkupLan bkup_p Type global Phantom bkup_p AutoStart global 1 bkup_p Critical global 1 bkup_p Enabled global 1 bkup_p LastOnline global ossadm1 bkup_p MonitorOnly global 0 bkup_p ResourceOwner global bkup_p TriggerEvent global 0 bkup_p ArgListValues ossadm1 "" bkup_p ArgListValues ossadm2 "" bkup_p ConfidenceLevel ossadm1 100 bkup_p ConfidenceLevel ossadm2 100 bkup_p ConfidenceMsg ossadm1 bkup_p ConfidenceMsg ossadm2 bkup_p Flags ossadm1 bkup_p Flags ossadm2 bkup_p IState ossadm1 not waiting bkup_p IState ossadm2 not waiting bkup_p MonitorMethod ossadm1 Traditional bkup_p MonitorMethod ossadm2 Traditional bkup_p Probed ossadm1 1 bkup_p Probed ossadm2 1 bkup_p Start ossadm1 1 bkup_p Start ossadm2 1 bkup_p State ossadm1 ONLINE bkup_p State ossadm2 ONLINE bkup_p ComputeStats global 0 bkup_p TriggerResStateChange global 0 bkup_p ContainerInfo ossadm1 Type Name Enabled bkup_p ContainerInfo ossadm2 Type Name Enabled bkup_p MonitorTimeStats ossadm1 Avg 0 TS bkup_p MonitorTimeStats ossadm2 Avg 0 TS bkup_p ResourceInfo ossadm1 State Stale Msg TS bkup_p ResourceInfo ossadm2 State Valid Msg TS ossadm2{root} # hagrp -resources Sybase1 sybasedg sybmaster_mount syblog_mount sybdata_mount pmsyblog_mount pmsybdata_mount fmsyblog_mount fmsybdata_mount dbdumps_mount syb1_ip syb1_p1 syb1bak_ip syb1bak_p1 masterdataservice masterdataservice_BACKUP stop_sybase ossadm2{root} # hares -display syb1bak_p1 #Resource Attribute System Value syb1bak_p1 Group global Sybase1 syb1bak_p1 Type global Proxy syb1bak_p1 AutoStart global 1 syb1bak_p1 Critical global 0 syb1bak_p1 Enabled global 1 syb1bak_p1 LastOnline global ossadm2 syb1bak_p1 MonitorOnly global 0 syb1bak_p1 ResourceOwner global syb1bak_p1 TriggerEvent global 0 syb1bak_p1 ArgListValues ossadm1 TargetResName 1 bkup_nic TargetSysName 1 "" TargetResName:Probed 1 1 TargetResName:State 1 2 syb1bak_p1 ArgListValues ossadm2 TargetResName 1 bkup_nic TargetSysName 1 "" TargetResName:Probed 1 1 TargetResName:State 1 2 syb1bak_p1 ConfidenceLevel ossadm1 0 syb1bak_p1 ConfidenceLevel ossadm2 0 syb1bak_p1 ConfidenceMsg ossadm1 syb1bak_p1 ConfidenceMsg ossadm2 syb1bak_p1 Flags ossadm1 syb1bak_p1 Flags ossadm2 syb1bak_p1 IState ossadm1 not waiting syb1bak_p1 IState ossadm2 not waiting syb1bak_p1 MonitorMethod ossadm1 Traditional syb1bak_p1 MonitorMethod ossadm2 Traditional syb1bak_p1 Probed ossadm1 1 syb1bak_p1 Probed ossadm2 1 syb1bak_p1 Start ossadm1 0 syb1bak_p1 Start ossadm2 0 syb1bak_p1 State ossadm1 ONLINE syb1bak_p1 State ossadm2 ONLINE syb1bak_p1 ComputeStats global 0 syb1bak_p1 ResourceInfo global State Stale Msg TS syb1bak_p1 TargetResName global bkup_nic syb1bak_p1 TargetSysName global syb1bak_p1 TriggerResStateChange global 0 syb1bak_p1 ContainerInfo ossadm1 Type Name Enabled syb1bak_p1 ContainerInfo ossadm2 Type Name Enabled syb1bak_p1 MonitorTimeStats ossadm1 Avg 0 TS syb1bak_p1 MonitorTimeStats ossadm2 Avg 0 TS ossadm2{root} # hares -display syb1bak_ip #Resource Attribute System Value syb1bak_ip Group global Sybase1 syb1bak_ip Type global IP syb1bak_ip AutoStart global 1 syb1bak_ip Critical global 0 syb1bak_ip Enabled global 1 syb1bak_ip LastOnline global ossadm1 syb1bak_ip MonitorOnly global 0 syb1bak_ip ResourceOwner global syb1bak_ip TriggerEvent global 0 syb1bak_ip ArgListValues ossadm1 Device 1 oce12 Address 1 10.41.78.138 NetMask 1 255.255.255.224 Options 1 "" ArpDelay 1 1 IfconfigTwice 1 1 RouteOptions 1 "" PrefixLen 1 0 ExclusiveIPZone 1 0 syb1bak_ip ArgListValues ossadm2 Device 1 oce12 Address 1 10.41.78.138 NetMask 1 255.255.255.224 Options 1 "" ArpDelay 1 1 IfconfigTwice 1 1 RouteOptions 1 "" PrefixLen 1 0 ExclusiveIPZone 1 0 syb1bak_ip ConfidenceLevel ossadm1 100 syb1bak_ip ConfidenceLevel ossadm2 0 syb1bak_ip ConfidenceMsg ossadm1 syb1bak_ip ConfidenceMsg ossadm2 syb1bak_ip Flags ossadm1 syb1bak_ip Flags ossadm2 syb1bak_ip IState ossadm1 not waiting syb1bak_ip IState ossadm2 not waiting syb1bak_ip MonitorMethod ossadm1 Traditional syb1bak_ip MonitorMethod ossadm2 Traditional syb1bak_ip Probed ossadm1 1 syb1bak_ip Probed ossadm2 1 syb1bak_ip Start ossadm1 1 syb1bak_ip Start ossadm2 0 syb1bak_ip State ossadm1 ONLINE syb1bak_ip State ossadm2 OFFLINE syb1bak_ip Address global 10.41.78.138 syb1bak_ip ArpDelay global 1 syb1bak_ip ComputeStats global 0 syb1bak_ip ExclusiveIPZone global 0 syb1bak_ip IfconfigTwice global 1 syb1bak_ip NetMask global 255.255.255.224 syb1bak_ip Options global syb1bak_ip PrefixLen global 0 syb1bak_ip ResourceInfo global State Stale Msg TS syb1bak_ip RouteOptions global syb1bak_ip TriggerResStateChange global 0 syb1bak_ip ContainerInfo ossadm1 Type Name Enabled syb1bak_ip ContainerInfo ossadm2 Type Name Enabled syb1bak_ip Device ossadm1 oce12 syb1bak_ip Device ossadm2 oce12 syb1bak_ip MonitorTimeStats ossadm1 Avg 0 TS syb1bak_ip MonitorTimeStats ossadm2 Avg 0 TS ossadm2{root} # hares -display ossbak_p1 #Resource Attribute System Value ossbak_p1 Group global Ossfs ossbak_p1 Type global Proxy ossbak_p1 AutoStart global 1 ossbak_p1 Critical global 0 ossbak_p1 Enabled global 1 ossbak_p1 LastOnline global ossadm2 ossbak_p1 MonitorOnly global 0 ossbak_p1 ResourceOwner global ossbak_p1 TriggerEvent global 0 ossbak_p1 ArgListValues ossadm1 TargetResName 1 bkup_nic TargetSysName 1 "" TargetResName:Probed 1 1 TargetResName:State 1 2 ossbak_p1 ArgListValues ossadm2 TargetResName 1 bkup_nic TargetSysName 1 "" TargetResName:Probed 1 1 TargetResName:State 1 2 ossbak_p1 ConfidenceLevel ossadm1 0 ossbak_p1 ConfidenceLevel ossadm2 0 ossbak_p1 ConfidenceMsg ossadm1 ossbak_p1 ConfidenceMsg ossadm2 ossbak_p1 Flags ossadm1 ossbak_p1 Flags ossadm2 ossbak_p1 IState ossadm1 not waiting ossbak_p1 IState ossadm2 not waiting ossbak_p1 MonitorMethod ossadm1 Traditional ossbak_p1 MonitorMethod ossadm2 Traditional ossbak_p1 Probed ossadm1 1 ossbak_p1 Probed ossadm2 1 ossbak_p1 Start ossadm1 0 ossbak_p1 Start ossadm2 0 ossbak_p1 State ossadm1 ONLINE ossbak_p1 State ossadm2 ONLINE ossbak_p1 ComputeStats global 0 ossbak_p1 ResourceInfo global State Stale Msg TS ossbak_p1 TargetResName global bkup_nic ossbak_p1 TargetSysName global ossbak_p1 TriggerResStateChange global 0 ossbak_p1 ContainerInfo ossadm1 Type Name Enabled ossbak_p1 ContainerInfo ossadm2 Type Name Enabled ossbak_p1 MonitorTimeStats ossadm1 Avg 0 TS ossbak_p1 MonitorTimeStats ossadm2 Avg 0 TS ossadm2{root} # hares -display ossbak_ip #Resource Attribute System Value ossbak_ip Group global Ossfs ossbak_ip Type global IP ossbak_ip AutoStart global 1 ossbak_ip Critical global 0 ossbak_ip Enabled global 1 ossbak_ip LastOnline global ossadm2 ossbak_ip MonitorOnly global 0 ossbak_ip ResourceOwner global ossbak_ip TriggerEvent global 0 ossbak_ip ArgListValues ossadm1 Device 1 oce12 Address 1 10.41.78.137 NetMask 1 255.255.255.224 Options 1 "" ArpDelay 1 1 IfconfigTwice 1 1 RouteOptions 1 "" PrefixLen 1 0 ExclusiveIPZone 1 0 ossbak_ip ArgListValues ossadm2 Device 1 oce12 Address 1 10.41.78.137 NetMask 1 255.255.255.224 Options 1 "" ArpDelay 1 1 IfconfigTwice 1 1 RouteOptions 1 "" PrefixLen 1 0 ExclusiveIPZone 1 0 ossbak_ip ConfidenceLevel ossadm1 0 ossbak_ip ConfidenceLevel ossadm2 100 ossbak_ip ConfidenceMsg ossadm1 ossbak_ip ConfidenceMsg ossadm2 ossbak_ip Flags ossadm1 ossbak_ip Flags ossadm2 ossbak_ip IState ossadm1 not waiting ossbak_ip IState ossadm2 not waiting ossbak_ip MonitorMethod ossadm1 Traditional ossbak_ip MonitorMethod ossadm2 Traditional ossbak_ip Probed ossadm1 1 ossbak_ip Probed ossadm2 1 ossbak_ip Start ossadm1 0 ossbak_ip Start ossadm2 1 ossbak_ip State ossadm1 OFFLINE ossbak_ip State ossadm2 ONLINE ossbak_ip Address global 10.41.78.137 ossbak_ip ArpDelay global 1 ossbak_ip ComputeStats global 0 ossbak_ip ExclusiveIPZone global 0 ossbak_ip IfconfigTwice global 1 ossbak_ip NetMask global 255.255.255.224 ossbak_ip Options global ossbak_ip PrefixLen global 0 ossbak_ip ResourceInfo global State Stale Msg TS ossbak_ip RouteOptions global ossbak_ip TriggerResStateChange global 0 ossbak_ip ContainerInfo ossadm1 Type Name Enabled ossbak_ip ContainerInfo ossadm2 Type Name Enabled ossbak_ip Device ossadm1 oce12 ossbak_ip Device ossadm2 oce12 ossbak_ip MonitorTimeStats ossadm1 Avg 0 TS ossbak_ip MonitorTimeStats ossadm2 Avg 0 TS ossadm2{root} # hatype -display NIC #Type Attribute Value NIC AEPTimeout 0 NIC ActionTimeout 30 NIC AgentClass TS NIC AgentDirectory NIC AgentFailedOn NIC AgentFile NIC AgentPriority 0 NIC AgentReplyTimeout 130 NIC AgentStartTimeout 60 NIC AlertOnMonitorTimeouts 0 NIC ArgList Device PingOptimize NetworkHosts Protocol NetworkType ExclusiveIPZone NIC AttrChangedTimeout 60 NIC CleanRetryLimit 0 NIC CleanTimeout 60 NIC CloseTimeout 60 NIC ConfInterval 600 NIC ContainerOpts RunInContainer 0 PassCInfo 1 NIC EPClass -1 NIC EPPriority -1 NIC ExternalStateChange NIC FaultOnMonitorTimeouts 4 NIC FaultPropagation 1 NIC FireDrill 0 NIC IMF Mode 0 MonitorFreq 1 RegisterRetryLimit 3 NIC IMFRegList NIC InfoInterval 0 NIC InfoTimeout 30 NIC LevelTwoMonitorFreq 0 NIC LogDbg NIC LogFileSize 33554432 NIC MonitorInterval 60 NIC MonitorStatsParam Frequency 0 ExpectedValue 100 ValueThreshold 100 AvgThreshold 40 NIC MonitorTimeout 120 NIC NumThreads 10 NIC OfflineMonitorInterval 60 NIC OfflineTimeout 300 NIC OfflineWaitLimit 0 NIC OnlineClass -1 NIC OnlinePriority -1 NIC OnlineRetryLimit 0 NIC OnlineTimeout 300 NIC OnlineWaitLimit 2 NIC OpenTimeout 60 NIC Operations None NIC RestartLimit 0 NIC ScriptClass TS NIC ScriptPriority 0 NIC SourceFile /etc/VRTSvcs/conf/config/types.cf NIC SupportedActions device.vfd clearNICFaultInZone NIC ToleranceLimit 0 NIC TypeOwnerSolved2.1KViews0likes4Comments
Veritas cluster issue
Hi We have a VCS of two node. Two file systems are under VCS configured from VXVM. One file system got 100% full.Now we have rebooted the cluster node & the mount points started to show,but after some time that file system got disappear,Checked with concerned DG that also got disabed,we tried to manually import & got succeded.But after starting the volumes we tried to mount the filesystem under the specified mount point. But the below error giving--- # mount /dev/vx/dsk/bgw1dg/vol01 /var/opt/BGw/Server1 mount: /dev/vx/dsk/bgw1dg/vol01 is not this fstype # Kindly suggest what could be the cause. regards...ArupSolved2.2KViews4likes5CommentsVcs cluster start issue
if we have two node cluster with nodes A and B. running service groups A on node A and B on node B In generally speaking, starting the 2nd node B when node A is runinng with VCS and ServiceGroup, will not cause any issue. lets say if we we have bring the A node by force seeding with /sbin/gabconfig -x and is running SG A and B. Now, we need to bring the second node up and form cluster. How should we go about in a most proper manner with minimal outage.Solved638Views0likes1CommentNeed to know the root cause for the service outage in vcs cluster
Hello Friends, I need your help in finding out what has caused the service outage and how can we overcome this situation logs from message file :--> /var/adm/messages ========================================== Sep 2 23:35:04 duadm2 genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (oce1) node 0 in trouble Sep 2 23:35:05 duadm2 genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (oce0) node 0 in trouble Sep 2 23:35:09 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 8 sec (2883648) Sep 2 23:35:10 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 8 sec (6644221) Sep 2 23:35:10 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 9 sec (2883648) Sep 2 23:35:11 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 9 sec (6644221) Sep 2 23:35:11 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 10 sec (2883648) Sep 2 23:35:12 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 10 sec (6644221) Sep 2 23:35:12 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 11 sec (2883648) Sep 2 23:35:13 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 11 sec (6644221) Sep 2 23:35:13 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 12 sec (2883648) Sep 2 23:35:14 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 12 sec (6644221) Sep 2 23:35:14 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 13 sec (2883648) Sep 2 23:35:15 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 13 sec (6644221) Sep 2 23:35:15 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 14 sec (2883648) Sep 2 23:35:15 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 0. 4 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 0. 3 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 0. 4 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 0. 2 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 14 sec (6644221) Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 0. 3 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 0. 1 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 0 inactive 15 sec (2883648) Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 0. 2 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 0. 0 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 0. 1 more to go. Sep 2 23:35:16 duadm2 genunix: [ID 205468 kern.notice] LLT INFO V-14-1-10509 link 2 (oce0) node 0 expired Sep 2 23:35:17 duadm2 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 0. 0 more to go. Sep 2 23:35:17 duadm2 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 0 inactive 15 sec (6644221) Sep 2 23:35:17 duadm2 genunix: [ID 205468 kern.notice] LLT INFO V-14-1-10509 link 0 (oce1) node 0 expired Sep 2 23:35:21 duadm2 genunix: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port h gen 288706 membership 01 Sep 2 23:35:21 duadm2 genunix: [ID 608499 kern.notice] GAB INFO V-15-1-20037 Port h gen 288706 jeopardy ;1 Sep 2 23:35:21 duadm2 genunix: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port a gen 288701 membership 01 Sep 2 23:35:21 duadm2 genunix: [ID 608499 kern.notice] GAB INFO V-15-1-20037 Port a gen 288701 jeopardy ;1 Sep 2 23:35:21 duadm2 genunix: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port b gen 288704 membership 01 Sep 2 23:35:21 duadm2 genunix: [ID 608499 kern.notice] GAB INFO V-15-1-20037 Port b gen 288704 jeopardy ;1 Sep 2 23:35:21 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS INFO V-16-1-10077 Received new cluster membership Sep 2 23:35:21 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10111 System duadm2 (Node '1') is in Regular and Jeopardy Memberships - Membership: 0x3 , Jeopardy: 0x2 Sep 2 23:56:50 duadm2 in.mpathd[6542]: [ID 585766 daemon.error] Cannot meet requested failure detection time of 10000 ms on (inet oce2) new failure detection time for group "stor_mnic" is 48426 ms Sep 2 23:56:57 duadm2 in.mpathd[6542]: [ID 594170 daemon.error] NIC failure detected on oce0 of group pub_mnic Sep 2 23:56:57 duadm2 in.mpathd[6542]: [ID 832587 daemon.error] Successfully failed over from NIC oce0 to NIC oce9 Sep 2 23:57:01 duadm2 in.mpathd[6542]: [ID 299542 daemon.error] NIC repair detected on oce0 of group pub_mnic Sep 2 23:57:01 duadm2 in.mpathd[6542]: [ID 620804 daemon.error] Successfully failed back to NIC oce0 Sep 2 23:57:10 duadm2 in.mpathd[6542]: [ID 594170 daemon.error] NIC failure detected on oce0 of group pub_mnic Sep 2 23:57:10 duadm2 in.mpathd[6542]: [ID 832587 daemon.error] Successfully failed over from NIC oce0 to NIC oce9 Sep 2 23:57:10 duadm2 ip: [ID 876157 kern.warning] WARNING: node 44:1e:a1:74:e5:46 is using our IP address 010.001.014.027 on oce9 Sep 2 23:57:10 duadm2 ip: [ID 876157 kern.warning] WARNING: node 44:1e:a1:74:e5:46 is using our IP address 010.001.014.030 on oce9 Sep 2 23:57:10 duadm2 ip: [ID 876157 kern.warning] WARNING: node 44:1e:a1:74:e5:46 is using our IP address 010.001.014.027 on oce9 Sep 2 23:57:10 duadm2 ip: [ID 876157 kern.warning] WARNING: node 44:1e:a1:74:e5:46 is using our IP address 010.001.014.030 on oce9 Sep 2 23:57:10 duadm2 ip: [ID 876157 kern.warning] WARNING: node 44:1e:a1:74:e5:46 is using our IP address 010.001.014.027 on oce9 Sep 2 23:57:10 duadm2 ip: [ID 876157 kern.warning] WARNING: node 44:1e:a1:74:e5:46 is using our IP address 010.001.014.030 on oce9 Sep 2 23:57:10 duadm2 ip: [ID 567813 kern.warning] WARNING: oce9:2 has duplicate address 010.001.014.027 (claimed by 44:1e:a1:74:e5:46); disabled Sep 2 23:57:10 duadm2 ip: [ID 567813 kern.warning] WARNING: oce9:1 has duplicate address 010.001.014.030 (claimed by 44:1e:a1:74:e5:46); disabled Sep 2 23:57:11 duadm2 in.mpathd[6542]: [ID 299542 daemon.error] NIC repair detected on oce0 of group pub_mnic Sep 2 23:57:11 duadm2 in.mpathd[6542]: [ID 620804 daemon.error] Successfully failed back to NIC oce0 Sep 2 23:57:20 duadm2 in.mpathd[6542]: [ID 168056 daemon.error] All Interfaces in group pub_mnic have failed Sep 2 23:57:27 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource pub_mnic (Owner: Unspecified, Group: PubLan) is FAULTED (timed out) on s ys duadm2 Sep 2 23:57:28 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group PubLan is faulted on system duadm2 Sep 2 23:57:29 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource pub_mnic (Owner: Unspecified, Group: PubLan) is FAULTED (timed out) on s ys duadm1 Sep 2 23:57:30 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource stor_mnic (Owner: Unspecified, Group: StorLan) is FAULTED (timed out) on sys duadm1 Sep 2 23:57:31 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group PubLan is faulted on system duadm1 Sep 2 23:57:53 duadm2 in.mpathd[6542]: [ID 594170 daemon.error] NIC failure detected on oce11 of group stor_mnic Sep 2 23:57:53 duadm2 in.mpathd[6542]: [ID 832587 daemon.error] Successfully failed over from NIC oce11 to NIC oce2 Sep 2 23:57:57 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource ossfs_p1 (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sy s duadm1 Sep 2 23:57:57 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource syb1_p1 (Owner: Unspecified, Group: Sybase1) is FAULTED (timed out) on s ys duadm1 Sep 2 23:57:58 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13067 (duadm1) Agent is calling clean for resource(ossfs_ip) because the resource becam e OFFLINE unexpectedly, on its own. Sep 2 23:57:59 duadm2 in.mpathd[6542]: [ID 168056 daemon.error] All Interfaces in group stor_mnic have failed Sep 2 23:58:02 duadm2 AgentFramework[6332]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13067 Thread(8) Agent is calling clean for resource(syb1_ip) because the res ource became OFFLINE unexpectedly, on its own. Sep 2 23:58:02 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13067 (duadm2) Agent is calling clean for resource(syb1_ip) because the resource became OFFLINE unexpectedly, on its own. Sep 2 23:58:02 duadm2 AgentFramework[6332]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13068 Thread(8) Resource(syb1_ip) - clean completed successfully. Sep 2 23:58:05 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource syb1_p1 (Owner: Unspecified, Group: Sybase1) is FAULTED (timed out) on s ys duadm2 Sep 2 23:58:05 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource ossfs_p1 (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sy s duadm2 Sep 2 23:58:08 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource stor_mnic (Owner: Unspecified, Group: StorLan) is FAULTED (timed out) on sys duadm2 Sep 2 23:58:20 duadm2 ldap_cachemgr[516]: [ID 293258 daemon.warning] libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP ser ver Sep 2 23:58:21 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13067 (duadm1) Agent is calling clean for resource(stor_p) because the resource became OFFLINE unexpectedly, on its own. Sep 2 23:58:21 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13070 (duadm1) Resource(stor_p) - clean not implemented. Sep 2 23:58:22 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group StorLan is faulted on system duadm1 Sep 2 23:58:24 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13027 (duadm1) Resource(cluster_maint) - monitor procedure did not complete within the expected time. Sep 2 23:58:30 duadm2 AgentFramework[6345]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13067 Thread(3) Agent is calling clean for resource(stor_p) because the reso urce became OFFLINE unexpectedly, on its own. Sep 2 23:58:30 duadm2 AgentFramework[6345]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13070 Thread(3) Resource(stor_p) - clean not implemented. Sep 2 23:58:30 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13067 (duadm2) Agent is calling clean for resource(stor_p) because the resource became OFFLINE unexpectedly, on its own. Sep 2 23:58:30 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13070 (duadm2) Resource(stor_p) - clean not implemented. Sep 2 23:58:31 duadm2 Had[6267]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group StorLan is faulted on system duadm2 Sep 2 23:59:19 duadm2 ldap_cachemgr[516]: [ID 293258 daemon.warning] libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP ser ver Sep 2 23:59:19 duadm2 ldap_cachemgr[516]: [ID 545954 daemon.error] libsldap: makeConnection: failed to open connection to 10.1.14.35 Sep 2 23:59:19 duadm2 ldap_cachemgr[516]: [ID 687686 daemon.warning] libsldap: Falling back to anonymous, non-SSL mode for __ns_ldap_getRootDSE. openConnection: si mple bind failed - Can't connect to the LDAP server Sep 2 23:59:20 duadm2 ldap_cachemgr[516]: [ID 293258 daemon.warning] libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP ser ver Sep 2 23:59:20 duadm2 ldap_cachemgr[516]: [ID 292100 daemon.warning] libsldap: could not remove 10.1.14.35 from servers list Sep 2 23:59:20 duadm2 ldap_cachemgr[516]: [ID 292100 daemon.warning] libsldap: could not remove 10.1.14.37 from servers list Sep 2 23:59:20 duadm2 ldap_cachemgr[516]: [ID 293258 daemon.warning] libsldap: Status: 7 Mesg: Session error no available conn. Sep 2 23:59:20 duadm2 ldap_cachemgr[516]: [ID 186574 daemon.error] Error: Unable to refresh profile:default: Session error no available conn. Sep 3 00:00:08 duadm2 svc.startd[9]: [ID 122153 daemon.warning] svc:/ericsson/eric_monitor/ddc:default: Method or service exit timed out. Killing contract 145156. Sep 3 00:00:08 duadm2 svc.startd[9]: [ID 636263 daemon.warning] svc:/ericsson/eric_monitor/ddc:default: Method "/opt/ERICddc/bin/ddc stop" failed due to signal KIL L. Sep 3 00:00:19 duadm2 ldap_cachemgr[516]: [ID 293258 daemon.warning] libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP ser ver Sep 3 00:00:19 duadm2 ldap_cachemgr[516]: [ID 545954 daemon.error] libsldap: makeConnection: failed to open connection to 10.1.14.35 Sep 3 00:00:19 duadm2 ldap_cachemgr[516]: [ID 293258 daemon.warning] libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP ser ver Sep 3 00:00:19 duadm2 ldap_cachemgr[516]: [ID 545954 daemon.error] libsldap: makeConnection: failed to open connection to 10.1.14.37 Sep 3 00:00:19 duadm2 ldap_cachemgr[516]: [ID 687686 daemon.warning] libsldap: Falling back to anonymous, non-SSL mode for __ns_ldap_getRootDSE. openConnection: si mple bind failed - Can't connect to the LDAP server Sep 3 00:00:20 duadm2 ldap_cachemgr[516]: [ID 293258 daemon.warning] libsldap: Status: 91 Mesg: openConnection: simple bind failed - Can't connect to the LDAP server Sep 3 00:00:20 duadm2 last message repeated 1 time Sep 3 00:00:20 duadm2 ldap_cachemgr[516]: [ID 545954 daemon.error] libsldap: makeConnection: failed to open connection to 10.1.14.35 Sep 3 00:00:20 duadm2 ldap_cachemgr[516]: [ID 545954 daemon.error] libsldap: makeConnection: failed to open connection to 10.1.14.37 Sep 3 00:00:20 duadm2 ldap_cachemgr[516]: [ID 687686 daemon.warning] libsldap: Falling back to anonymous, non-SSL mode for __ns_ldap_getRootDSE. openConnection: simple bind failed - Can't connect to the LDAP server Sep 3 00:00:20 duadm2 last message repeated 1 time ============================================================ engine logs :--> 2013/09/02 23:35:21 VCS INFO V-16-1-10077 Received new cluster membership 2013/09/02 23:35:21 VCS NOTICE V-16-1-10112 System (duadm1) - Membership: 0x3, DDNA: 0x2 2013/09/02 23:35:21 VCS ERROR V-16-1-10111 System duadm2 (Node '1') is in Regular and Jeopardy Memberships - Membership: 0x3, Jeopardy: 0x2 2013/09/02 23:35:25 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. Previous status = oce1 UP oce8 UP oce0 UP; Current status = oce1 DOWN oce8 UP oce0 DOWN. 2013/09/02 23:57:27 VCS ERROR V-16-1-10303 Resource pub_mnic (Owner: Unspecified, Group: PubLan) is FAULTED (timed out) on sys duadm2 2013/09/02 23:57:27 VCS NOTICE V-16-1-10300 Initiating Offline of Resource pub_p (Owner: Unspecified, Group: PubLan) on System duadm2 2013/09/02 23:57:27 VCS INFO V-16-6-0 (duadm2) resfault:(resfault) Invoked with arg0=duadm2, arg1=pub_mnic, arg2=ONLINE 2013/09/02 23:57:27 VCS INFO V-16-0 (duadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm2 ,arg2=pub_mnic 2013/09/02 23:57:27 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm2 pub_mnic ONLINE successfully 2013/09/02 23:57:28 VCS INFO V-16-1-10305 Resource pub_p (Owner: Unspecified, Group: PubLan) is offline on duadm2 (VCS initiated) 2013/09/02 23:57:28 VCS ERROR V-16-1-10205 Group PubLan is faulted on system duadm2 2013/09/02 23:57:28 VCS NOTICE V-16-1-10446 Group PubLan is offline on system duadm2 2013/09/02 23:57:28 VCS INFO V-16-1-10493 Evaluating duadm1 as potential target node for group PubLan 2013/09/02 23:57:28 VCS INFO V-16-1-50010 Group PubLan is online or faulted on system duadm1 2013/09/02 23:57:28 VCS INFO V-16-1-10493 Evaluating duadm2 as potential target node for group PubLan 2013/09/02 23:57:28 VCS INFO V-16-1-50010 Group PubLan is online or faulted on system duadm2 2013/09/02 23:57:28 VCS NOTICE V-16-1-10235 Restart is set for group PubLan. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2013/09/02 23:57:28 VCS INFO V-16-6-0 (duadm2) postoffline:(postoffline) Invoked with arg0=duadm2, arg1=PubLan 2013/09/02 23:57:28 VCS INFO V-16-2-13075 (duadm1) Resource(ossfs_ip) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(1). 2013/09/02 23:57:28 VCS INFO V-16-6-0 (duadm2) postoffline:Executing /ericsson/core/cluster/scripts/postoffline.sh with arg0=duadm2, arg1=PubLan 2013/09/02 23:57:28 VCS INFO V-16-6-0 (duadm2) postoffline.sh:PubLan:Nothing done 2013/09/02 23:57:28 VCS INFO V-16-6-0 (duadm2) postoffline:Completed execution of /ericsson/core/cluster/scripts/postoffline.sh for group PubLan 2013/09/02 23:57:29 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline duadm2 PubLan successfully 2013/09/02 23:57:29 VCS ERROR V-16-1-10303 Resource pub_mnic (Owner: Unspecified, Group: PubLan) is FAULTED (timed out) on sys duadm1 2013/09/02 23:57:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource pub_p (Owner: Unspecified, Group: PubLan) on System duadm1 2013/09/02 23:57:30 VCS INFO V-16-6-0 (duadm1) resfault:(resfault) Invoked with arg0=duadm1, arg1=pub_mnic, arg2=ONLINE 2013/09/02 23:57:30 VCS INFO V-16-0 (duadm1) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm1 ,arg2=pub_mnic 2013/09/02 23:57:30 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm1 pub_mnic ONLINE successfully 2013/09/02 23:57:31 VCS ERROR V-16-1-10303 Resource stor_mnic (Owner: Unspecified, Group: StorLan) is FAULTED (timed out) on sys duadm1 2013/09/02 23:57:31 VCS WARNING V-16-1-13310 Offlining parent group DDCMon on system duadm1 2013/09/02 23:57:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ddc_app (Owner: Unspecified, Group: DDCMon) on System duadm1 2013/09/02 23:57:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource hyperic_app (Owner: Unspecified, Group: DDCMon) on System duadm1 2013/09/02 23:57:31 VCS INFO V-16-1-10305 Resource pub_p (Owner: Unspecified, Group: PubLan) is offline on duadm1 (VCS initiated) 2013/09/02 23:57:31 VCS ERROR V-16-1-10205 Group PubLan is faulted on system duadm1 2013/09/02 23:57:31 VCS NOTICE V-16-1-10446 Group PubLan is offline on system duadm1 2013/09/02 23:57:31 VCS INFO V-16-1-10493 Evaluating duadm1 as potential target node for group PubLan 2013/09/02 23:57:31 VCS INFO V-16-1-50010 Group PubLan is online or faulted on system duadm1 2013/09/02 23:57:31 VCS INFO V-16-1-10493 Evaluating duadm2 as potential target node for group PubLan 2013/09/02 23:57:31 VCS INFO V-16-1-50010 Group PubLan is online or faulted on system duadm2 2013/09/02 23:57:31 VCS NOTICE V-16-1-10235 Restart is set for group PubLan. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2013/09/02 23:57:31 VCS INFO V-16-6-0 (duadm1) resfault:(resfault) Invoked with arg0=duadm1, arg1=stor_mnic, arg2=ONLINE 2013/09/02 23:57:31 VCS INFO V-16-6-0 (duadm1) postoffline:(postoffline) Invoked with arg0=duadm1, arg1=PubLan 2013/09/02 23:57:31 VCS INFO V-16-0 (duadm1) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm1 ,arg2=stor_mnic 2013/09/02 23:57:31 VCS INFO V-16-6-0 (duadm1) postoffline:Executing /ericsson/core/cluster/scripts/postoffline.sh with arg0=duadm1, arg1=PubLan 2013/09/02 23:57:31 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm1 stor_mnic ONLINE successfully 2013/09/02 23:57:31 VCS INFO V-16-6-0 (duadm1) postoffline.sh:PubLan:Nothing done 2013/09/02 23:57:31 VCS INFO V-16-6-0 (duadm1) postoffline:Completed execution of /ericsson/core/cluster/scripts/postoffline.sh for group PubLan 2013/09/02 23:57:31 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline duadm1 PubLan successfully 2013/09/02 23:57:32 VCS INFO V-16-2-13075 (duadm2) Resource(syb1_ip) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(1). 2013/09/02 23:57:41 VCS INFO V-16-2-13075 (duadm1) Resource(pms_ip) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(1). 2013/09/02 23:57:41 VCS INFO V-16-2-13075 (duadm1) Resource(snmp_ip) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(1). 2013/09/02 23:57:41 VCS INFO V-16-2-13075 (duadm1) Resource(cms_ip) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(1). 2013/09/02 23:57:57 VCS ERROR V-16-1-10303 Resource ossfs_p1 (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sys duadm1 2013/09/02 23:57:57 VCS NOTICE V-16-1-10300 Initiating Offline of Resource snmp_ip (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/02 23:57:57 VCS NOTICE V-16-1-10300 Initiating Offline of Resource pms_ip (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/02 23:57:57 VCS NOTICE V-16-1-10300 Initiating Offline of Resource stop_oss (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/02 23:57:57 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cms_ip (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/02 23:57:57 VCS ERROR V-16-1-10303 Resource syb1_p1 (Owner: Unspecified, Group: Sybase1) is FAULTED (timed out) on sys duadm1 2013/09/02 23:57:58 VCS INFO V-16-1-10305 Resource snmp_ip (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/02 23:57:58 VCS INFO V-16-1-10305 Resource pms_ip (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/02 23:57:58 VCS INFO V-16-1-10305 Resource cms_ip (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/02 23:57:58 VCS INFO V-16-6-0 (duadm1) resfault:(resfault) Invoked with arg0=duadm1, arg1=syb1_p1, arg2=ONLINE 2013/09/02 23:57:58 VCS INFO V-16-6-0 (duadm1) resfault:(resfault) Invoked with arg0=duadm1, arg1=ossfs_p1, arg2=ONLINE 2013/09/02 23:57:58 VCS INFO V-16-0 (duadm1) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm1 ,arg2=syb1_p1 2013/09/02 23:57:58 VCS INFO V-16-0 (duadm1) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm1 ,arg2=ossfs_p1 2013/09/02 23:57:58 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm1 syb1_p1 ONLINE successfully 2013/09/02 23:57:58 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm1 ossfs_p1 ONLINE successfully 2013/09/02 23:57:58 VCS ERROR V-16-2-13067 (duadm1) Agent is calling clean for resource(ossfs_ip) because the resource became OFFLINE unexpectedly, on its own. 2013/09/02 23:57:58 VCS INFO V-16-2-13068 (duadm1) Resource(ossfs_ip) - clean completed successfully. 2013/09/02 23:57:59 VCS INFO V-16-1-10307 Resource ossfs_ip (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (Not initiated by VCS) 2013/09/02 23:57:59 VCS INFO V-16-6-0 (duadm1) resfault:(resfault) Invoked with arg0=duadm1, arg1=ossfs_ip, arg2=ONLINE 2013/09/02 23:57:59 VCS INFO V-16-0 (duadm1) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm1 ,arg2=ossfs_ip 2013/09/02 23:57:59 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm1 ossfs_ip ONLINE successfully 2013/09/02 23:58:02 VCS ERROR V-16-2-13067 (duadm2) Agent is calling clean for resource(syb1_ip) because the resource became OFFLINE unexpectedly, on its own. 2013/09/02 23:58:02 VCS INFO V-16-2-13068 (duadm2) Resource(syb1_ip) - clean completed successfully. 2013/09/02 23:58:02 VCS INFO V-16-1-10307 Resource syb1_ip (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (Not initiated by VCS) 2013/09/02 23:58:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource stop_sybase (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/02 23:58:02 VCS INFO V-16-6-0 (duadm2) resfault:(resfault) Invoked with arg0=duadm2, arg1=syb1_ip, arg2=ONLINE 2013/09/02 23:58:02 VCS INFO V-16-10001-88 (duadm2) Application:stop_sybase:offline:Executed [/ericsson/core/cluster/scripts/stop_sybase.sh stop] successfully. 2013/09/02 23:58:02 VCS INFO V-16-0 (duadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm2 ,arg2=syb1_ip 2013/09/02 23:58:02 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm2 syb1_ip ONLINE successfully 2013/09/02 23:58:03 VCS INFO V-16-1-50135 User root fired command: hagrp -switch Oss duadm2 from localhost 2013/09/02 23:58:03 VCS NOTICE V-16-1-10208 Initiating switch of group Oss from system duadm1 to system duadm2 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource activemq (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource activemq_oss_loggingbroker (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource alex (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource apache (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cron (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fmria (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource glassfish (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource imgr_tomcat (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource restart_mc (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource syb_log_mon (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource syb_proc_mon (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource trapdist (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource vrsnt_log_mon (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:58:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/02 23:58:03 VCS INFO V-16-10001-88 (duadm1) Application:cron:offline:Executed [/ericsson/core/cluster/scripts/cron.sh stop] successfully. 2013/09/02 23:58:03 VCS INFO V-16-10001-88 (duadm1) Application:alex:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_3pp/alex:default] successfully. 2013/09/02 23:58:03 VCS INFO V-16-10001-88 (duadm1) Application:apache:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /network/http:apache2] successfully. 2013/09/02 23:58:05 VCS INFO V-16-10001-88 (duadm1) Application:restart_mc:offline:Executed [/ericsson/core/cluster/scripts/restart_mc.sh stop] successfully. 2013/09/02 23:58:05 VCS ERROR V-16-1-10303 Resource syb1_p1 (Owner: Unspecified, Group: Sybase1) is FAULTED (timed out) on sys duadm2 2013/09/02 23:58:05 VCS ERROR V-16-1-10303 Resource ossfs_p1 (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sys duadm2 2013/09/02 23:58:05 VCS INFO V-16-6-0 (duadm2) resfault:(resfault) Invoked with arg0=duadm2, arg1=syb1_p1, arg2=ONLINE 2013/09/02 23:58:05 VCS INFO V-16-6-0 (duadm2) resfault:(resfault) Invoked with arg0=duadm2, arg1=ossfs_p1, arg2=ONLINE 2013/09/02 23:58:05 VCS INFO V-16-0 (duadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm2 ,arg2=syb1_p1 2013/09/02 23:58:05 VCS INFO V-16-0 (duadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm2 ,arg2=ossfs_p1 2013/09/02 23:58:05 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm2 syb1_p1 ONLINE successfully 2013/09/02 23:58:05 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm2 ossfs_p1 ONLINE successfully 2013/09/02 23:58:05 VCS INFO V-16-1-10305 Resource stop_sybase (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/02 23:58:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource masterdataservice_BACKUP (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/02 23:58:08 VCS ERROR V-16-1-10303 Resource stor_mnic (Owner: Unspecified, Group: StorLan) is FAULTED (timed out) on sys duadm2 2013/09/02 23:58:08 VCS WARNING V-16-1-13310 Offlining parent group DDCMon on system duadm2 2013/09/02 23:58:08 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ddc_app (Owner: Unspecified, Group: DDCMon) on System duadm2 2013/09/02 23:58:08 VCS NOTICE V-16-1-10300 Initiating Offline of Resource hyperic_app (Owner: Unspecified, Group: DDCMon) on System duadm2 2013/09/02 23:58:08 VCS INFO V-16-6-0 (duadm2) resfault:(resfault) Invoked with arg0=duadm2, arg1=stor_mnic, arg2=ONLINE 2013/09/02 23:58:08 VCS INFO V-16-0 (duadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm2 ,arg2=stor_mnic 2013/09/02 23:58:08 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm2 stor_mnic ONLINE successfully 2013/09/02 23:58:21 VCS ERROR V-16-2-13067 (duadm1) Agent is calling clean for resource(stor_p) because the resource became OFFLINE unexpectedly, on its own. 2013/09/02 23:58:21 VCS ERROR V-16-2-13070 (duadm1) Resource(stor_p) - clean not implemented. 2013/09/02 23:58:22 VCS INFO V-16-1-10307 Resource stor_p (Owner: Unspecified, Group: StorLan) is offline on duadm1 (Not initiated by VCS) 2013/09/02 23:58:22 VCS ERROR V-16-1-10205 Group StorLan is faulted on system duadm1 2013/09/02 23:58:22 VCS NOTICE V-16-1-10446 Group StorLan is offline on system duadm1 2013/09/02 23:58:22 VCS INFO V-16-1-10493 Evaluating duadm1 as potential target node for group StorLan 2013/09/02 23:58:22 VCS INFO V-16-1-50010 Group StorLan is online or faulted on system duadm1 2013/09/02 23:58:22 VCS INFO V-16-1-10493 Evaluating duadm2 as potential target node for group StorLan 2013/09/02 23:58:22 VCS INFO V-16-1-50010 Group StorLan is online or faulted on system duadm2 2013/09/02 23:58:22 VCS NOTICE V-16-1-10235 Restart is set for group StorLan. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2013/09/02 23:58:22 VCS INFO V-16-6-0 (duadm1) resfault:(resfault) Invoked with arg0=duadm1, arg1=stor_p, arg2=ONLINE 2013/09/02 23:58:22 VCS INFO V-16-6-0 (duadm1) postoffline:(postoffline) Invoked with arg0=duadm1, arg1=StorLan 2013/09/02 23:58:22 VCS INFO V-16-0 (duadm1) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm1 ,arg2=stor_p 2013/09/02 23:58:22 VCS INFO V-16-6-0 (duadm1) postoffline:Executing /ericsson/core/cluster/scripts/postoffline.sh with arg0=duadm1, arg1=StorLan 2013/09/02 23:58:22 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm1 stor_p ONLINE successfully 2013/09/02 23:58:22 VCS INFO V-16-6-0 (duadm1) postoffline.sh:StorLan:Nothing done 2013/09/02 23:58:22 VCS INFO V-16-6-0 (duadm1) postoffline:Completed execution of /ericsson/core/cluster/scripts/postoffline.sh for group StorLan 2013/09/02 23:58:22 VCS INFO V-16-6-15002 (duadm1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline duadm1 StorLan successfully 2013/09/02 23:58:24 VCS ERROR V-16-2-13027 (duadm1) Resource(cluster_maint) - monitor procedure did not complete within the expected time. 2013/09/02 23:58:30 VCS ERROR V-16-2-13067 (duadm2) Agent is calling clean for resource(stor_p) because the resource became OFFLINE unexpectedly, on its own. 2013/09/02 23:58:30 VCS ERROR V-16-2-13070 (duadm2) Resource(stor_p) - clean not implemented. 2013/09/02 23:58:31 VCS INFO V-16-1-10307 Resource stor_p (Owner: Unspecified, Group: StorLan) is offline on duadm2 (Not initiated by VCS) 2013/09/02 23:58:31 VCS ERROR V-16-1-10205 Group StorLan is faulted on system duadm2 2013/09/02 23:58:31 VCS NOTICE V-16-1-10446 Group StorLan is offline on system duadm2 2013/09/02 23:58:31 VCS INFO V-16-1-10493 Evaluating duadm1 as potential target node for group StorLan 2013/09/02 23:58:31 VCS INFO V-16-1-50010 Group StorLan is online or faulted on system duadm1 2013/09/02 23:58:31 VCS INFO V-16-1-10493 Evaluating duadm2 as potential target node for group StorLan 2013/09/02 23:58:31 VCS INFO V-16-1-50010 Group StorLan is online or faulted on system duadm2 2013/09/02 23:58:31 VCS NOTICE V-16-1-10235 Restart is set for group StorLan. Group will be brought online if fault on persistent resource clears. If group is brought online anywhere else from AutoStartList or manually, then Restart will be reset 2013/09/02 23:58:31 VCS INFO V-16-6-0 (duadm2) resfault:(resfault) Invoked with arg0=duadm2, arg1=stor_p, arg2=ONLINE 2013/09/02 23:58:31 VCS INFO V-16-6-0 (duadm2) postoffline:(postoffline) Invoked with arg0=duadm2, arg1=StorLan 2013/09/02 23:58:31 VCS INFO V-16-0 (duadm2) resfault:(resfault.sh) Invoked with arg0=/ericsson/core/cluster/scripts/resfault.sh, arg1=duadm2 ,arg2=stor_p 2013/09/02 23:58:31 VCS INFO V-16-6-0 (duadm2) postoffline:Executing /ericsson/core/cluster/scripts/postoffline.sh with arg0=duadm2, arg1=StorLan 2013/09/02 23:58:31 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault duadm2 stor_p ONLINE successfully 2013/09/02 23:58:31 VCS INFO V-16-6-0 (duadm2) postoffline.sh:StorLan:Nothing done 2013/09/02 23:58:31 VCS INFO V-16-6-0 (duadm2) postoffline:Completed execution of /ericsson/core/cluster/scripts/postoffline.sh for group StorLan 2013/09/02 23:58:31 VCS INFO V-16-6-15002 (duadm2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline duadm2 StorLan successfully 2013/09/02 23:59:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/02 23:59:33 VCS INFO V-16-10001-88 (duadm1) Application:syb_log_mon:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_3pp/sybase_log_monitor:default] successfully. 2013/09/02 23:59:33 VCS INFO V-16-10001-88 (duadm1) Application:syb_proc_mon:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_3pp/sybase_process_monitor:default] successfully. 2013/09/02 23:59:34 VCS INFO V-16-10001-88 (duadm1) Application:fmria:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_ep/riadaemon:default] successfully. 2013/09/02 23:59:35 VCS INFO V-16-10001-88 (duadm1) Application:trapdist:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_ep/trapd:default] successfully. 2013/09/02 23:59:35 VCS INFO V-16-10001-88 (duadm1) Application:vrsnt_log_mon:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_3pp/versant_log_monitor:default] successfully. 2013/09/02 23:59:36 VCS INFO V-16-1-10305 Resource alex (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:37 VCS INFO V-16-10001-88 (duadm1) Application:imgr_tomcat:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_3pp/tomcat:default] successfully. 2013/09/02 23:59:37 VCS INFO V-16-1-10305 Resource apache (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:38 VCS INFO V-16-1-10305 Resource cron (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:40 VCS INFO V-16-2-13075 (duadm1) Resource(tomcat) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2). 2013/09/02 23:59:50 VCS INFO V-16-1-10305 Resource restart_mc (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource smssr (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/02 23:59:50 VCS INFO V-16-1-10305 Resource syb_proc_mon (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:52 VCS INFO V-16-1-10305 Resource syb_log_mon (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:52 VCS INFO V-16-1-10305 Resource trapdist (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:52 VCS INFO V-16-1-10305 Resource fmria (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:54 VCS INFO V-16-1-10305 Resource vrsnt_log_mon (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/02 23:59:55 VCS INFO V-16-10001-88 (duadm1) Application:smssr:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/ossrc/ssr:default] successfully. 2013/09/02 23:59:57 VCS INFO V-16-1-10305 Resource imgr_tomcat (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:00:00 VCS INFO V-16-1-10305 Resource smssr (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:00:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource supervisor (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:00:03 VCS INFO V-16-10001-88 (duadm1) Application:supervisor:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/ossrc/ssrProcessSupervisor:default] successfully. 2013/09/03 00:00:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:00:06 VCS INFO V-16-1-10305 Resource supervisor (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:00:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource tbs (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:00:11 VCS INFO V-16-2-13075 (duadm1) Resource(tomcat) has reported unexpected OFFLINE 2 times, which is still within the ToleranceLimit(2). 2013/09/03 00:01:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:01:08 VCS ERROR V-16-2-13067 (duadm1) Agent is calling clean for resource(tomcat) because the resource became OFFLINE unexpectedly, on its own. 2013/09/03 00:01:08 VCS INFO V-16-2-13068 (duadm1) Resource(tomcat) - clean completed successfully. 2013/09/03 00:01:08 VCS ERROR V-16-2-13073 (duadm1) Resource(tomcat) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 2) the resource. 2013/09/03 00:01:08 VCS INFO V-16-10001-88 (duadm1) Application:tomcat:online:Executed [/ericsson/core/cluster/scripts/svc.sh start /ericsson/eric_3pp/tomcat:default] successfully. 2013/09/03 00:01:09 VCS INFO V-16-2-13716 (duadm1) Resource(tomcat): Output of the completed operation (online) ============================================== svcadm: Instance "svc:/ericsson/eric_3pp/tomcat:default" is not in a maintenance or degraded state. ============================================== 2013/09/03 00:01:11 VCS NOTICE V-16-2-13076 (duadm1) Agent has successfully restarted resource(tomcat). 2013/09/03 00:01:11 VCS INFO V-16-1-55031 Resource tomcat in online state received recurring online message on system duadm1 2013/09/03 00:01:35 VCS INFO V-16-10001-88 (duadm1) Application:activemq_oss_loggingbroker:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_3pp/activemq_oss_loggingbroker:default] successfully. 2013/09/03 00:01:35 VCS INFO V-16-10001-88 (duadm1) Application:activemq:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_3pp/activemq:default] successfully. 2013/09/03 00:01:36 VCS INFO V-16-2-13716 (duadm1) Resource(activemq): Output of the completed operation (offline) ============================================== svcadm: Instance "svc:/ericsson/eric_3pp/activemq:default" is in maintenance state. ============================================== 2013/09/03 00:01:36 VCS INFO V-16-2-13716 (duadm1) Resource(activemq_oss_loggingbroker): Output of the completed operation (offline) ============================================== svcadm: Instance "svc:/ericsson/eric_3pp/activemq_oss_loggingbroker:default" is in maintenance state. ============================================== 2013/09/03 00:01:38 VCS INFO V-16-1-10305 Resource activemq (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:01:38 VCS INFO V-16-1-10305 Resource activemq_oss_loggingbroker (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:01:39 VCS ERROR V-16-20018-25 (duadm2) SybaseBk:masterdataservice_BACKUP:offline:Sybase Backup service masterdataservice_BACKUP cannot be stopped by isql - shutdown 2013/09/03 00:01:39 VCS ERROR V-16-20018-17 (duadm2) SybaseBk:masterdataservice_BACKUP:offline:clean scripts should do the process kill sequence 2013/09/03 00:01:39 VCS INFO V-16-2-13716 (duadm2) Resource(masterdataservice_BACKUP): Output of the completed operation (offline) ============================================== Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. ============================================== 2013/09/03 00:01:40 VCS ERROR V-16-2-13064 (duadm2) Agent is calling clean for resource(masterdataservice_BACKUP) because the resource is up even after offline completed. 2013/09/03 00:01:40 VCS ERROR V-16-20018-20 (duadm2) SybaseBk:masterdataservice_BACKUP:clean:kill -15 of 16419 2013/09/03 00:02:01 VCS INFO V-16-2-13068 (duadm2) Resource(masterdataservice_BACKUP) - clean completed successfully. 2013/09/03 00:02:01 VCS WARNING V-16-20018-301 (duadm2) SybaseBk:masterdataservice_BACKUP:monitor:Open for backupserver failed, setting cookie to NULL 2013/09/03 00:02:01 VCS INFO V-16-1-10305 Resource masterdataservice_BACKUP (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:02:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource masterdataservice (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:02:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:02:32 VCS WARNING V-16-2-13011 (duadm1) Resource(hyperic_app): offline procedure did not complete within the expected time. 2013/09/03 00:02:32 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(hyperic_app) because offline did not complete within the expected time. 2013/09/03 00:02:32 VCS WARNING V-16-2-13011 (duadm1) Resource(ddc_app): offline procedure did not complete within the expected time. 2013/09/03 00:02:32 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(ddc_app) because offline did not complete within the expected time. 2013/09/03 00:02:32 VCS INFO V-16-2-13068 (duadm1) Resource(hyperic_app) - clean completed successfully. 2013/09/03 00:02:32 VCS INFO V-16-2-13068 (duadm1) Resource(ddc_app) - clean completed successfully. 2013/09/03 00:02:34 VCS INFO V-16-1-10305 Resource hyperic_app (Owner: Unspecified, Group: DDCMon) is offline on duadm1 (VCS initiated) 2013/09/03 00:02:34 VCS INFO V-16-1-10305 Resource ddc_app (Owner: Unspecified, Group: DDCMon) is offline on duadm1 (VCS initiated) 2013/09/03 00:02:34 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ddc_mount (Owner: Unspecified, Group: DDCMon) on System duadm1 2013/09/03 00:03:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:03:05 VCS WARNING V-16-2-13011 (duadm1) Resource(glassfish): offline procedure did not complete within the expected time. 2013/09/03 00:03:05 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(glassfish) because offline did not complete within the expected time. 2013/09/03 00:03:07 VCS INFO V-16-10001-88 (duadm1) Application:tbs:offline:Executed [/ericsson/core/cluster/scripts/svc.sh stop /ericsson/eric_ep/TBS:default] successfully. 2013/09/03 00:03:08 VCS INFO V-16-2-13716 (duadm1) Resource(tbs): Output of the completed operation (offline) ============================================== svcadm: Instance "svc:/ericsson/eric_ep/TBS:default" is in maintenance state. ============================================== 2013/09/03 00:03:08 VCS WARNING V-16-2-13011 (duadm2) Resource(hyperic_app): offline procedure did not complete within the expected time. 2013/09/03 00:03:08 VCS WARNING V-16-2-13011 (duadm2) Resource(ddc_app): offline procedure did not complete within the expected time. 2013/09/03 00:03:08 VCS ERROR V-16-2-13063 (duadm2) Agent is calling clean for resource(hyperic_app) because offline did not complete within the expected time. 2013/09/03 00:03:08 VCS ERROR V-16-2-13063 (duadm2) Agent is calling clean for resource(ddc_app) because offline did not complete within the expected time. 2013/09/03 00:03:08 VCS INFO V-16-2-13068 (duadm2) Resource(ddc_app) - clean completed successfully. 2013/09/03 00:03:08 VCS INFO V-16-2-13068 (duadm2) Resource(hyperic_app) - clean completed successfully. 2013/09/03 00:03:10 VCS INFO V-16-1-10305 Resource tbs (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ext_notif (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ext_nsa (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource log_service (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource notif (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource nsa (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oad (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource opendj (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource osagent (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource rmi_reg (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource rmi_reg_ext (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource sb_nsa (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource sentinel (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource time_service (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource tomcat (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:03:10 VCS INFO V-16-1-10305 Resource ddc_app (Owner: Unspecified, Group: DDCMon) is offline on duadm2 (VCS initiated) 2013/09/03 00:03:10 VCS INFO V-16-1-10305 Resource hyperic_app (Owner: Unspecified, Group: DDCMon) is offline on duadm2 (VCS initiated) 2013/09/03 00:03:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ddc_mount (Owner: Unspecified, Group: DDCMon) on System duadm2 2013/09/03 00:04:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:04:06 VCS ERROR V-16-2-13006 (duadm1) Resource(glassfish): clean procedure did not complete within the expected time. 2013/09/03 00:05:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:06:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:07:01 VCS INFO V-16-2-13003 (duadm2) Resource(masterdataservice): Output of the timed out operation (offline) Password: CT-LIBRARY error: ct_connect(): user api layer: internal Client Library error: Read from the server has timed out. 2013/09/03 00:07:01 VCS WARNING V-16-2-13011 (duadm2) Resource(masterdataservice): offline procedure did not complete within the expected time. 2013/09/03 00:07:01 VCS ERROR V-16-2-13063 (duadm2) Agent is calling clean for resource(masterdataservice) because offline did not complete within the expected time. 2013/09/03 00:07:02 VCS ERROR V-16-20018-20 (duadm2) Sybase:masterdataservice:clean:kill -15 of 16142 2013/09/03 00:07:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:07:22 VCS INFO V-16-20018-51 (duadm2) Sybase:masterdataservice:clean:Could not open directory [/var/tmp/sybase_shm/masterdataservice] for read [No such file or directory] 2013/09/03 00:07:22 VCS NOTICE V-16-20018-78 (duadm2) Sybase:masterdataservice:clean:Cannot remove Shared Memory. 2013/09/03 00:07:22 VCS INFO V-16-2-13068 (duadm2) Resource(masterdataservice) - clean completed successfully. 2013/09/03 00:07:23 VCS WARNING V-16-20018-301 (duadm2) Sybase:masterdataservice:monitor:Open for dataserver failed, setting cookie to NULL 2013/09/03 00:07:23 VCS INFO V-16-1-10305 Resource masterdataservice (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource syb1bak_ip (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource dbdumps_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fmsybdata_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fmsyblog_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource pmsybdata_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource pmsyblog_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource sybdata_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource syblog_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource sybmaster_mount (Owner: Unspecified, Group: Sybase1) on System duadm2 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource dbdumps_mount (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource fmsyblog_mount (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource pmsybdata_mount (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource pmsyblog_mount (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource sybdata_mount (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource fmsybdata_mount (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource syblog_mount (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:25 VCS INFO V-16-1-10305 Resource syb1bak_ip (Owner: Unspecified, Group: Sybase1) is offline on duadm2 (VCS initiated) 2013/09/03 00:07:35 VCS WARNING V-16-2-13011 (duadm1) Resource(ddc_mount): offline procedure did not complete within the expected time. 2013/09/03 00:07:35 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(ddc_mount) because offline did not complete within the expected time. 2013/09/03 00:07:35 VCS INFO V-16-10001-5530 (duadm1) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:08:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:08:11 VCS WARNING V-16-2-13011 (duadm1) Resource(ext_nsa): offline procedure did not complete within the expected time. 2013/09/03 00:08:11 VCS WARNING V-16-2-13011 (duadm1) Resource(notif): offline procedure did not complete within the expected time. 2013/09/03 00:08:11 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(notif) because offline did not complete within the expected time. 2013/09/03 00:08:11 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(ext_nsa) because offline did not complete within the expected time. 2013/09/03 00:08:11 VCS WARNING V-16-2-13011 (duadm1) Resource(oad): offline procedure did not complete within the expected time. 2013/09/03 00:08:11 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(oad) because offline did not complete within the expected time. 2013/09/03 00:08:11 VCS INFO V-16-2-13068 (duadm1) Resource(oad) - clean completed successfully. 2013/09/03 00:08:11 VCS INFO V-16-2-13068 (duadm1) Resource(notif) - clean completed successfully. 2013/09/03 00:08:11 VCS WARNING V-16-2-13011 (duadm1) Resource(ext_notif): offline procedure did not complete within the expected time. 2013/09/03 00:08:11 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(ext_notif) because offline did not complete within the expected time. 2013/09/03 00:08:11 VCS WARNING V-16-2-13011 (duadm1) Resource(log_service): offline procedure did not complete within the expected time. 2013/09/03 00:08:11 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(log_service) because offline did not complete within the expected time. 2013/09/03 00:08:11 VCS WARNING V-16-2-13011 (duadm1) Resource(nsa): offline procedure did not complete within the expected time. 2013/09/03 00:08:11 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(nsa) because offline did not complete within the expected time. 2013/09/03 00:08:11 VCS INFO V-16-2-13068 (duadm1) Resource(nsa) - clean completed successfully. 2013/09/03 00:08:11 VCS INFO V-16-2-13068 (duadm1) Resource(ext_nsa) - clean completed successfully. 2013/09/03 00:08:12 VCS WARNING V-16-2-13011 (duadm2) Resource(ddc_mount): offline procedure did not complete within the expected time. 2013/09/03 00:08:12 VCS ERROR V-16-2-13063 (duadm2) Agent is calling clean for resource(ddc_mount) because offline did not complete within the expected time. 2013/09/03 00:08:12 VCS INFO V-16-10001-5530 (duadm2) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:08:12 VCS WARNING V-16-2-13011 (duadm1) Resource(osagent): offline procedure did not complete within the expected time. 2013/09/03 00:08:12 VCS WARNING V-16-2-13011 (duadm1) Resource(opendj): offline procedure did not complete within the expected time. 2013/09/03 00:08:12 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(osagent) because offline did not complete within the expected time. 2013/09/03 00:08:12 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(opendj) because offline did not complete within the expected time. 2013/09/03 00:08:12 VCS INFO V-16-2-13068 (duadm1) Resource(osagent) - clean completed successfully. 2013/09/03 00:08:36 VCS ERROR V-16-2-13006 (duadm1) Resource(ddc_mount): clean procedure did not complete within the expected time. 2013/09/03 00:09:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:09:07 VCS WARNING V-16-2-13011 (duadm1) Resource(rmi_reg_ext): offline procedure did not complete within the expected time. 2013/09/03 00:09:07 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(rmi_reg_ext) because offline did not complete within the expected time. 2013/09/03 00:09:07 VCS INFO V-16-2-13068 (duadm1) Resource(rmi_reg_ext) - clean completed successfully. 2013/09/03 00:09:11 VCS ERROR V-16-2-13006 (duadm1) Resource(ext_notif): clean procedure did not complete within the expected time. 2013/09/03 00:09:11 VCS ERROR V-16-2-13006 (duadm1) Resource(log_service): clean procedure did not complete within the expected time. 2013/09/03 00:09:13 VCS ERROR V-16-2-13006 (duadm2) Resource(ddc_mount): clean procedure did not complete within the expected time. 2013/09/03 00:09:13 VCS ERROR V-16-2-13006 (duadm1) Resource(opendj): clean procedure did not complete within the expected time. 2013/09/03 00:09:28 VCS INFO V-16-1-10305 Resource oad (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:09:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource gui_nsa (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:09:28 VCS INFO V-16-1-10305 Resource nsa (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:09:28 VCS INFO V-16-1-10305 Resource notif (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:09:30 VCS INFO V-16-1-10305 Resource rmi_reg_ext (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:09:30 VCS INFO V-16-1-10305 Resource osagent (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:09:30 VCS INFO V-16-1-10305 Resource ext_nsa (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:09:32 VCS INFO V-16-1-10305 Resource glassfish (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:09:36 VCS ERROR V-16-2-13077 (duadm1) Agent is unable to offline resource(ddc_mount). Administrative intervention may be required. 2013/09/03 00:09:36 VCS INFO V-16-10001-5530 (duadm1) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:10:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:10:12 VCS ERROR V-16-2-13210 (duadm1) Agent is calling clean for resource(cluster_maint) because 4 successive invocations of the monitor procedure did not complete within the expected time. 2013/09/03 00:10:13 VCS ERROR V-16-2-13077 (duadm2) Agent is unable to offline resource(ddc_mount). Administrative intervention may be required. 2013/09/03 00:10:13 VCS INFO V-16-10001-5530 (duadm2) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:10:13 VCS INFO V-16-1-10305 Resource ext_notif (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:10:13 VCS INFO V-16-1-10305 Resource log_service (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:10:16 VCS INFO V-16-1-10305 Resource opendj (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:11:03 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:11:13 VCS ERROR V-16-2-13006 (duadm1) Resource(cluster_maint): clean procedure did not complete within the expected time. 2013/09/03 00:11:37 VCS INFO V-16-10001-5530 (duadm1) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:12:04 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:12:14 VCS INFO V-16-10001-5530 (duadm2) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:12:16 VCS INFO V-16-2-13026 (duadm1) Resource(cluster_maint) - monitor procedure finished successfully after failing to complete within the expected time for (4) consecutive times. 2013/09/03 00:12:16 VCS INFO V-16-2-13075 (duadm1) Resource(cluster_maint) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2). 2013/09/03 00:12:24 VCS WARNING V-16-2-13011 (duadm2) Resource(sybmaster_mount): offline procedure did not complete within the expected time. 2013/09/03 00:12:24 VCS ERROR V-16-2-13063 (duadm2) Agent is calling clean for resource(sybmaster_mount) because offline did not complete within the expected time. 2013/09/03 00:12:24 VCS INFO V-16-10001-5530 (duadm2) Mount:sybmaster_mount:clean:Checking for loopback mount. 2013/09/03 00:13:04 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:13:13 VCS WARNING V-16-2-13011 (duadm1) Resource(tomcat): offline procedure did not complete within the expected time. 2013/09/03 00:13:13 VCS WARNING V-16-2-13011 (duadm1) Resource(time_service): offline procedure did not complete within the expected time. 2013/09/03 00:13:13 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(tomcat) because offline did not complete within the expected time. 2013/09/03 00:13:13 VCS WARNING V-16-2-13011 (duadm1) Resource(rmi_reg): offline procedure did not complete within the expected time. 2013/09/03 00:13:13 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(time_service) because offline did not complete within the expected time. 2013/09/03 00:13:13 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(rmi_reg) because offline did not complete within the expected time. 2013/09/03 00:13:13 VCS WARNING V-16-2-13011 (duadm1) Resource(sb_nsa): offline procedure did not complete within the expected time. 2013/09/03 00:13:13 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(sb_nsa) because offline did not complete within the expected time. 2013/09/03 00:13:13 VCS INFO V-16-2-13068 (duadm1) Resource(tomcat) - clean completed successfully. 2013/09/03 00:13:13 VCS INFO V-16-2-13068 (duadm1) Resource(rmi_reg) - clean completed successfully. 2013/09/03 00:13:14 VCS WARNING V-16-2-13011 (duadm1) Resource(sentinel): offline procedure did not complete within the expected time. 2013/09/03 00:13:14 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(sentinel) because offline did not complete within the expected time. 2013/09/03 00:13:14 VCS INFO V-16-2-13068 (duadm1) Resource(sentinel) - clean completed successfully. 2013/09/03 00:13:15 VCS INFO V-16-1-10305 Resource rmi_reg (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:13:15 VCS INFO V-16-1-10305 Resource tomcat (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:13:16 VCS INFO V-16-1-10305 Resource sentinel (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:13:16 VCS INFO V-16-2-13075 (duadm1) Resource(cluster_maint) has reported unexpected OFFLINE 2 times, which is still within the ToleranceLimit(2). 2013/09/03 00:13:25 VCS ERROR V-16-2-13006 (duadm2) Resource(sybmaster_mount): clean procedure did not complete within the expected time. 2013/09/03 00:13:38 VCS INFO V-16-10001-5530 (duadm1) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:14:04 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:14:14 VCS ERROR V-16-2-13006 (duadm1) Resource(sb_nsa): clean procedure did not complete within the expected time. 2013/09/03 00:14:14 VCS ERROR V-16-2-13006 (duadm1) Resource(time_service): clean procedure did not complete within the expected time. 2013/09/03 00:14:15 VCS ERROR V-16-2-13067 (duadm1) Agent is calling clean for resource(cluster_maint) because the resource became OFFLINE unexpectedly, on its own. 2013/09/03 00:14:15 VCS INFO V-16-10001-5530 (duadm2) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:14:16 VCS INFO V-16-1-10305 Resource time_service (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:14:16 VCS INFO V-16-1-10305 Resource sb_nsa (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:14:25 VCS ERROR V-16-2-13077 (duadm2) Agent is unable to offline resource(sybmaster_mount). Administrative intervention may be required. 2013/09/03 00:14:26 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2013/09/03 00:14:26 VCS INFO V-16-10001-5530 (duadm2) Mount:sybmaster_mount:clean:Checking for loopback mount. 2013/09/03 00:14:39 VCS WARNING V-16-2-13011 (duadm1) Resource(gui_nsa): offline procedure did not complete within the expected time. 2013/09/03 00:14:39 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(gui_nsa) because offline did not complete within the expected time. 2013/09/03 00:14:39 VCS INFO V-16-2-13068 (duadm1) Resource(gui_nsa) - clean completed successfully. 2013/09/03 00:14:41 VCS INFO V-16-1-10305 Resource gui_nsa (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:14:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource change_versant (Owner: Unspecified, Group: Oss) on System duadm1 2013/09/03 00:15:04 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:15:16 VCS ERROR V-16-2-13006 (duadm1) Resource(cluster_maint): clean procedure did not complete within the expected time. 2013/09/03 00:15:40 VCS INFO V-16-10001-5530 (duadm1) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:16:04 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:16:16 VCS INFO V-16-10001-5530 (duadm2) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:16:27 VCS INFO V-16-10001-5530 (duadm2) Mount:sybmaster_mount:clean:Checking for loopback mount. 2013/09/03 00:17:04 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:17:41 VCS INFO V-16-10001-5530 (duadm1) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:17:58 VCS WARNING V-16-2-13011 (duadm1) Resource(stop_oss): offline procedure did not complete within the expected time. 2013/09/03 00:17:58 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(stop_oss) because offline did not complete within the expected time. 2013/09/03 00:18:04 VCS INFO V-16-1-50135 User root fired command: hagrp -switch Oss duadm2 from localhost 2013/09/03 00:18:04 VCS INFO V-16-10001-1 (duadm1) Application:stop_oss:stop_oss.sh:Waiting for Oss to go Offline 2013/09/03 00:18:18 VCS INFO V-16-10001-5530 (duadm2) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:18:28 VCS INFO V-16-10001-5530 (duadm2) Mount:sybmaster_mount:clean:Checking for loopback mount. 2013/09/03 00:18:59 VCS ERROR V-16-2-13006 (duadm1) Resource(stop_oss): clean procedure did not complete within the expected time. 2013/09/03 00:19:42 VCS INFO V-16-10001-5530 (duadm1) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:19:42 VCS WARNING V-16-2-13011 (duadm1) Resource(change_versant): offline procedure did not complete within the expected time. 2013/09/03 00:19:42 VCS ERROR V-16-2-13063 (duadm1) Agent is calling clean for resource(change_versant) because offline did not complete within the expected time. 2013/09/03 00:19:42 VCS INFO V-16-2-13068 (duadm1) Resource(change_versant) - clean completed successfully. 2013/09/03 00:19:44 VCS INFO V-16-1-10305 Resource change_versant (Owner: Unspecified, Group: Oss) is offline on duadm1 (VCS initiated) 2013/09/03 00:19:44 VCS NOTICE V-16-1-10446 Group Oss is offline on system duadm1 2013/09/03 00:19:44 VCS INFO V-16-6-15025 (duadm2) hatrigger:invoking nfs_preonline 2013/09/03 00:19:44 VCS INFO V-16-6-15076 (duadm2) hatrigger:invoking regular preonline trigger if it exists 2013/09/03 00:19:44 VCS INFO V-16-6-0 (duadm1) postoffline:(postoffline) Invoked with arg0=duadm1, arg1=Oss 2013/09/03 00:19:44 VCS INFO V-16-6-0 (duadm1) postoffline:Executing /ericsson/core/cluster/scripts/postoffline.sh with arg0=duadm1, arg1=Oss 2013/09/03 00:19:44 VCS INFO V-16-6-0 (duadm2) preonline:(preonline) Invoked with arg0=duadm2, arg1=Oss 2013/09/03 00:19:44 VCS INFO V-16-6-0 (duadm2) preonline:Executing /ericsson/core/cluster/scripts/preonline.sh with arg0=duadm2, arg1=Oss 2013/09/03 00:19:44 VCS INFO V-16-6-0 (duadm1) postoffline.sh:Oss:Stop old FM processes 2013/09/03 00:19:44 VCS INFO V-16-6-0 (duadm1) postoffline.sh:Oss:Stopping dummy SMF resources for Sybase 2013/09/03 00:19:52 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:00 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:01 VCS INFO V-16-1-10305 Resource stop_oss (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cluster_maint (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource smrs_nfs (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ossbak_ip (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource eba_ebsg_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource eba_ebss_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource eba_ebsw_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource eba_rede_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource eba_rtt_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource home_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource nms_cosm_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource pmstor_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource segment1_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource sgwcg_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource versant_mount (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource a3pp_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ericsson_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource etc_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mail_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource opt_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource upgrade_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource usr_local_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource var_share (Owner: Unspecified, Group: Ossfs) on System duadm1 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource a3pp_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource ericsson_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource etc_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource opt_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource usr_local_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource mail_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource var_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:02 VCS INFO V-16-1-10305 Resource upgrade_share (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:03 VCS INFO V-16-1-10305 Resource ossbak_ip (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:20:08 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:17 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:19 VCS INFO V-16-10001-5530 (duadm2) Mount:ddc_mount:clean:Checking for loopback mount. 2013/09/03 00:20:25 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:29 VCS INFO V-16-10001-5530 (duadm2) Mount:sybmaster_mount:clean:Checking for loopback mount. 2013/09/03 00:20:33 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:41 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:49 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:20:57 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource eba_ebsw_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource eba_rtt_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource eba_rede_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource pmstor_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource eba_ebss_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource nms_cosm_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource eba_ebsg_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:03 VCS INFO V-16-1-10305 Resource segment1_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:04 VCS INFO V-16-1-10305 Resource versant_mount (Owner: Unspecified, Group: Ossfs) is offline on duadm1 (VCS initiated) 2013/09/03 00:21:05 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:21:13 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:21:21 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globally 2013/09/03 00:21:29 VCS INFO V-16-6-0 (duadm2) preonline.sh:Oss:Waiting for Sybase1 to come online globallySolved3.3KViews1like13Commentsswitching of service group didnot work vcs 5.0
we tried to do failover , however it didnot worked please find below logs and please help in finding the cause 2013/07/22 20:11:17 VCS INFO V-16-1-50859 Attempting to switch group Oss from system dukosgbs to system dukosgas 2013/07/22 20:11:17 VCS INFO V-16-1-50135 User root fired command: hagrp -switch Oss dukosgas from localhost 2013/07/22 20:11:17 VCS NOTICE V-16-1-50929 Initial tests indicate group Oss is able to switch to system dukosgas. Initiating offline of group on system dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10167 Initiating manual offline of group Oss on system dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource activemq (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource alex (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource apache (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cron (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ddc (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource glassfish (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource imgr_httpd (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource imgr_tomcat (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ldap_mon (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource log_service (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource netmgt_nettl (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource netmgt_ov (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ovtrc (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource restart_mc (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource syb_log_mon (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource syb_proc_mon (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource time_service (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource trapdist (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:11:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource vrsnt_log_mon (Owner: unknown, Group: Oss) on System dukosgbs 2013/07/22 20:45:26 VCS INFO V-16-1-50135 User root fired command: hagrp -switch Oss dukosgas from localhost 2013/07/22 22:19:35 VCS INFO V-16-2-13075 (dukosgbs) Resource(activemq) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2). 2013/07/22 22:20:35 VCS INFO V-16-2-13075 (dukosgbs) Resource(activemq) has reported unexpected OFFLINE 2 times, which is still within the ToleranceLimit(2). 2013/07/22 22:21:35 VCS ERROR V-16-2-13067 (dukosgbs) Agent is calling clean for resource(activemq) because the resource became OFFLINE unexpectedly, on its own. 2013/07/22 22:21:36 VCS INFO V-16-2-13068 (dukosgbs) Resource(activemq) - clean completed successfully. 2013/07/22 22:21:36 VCS ERROR V-16-2-13073 (dukosgbs) Resource(activemq) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 3) the resource. 2013/07/22 22:21:36 VCS INFO V-16-10001-3 (dukosgbs) Application:activemq:online:Executed /ericsson/hacs/scripts/svc.sh 2013/07/22 22:21:37 VCS INFO V-16-2-13001 (dukosgbs) Resource(activemq): Output of the completed operation (online) svcadm: Instance "svc:/ericsson/eric_3pp/activemq:default" is not in a maintenance or degraded state. 2013/07/22 22:21:38 VCS NOTICE V-16-2-13076 (dukosgbs) Agent has successfully restarted resource(activemq). 2013/07/22 22:23:05 VCS INFO V-16-1-50135 User root fired command: hagrp -clear Oss dukosgbs from localhost 2013/07/22 22:27:08 VCS INFO V-16-1-50135 User root fired command: hagrp -flush Oss dukosgbs from localhost 2013/07/22 22:29:40 VCS INFO V-16-1-50135 User root fired command: hagrp -clearadminwait Oss dukosgbs from localhost 2013/07/22 22:37:21 VCS INFO V-16-1-50135 User root fired command: hagrp -flush ClusterService dukosgbs from localhost 2013/07/22 22:37:21 VCS INFO V-16-1-50135 User root fired command: hagrp -flush Oss dukosgbs from localhost 2013/07/22 22:37:21 VCS INFO V-16-1-50135 User root fired command: hagrp -flush Ossfs dukosgbs from localhost 2013/07/22 22:37:21 VCS INFO V-16-1-50135 User root fired command: hagrp -flush Sybase1 dukosgbs from localhost 2013/07/22 22:38:14 VCS INFO V-16-1-50135 User root fired command: hagrp -flush ClusterService dukosgbs from localhost 2013/07/22 22:38:14 VCS INFO V-16-1-50135 User root fired command: hagrp -flush Oss dukosgbs from localhost 2013/07/22 22:38:14 VCS INFO V-16-1-50135 User root fired command: hagrp -flush Ossfs dukosgbs from localhost 2013/07/22 22:38:14 VCS INFO V-16-1-50135 User root fired command: hagrp -flush Sybase1 dukosgbs from localhost 2013/07/22 22:39:08 VCS INFO V-16-1-50135 User root fired command: hares -refreshinfo activemq from localhost 2013/07/22 22:40:06 VCS INFO V-16-1-50135 User root fired command: hares -refreshinfo activemq localclus from localhost 2013/07/22 22:42:09 VCS INFO V-16-1-50135 User root fired command: hares -flushinfo activemq localclus from localhost 2013/07/22 22:47:41 VCS INFO V-16-1-50135 User root fired command: hagrp -switch Oss dukosgbs from localhost 2013/07/22 22:50:52 VCS INFO V-16-1-50859 Attempting to switch group Oss from system dukosgbs to system dukosgas 2013/07/22 22:50:52 VCS INFO V-16-1-50135 User root fired command: hagrp -switch Oss dukosgas from localhost 2013/07/22 22:50:52 VCS NOTICE V-16-1-50929 Initial tests indicate group Oss is able to switch to system dukosgas. Initiating offline of group on system dukosgbs ===================================================================== the resources ware initiating to offline. But they did not come down. root@dukosgbs> hastatus -sum -- SYSTEM STATE -- System State Frozen A dukosgbs RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService dukosgbs Y N ONLINE B Oss dukosgbs Y N ONLINE|STOPPING B Ossfs dukosgbs Y N ONLINE B Sybase1 dukosgbs Y N ONLINE -- RESOURCES OFFLINING -- Group Type Resource System IState F Oss Application activemq dukosgbs W_OFFLINE_PROPAGATE F Oss Application alex dukosgbs W_OFFLINE_PROPAGATE F Oss Application apache dukosgbs W_OFFLINE_PROPAGATE F Oss Application cron dukosgbs W_OFFLINE_PROPAGATE F Oss Application ddc dukosgbs W_OFFLINE_PROPAGATE F Oss Application glassfish dukosgbs W_OFFLINE_PROPAGATE F Oss Application imgr_httpd dukosgbs W_OFFLINE_PROPAGATE F Oss Application imgr_tomcat dukosgbs W_OFFLINE_PROPAGATE F Oss Application ldap_mon dukosgbs W_OFFLINE_PROPAGATE F Oss Application log_service dukosgbs W_OFFLINE_PROPAGATE F Oss Application netmgt_nettl dukosgbs W_OFFLINE_PROPAGATE F Oss Application netmgt_ov dukosgbs W_OFFLINE_PROPAGATE F Oss Application ovtrc dukosgbs W_OFFLINE_PROPAGATE F Oss Application restart_mc dukosgbs W_OFFLINE_PROPAGATE F Oss Application syb_log_mon dukosgbs W_OFFLINE_PROPAGATE F Oss Application syb_proc_mon dukosgbs W_OFFLINE_PROPAGATE F Oss Application time_service dukosgbs W_OFFLINE_PROPAGATE F Oss Application trapdist dukosgbs W_OFFLINE_PROPAGATE F Oss Application vrsnt_log_mon dukosgbs W_OFFLINE_PROPAGATE -- WAN HEARTBEAT STATE -- Heartbeat To State L Icmp gran_cluster1 ALIVE -- REMOTE CLUSTER STATE -- Cluster State M gran_cluster1 RUNNING -- REMOTE SYSTEM STATE -- cluster:system State Frozen N gran_cluster1:dukosgas RUNNING 0Solved764Views0likes1CommentVERITAS STORAGE EXEC 5.5 REPLICATIONS PROBLEMS
Hi, I have two nodes in cluster and quotas defined in Node A are not replicated to a Node B, when I switch the cluster resources. Where policies and quotas are defined? Local: C: \ ProgramFiles \ Veritas \ StorageExec \ DB SCAudit.mdb or SCTrend.mdb Should not be in a cluster resource? Thank you all for your answers629Views0likes2Commentsneed a solution
we have 2 node cluster and with version 5.1 we experienced outage and I think it was due to below error messages can someone shed some light on these messages qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop OFFLINE qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop ONLINE fctl: [ID 999315 kern.warning] WARNING: fctl(4): AL_PA=0xe8 doesn't exist in LILP map scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0/ssd@w203400a0b875f9d9,0 (ssd3): Command failed to complete...Device is gone scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0/ssd@w203400a0b875f9d9,0 (ssd3): Command failed to complete...Device is gone scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0/ssd@w203400a0b875f9d9,0 (ssd3): Command failed to complete...Device is gone scsi: [ID 243001 kern.info] /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0 (fcp4): offlining lun=0 (trace=0), target=e8 (trace=2800004) vxdmp: [ID 631182 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 removed disk array 600A0B800075F9D9000000004D2334F5, datype = ST2540- vxdmp: [ID 443116 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 334/0x2c last message repeated 59 times vxdmp: [ID 480808 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x18 belonging to the dmpnode 334/0x28 due to open failure vxdmp: [ID 824220 kern.notice] NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 334/0x28 what is this dmpnode 334/0x28 signify, I forget how to map this to device as i only remember is tht its in hexadecimal. Also, what could be the cause of it ... is it due to HBA as issue starts with the message like below qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop OFFLINE qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop ONLINE fctl: [ID 999315 kern.warning] WARNING: fctl(4): AL_PA=0xe8 doesn't exist in LILP map1.8KViews2likes4Comments
Backup up VCS Cluster Config
Can anyone advise on what command or process is involved to backup the VCS configuration of SFW HADR 5.1 onwards. The old command was example : hasnap -backup -f clusterbackup.zip -n -m “Backup from March 25th 2007″ Apparently this is no longer supported on later versions of the software.Solved800Views0likes1Comment
VERITAS STORAGE EXEC 5.5 REPLICATIONS PROBLEMS
Hi, I have two nodes in cluster and quotas defined in Node A are not replicated to a Node B, when I switch the cluster resources. Where policies and quotas are defined? Local: C: \ ProgramFiles \ Veritas \ StorageExec \ DB SCAudit.mdb or SCTrend.mdb Should not be in a cluster resource? Thank you all for your answersneed a solution
we have 2 node cluster and with version 5.1 we experienced outage and I think it was due to below error messages can someone shed some light on these messages qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop OFFLINE qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop ONLINE fctl: [ID 999315 kern.warning] WARNING: fctl(4): AL_PA=0xe8 doesn't exist in LILP map scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0/ssd@w203400a0b875f9d9,0 (ssd3): Command failed to complete...Device is gone scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0/ssd@w203400a0b875f9d9,0 (ssd3): Command failed to complete...Device is gone scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0/ssd@w203400a0b875f9d9,0 (ssd3): Command failed to complete...Device is gone scsi: [ID 243001 kern.info] /pci@0,600000/pci@0/pci@9/SUNW,qlc@0/fp@0,0 (fcp4): offlining lun=0 (trace=0), target=e8 (trace=2800004) vxdmp: [ID 631182 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 removed disk array 600A0B800075F9D9000000004D2334F5, datype = ST2540- vxdmp: [ID 443116 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 i/o error occured (errno=0x6) on dmpnode 334/0x2c last message repeated 59 times vxdmp: [ID 480808 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x18 belonging to the dmpnode 334/0x28 due to open failure vxdmp: [ID 824220 kern.notice] NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 334/0x28 what is this dmpnode 334/0x28 signify, I forget how to map this to device as i only remember is tht its in hexadecimal. Also, what could be the cause of it ... is it due to HBA as issue starts with the message like below qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop OFFLINE qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop ONLINE fctl: [ID 999315 kern.warning] WARNING: fctl(4): AL_PA=0xe8 doesn't exist in LILP map