Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- VOX
- Availability
- Cluster Server
- VCS Log
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
VCS Log
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-09-2009 02:09 AM
Hi All,
Can anyone help me to analyse this VCS log? One of my applications failover suddenly and I need to find the reason behind this.
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10430 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:entryPointExit:Generating FFDC messages for resource [Web-ebill2app]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10443 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[ffdc:10027:Arch:GetOSType:Subroutine Arch::GetOSType is called
ffdc:10420:VCSagentFW:SetupEntry:Subroutine VCSagentFW::SetupEntry called with: Entry Point Name [monitor], Category ID [55008]
ffdc:10421:VCSagentFW:SetupLogging:Subroutine VCSagentFW::SetupLogging called with: Clean Reason [<undef>], Log Level [INFO]
ffdc:10425:VCSagentFW:SetupLogging:[monitor] Entered by resource instance [Web-ebill2app]
ffdc:20242:Resource found with State [2] and IState [0]
ffdc:20001:Subroutine <WebLogic9::ArgsValid> called with:
EntryPointName [monitor]
AdminURL []
BEA_HOME [/appl/bea]
WL_HOME [/appl/bea/weblogic92]
DomainName [ebilling2]
DomainDir [/appl/bea/user_projects/ebilling2]
ListenAddressPort [192.168.100.41:7002]
MonitorProgram []
nmListenAddressPort []
nmType [ssl]
ServerName [ebill2svr]
ServerRole [Administrative]
User [ebil2adm]
WLSUser [ebil2adm]
WLSPassword [drkPprGjkJhjEpiPfpMppJf]
ServerStartProgram [/appl/bea/user_projects/ebilling2/bin/startEBILLING2.sh]
ServerStopProgram [/appl/bea/user_projects/ebilling2/bin/stopEBILLING2.sh]
RequireAdminServer [0]
AdminServerMaxWait [60]
SecondLevelMonitor [0]
ffdc:20002:<WebLogic9::ArgsValid>:time is [1239246114]
ffdc:20003:<WebLogic9::ArgsValid>:This is non-NodeManaged based configuration.
ffdc:20243:All arguments validated successfully!
ffdc:20246:Subroutine <main::MonitorEntryPoint> called
ffdc:10439:VCSagentFW:GetVCSObjectHash:Subroutine [VCSagentFW::GetVCSObjectHash] called with ObjectName=[hatype], ObjectInstanceName=[WebLogic9]
ffdc:10209:Sys:RunWithEnvCmdWithOutputWithTimeOut:Subroutine Sys::RunWithEnvCmdWithOutputWithTimeOut called with: EnvFile [], Command [/opt/VRTSvcs/bin/hatype], Arguments [-display WebLogic9], User [root], Timeout [10], OutFile [<undef>], FromDir [<undef>]
ffdc:10211:Sys:RunWithEnvCmdWithOutputWithTimeOut:Environment file not set
ffdc:10218:Sys:RunWithEnvCmdWithOutputWithTimeOut:Going to run command line [/opt/VRTSvcs/bin/hatype -display WebLogic9], as User [root]
ffdc:10221:Sys:RunWithEnvCmdWithOutputWithTimeOut:Now going to run complete command line [/opt/VRTSvcs/bin/hatype -display WebLogic9]
ffdc:10445:VCSagentFW:messageEngineLog:[#Type Attribute Value
WebLogic9 AMF Mode 3 MonitorInterval 60 OfflineMonitorInterval 300 RegisterRetryLimit 3
WebLogic9 ActionTimeout 30
WebLogic9 AgentClass TS
WebLogic9 AgentDirectory /opt/VRTSagents/ha/bin/WebLogic9
WebLogic9 AgentFailedOn
WebLogic9 AgentFile /opt/VRTSvcs/bin/ScriptAgent
WebLogic9 AgentPriority 0
WebLogic9 AgentReplyTimeout 130
WebLogic9 AgentStartTimeout 60
WebLogic9 ArgList ResLogLevel State IState AdminURL BEA_HOME WL_HOME DomainName DomainDir ListenAddressPort MonitorProgram nmListenAddressPort nmType ServerName ServerRole User WLSUser WLSPassword ServerStartProgram ServerStopProgram RequireAdminServer AdminServerMaxWait SecondLevelMonitor]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10443 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[WebLogic9 AttrChangedTimeout 60
WebLogic9 CleanTimeout 60
WebLogic9 CloseTimeout 60
WebLogic9 ConfInterval 600
WebLogic9 FaultOnMonitorTimeouts 4
WebLogic9 FireDrill 0
WebLogic9 InfoInterval 0
WebLogic9 InfoTimeout 30
WebLogic9 LogDbg
WebLogic9 LogFileSize 33554432
WebLogic9 MonitorInterval 60
WebLogic9 MonitorStatsParam Frequency 0 ExpectedValue 100 ValueThreshold 100 AvgThreshold 40
WebLogic9 MonitorTimeout 60
WebLogic9 NumThreads 10
WebLogic9 OfflineMonitorInterval 300
WebLogic9 OfflineProcScanInterval 60
WebLogic9 OfflineTimeout 300
WebLogic9 OnlineRetryLimit 0
WebLogic9 OnlineTimeout 300
WebLogic9 OnlineWaitLimit 2
WebLogic9 OpenTimeout 60
WebLogic9 Operations OnOff
WebLogic9 ProcScanInterval 60
WebLogic9 ProcScanStatus 0
WebLogic9 RestartLimit 0
WebLogic9 ScriptClass TS
WebLogic9 ScriptPriority 0
WebLogic9 SourceFile ./WebLogic9Types.cf
WebLogic9 SupportedActions
WebLogic9 ToleranceLimit 0
]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AMF] [Mode 3 MonitorInterval 60 OfflineMonitorInterval 300 RegisterRetryLimit 3]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ActionTimeout] [30]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentClass] [TS]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentDirectory] [/opt/VRTSagents/ha/bin/WebLogic9]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentFailedOn] []
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentFile] [/opt/VRTSvcs/bin/ScriptAgent]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentPriority] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentReplyTimeout] [130]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentStartTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ArgList] [ResLogLevel State IState AdminURL BEA_HOME WL_HOME DomainName DomainDir ListenAddressPort MonitorProgram nmListenAddressPort nmType ServerName ServerRole User WLSUser WLSPassword ServerStartProgram ServerStopProgram RequireAdminServer AdminServerMaxWait SecondLevelMonitor]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AttrChangedTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [CleanTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [CloseTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ConfInterval] [600]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [FaultOnMonitorTimeouts] [4]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [FireDrill] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [InfoInterval] [0]]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10443 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [InfoTimeout] [30]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [LogDbg] []
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [LogFileSize] [33554432]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [MonitorInterval] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [MonitorStatsParam] [Frequency 0 ExpectedValue 100 ValueThreshold 100 AvgThreshold 40]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [MonitorTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [NumThreads] [10]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OfflineMonitorInterval] [300]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OfflineProcScanInterval] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OfflineTimeout] [300]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OnlineRetryLimit] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OnlineTimeout] [300]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OnlineWaitLimit] [2]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OpenTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [Operations] [OnOff]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ProcScanInterval] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ProcScanStatus] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [RestartLimit] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ScriptClass] [TS]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ScriptPriority] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [SourceFile] [./WebLogic9Types.cf]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [SupportedActions] []
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ToleranceLimit] [0]
ffdc:20082:Subroutine <WebLogic9::GetResourceState> called with arguments:
State [2]
iState [0]
EntryPointName [monitor]
BEA_HOME [/appl/bea]
WL_HOME [/appl/bea/weblogic92]
ListenAddressPort [192.168.100.41:7002]
nmListenAddressPort []
ServerName [ebill2svr]
ServerRole [Administrative]
DomainName [ebilling2]
DomainDir [/appl/bea/user_projects/ebilling2]
nmType [ssl]
WLSUser [ebil2adm]
WLSPassword [drkPprGjkJhjEpiPfpMppJf]
User [ebil2adm]
SecondLevelMonitor [0]
MonitorProgramAndArgs []
ExpirationTime [1239246171]
ffdc:20100:Subroutine <WebLogic9::GetWLSState> called with:
BEA_HOME [/appl/bea]
WL_HOME [/appl/bea/weblogic92]
EntryPointName [monitor]
AdminURL []]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10444 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[
User [ebil2adm]
WLSUser [ebil2adm]
WLSPassword [drkPprGjkJhjEpiPfpMppJf]
ServerName [ebill2svr]
ServerRole [Administrative]
DomainName [ebilling2]
DomainDir [/appl/bea/user_projects/ebilling2]
ListenAddressPort [192.168.100.41:7002]
nmListenAddressPort []
ExpirationTime [1239246171]
MonitorProgram []
State [2]
IState [0]
ffdc:20143:Subroutine <WebLogic9::ExtractIP> called with: HostPort [192.168.100.41:7002]
ffdc:20146:Subroutine <WebLogic9::CheckIfIPExistsLocally> called with: IP [192.168.100.41]
ffdc:10027:Arch:GetOSType:Subroutine Arch::GetOSType is called
ffdc:20147:<WebLogic9::CheckIfIPExistsLocally>:IP [192.168.100.41] is plumbed.
ffdc:10001:Comms:CheckPort:Subroutine Comms::CheckPort called with: Host [192.168.100.41], Port [7002]
ffdc:10002:Comms:CheckPort:Checking Host:[192.168.100.41] on Port:[7002], TimeoutSeconds:[<undef>]
ffdc:10004:Comms:CheckPort:Checking Host:[192.168.100.41] on Port:[7002], TimeoutSeconds:[<undef>] (Failed to Connect)
ffdc:20103:<WebLogic9::GetWLSState>:IP [192.168.100.41] not listening on [7002].
ffdc:20085:<WebLogic9::GetResourceState>:Found no WLS processes for Server [ebill2svr]
ffdc:10427:VCSagentFW:entryPointExit:Subroutine VCSagentFW::entryPointExit called with: Exit code [100]
ffdc:10428:VCSagentFW:entryPointExit:Exiting entry point [monitor] with exit code [100]
ffdc:10430:VCSagentFW:entryPointExit:Generating FFDC messages for resource [Web-ebill2app]
]
2009/04/09 10:58:57 VCS NOTICE V-16-55008-10438 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:logOutputFromCommand:Command line [/bin/env] produced exit code [0] and generated output [CLUSTER_HOME=/opt/VRTSvcs
CLUSTER_LOGDBG=
HOME=/root
HZ=
LANG=C
LC_ALL=
LOGNAME=root
MAIL=/var/mail/root
MANPATH=/usr/man:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/usr/sbin:/usr/ccs/bin:/opt/VRTS/bin:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/sbin:/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin
PWD=/var/VRTSvcs/diag/had
SHELL=/sbin/sh
SHLVL=1
TERM=vt100
TZ=Asia/Kuala_Lumpur
VAD_LOG_AGENT_NAME=WebLogic9
VAD_LOG_CATEGORY=55008
VAD_LOG_RESOURCE_NAME=Web-ebill2app
VAD_LOG_SCRIPT_NAME=monitor
VCS_AGFW=1
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG=/var/VRTSvcs
VCS_LOG_AGENT_NAME=WebLogic9
VCS_LOG_CATEGORY=55008
VCS_LOG_RESOURCE_NAME=Web-ebill2app
VCS_LOG_SCRIPT_NAME=monitor
_=/opt/VRTS/bin/hastart]
2009/04/09 10:58:57 VCS ERROR V-16-2-13067 (CS49PAPS2) Agent is calling clean for resource(Web-ebill2app) because the resource became OFFLINE unexpectedly, on its own.
2009/04/09 10:58:58 VCS NOTICE V-16-55008-10424 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:VCSagentFW:SetupLogging:[clean] Entered by resource instance [Web-ebill2app] with clean reason [4][Unexpected Offline]
2009/04/09 10:58:58 VCS NOTICE V-16-55008-20080 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:<WebLogic9::Kill>:WebLogic Server process not found.
2009/04/09 10:58:58 VCS INFO V-16-2-13068 (CS49PAPS2) Resource(Web-ebill2app) - clean completed successfully.
2009/04/09 10:59:00 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for resfault; script doesn't exist
2009/04/09 10:59:00 VCS INFO V-16-1-10305 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS INFO V-16-1-10305 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:02 VCS INFO V-16-1-10305 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10305 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:12 VCS ERROR V-16-1-10205 Group Ebill2ap-app is faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10446 Group Ebill2ap-app is offline on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating CS49PAPS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS INFO V-16-1-50010 Group Ebill2ap-app is online or faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating TS49SDAS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS INFO V-16-1-10298 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:13 VCS INFO V-16-6-15002 (CS49PAPS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline CS49PAPS2 Ebill2ap-app successfully
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for lvmvg_postoffline; script doesn't exist
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2009/04/09 10:59:13 VCS WARNING V-16-10001-1014 (TS49SDAS2) DiskGroup:Dg-ebill2app:online:Diskgroups will be imported without reservations
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1009 (TS49SDAS2) DiskGroup:Dg-ebill2app:online:vxdg import succeeded on Disk Group ebill2apdg
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1010 (TS49SDAS2) DiskGroup:Dg-ebill2app:online:Volumes in Disk Group ebill2apdg are started. Any mirrors are updated in background
2009/04/09 10:59:17 VCS INFO V-16-1-10298 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:17 VCS NOTICE V-16-10001-11501 (TS49SDAS2) Volume:Vol-ebill2app:online:Volume ebill2apvol is started. Any mirrors are updated in background
2009/04/09 10:59:18 VCS INFO V-16-1-10298 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:19 VCS INFO V-16-1-10298 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:20 VCS WARNING V-16-55008-20158 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>:Server Process is not running.
2009/04/09 10:59:26 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:31 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:36 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:41 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:46 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:51 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20162 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: First Level Check , server responding at [192.168.100.41:7002].
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20255 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<main::OnlineEntryPoint>:Resource [Web-ebill2app] started up successfully.
2009/04/09 10:59:58 VCS INFO V-16-1-10298 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:58 VCS NOTICE V-16-1-10447 Group Ebill2ap-app is online on system TS49SDAS2
2009/04/09 10:59:58 VCS NOTICE V-16-1-10448 Group Ebill2ap-app failed over to system TS49SDAS2
2009/04/09 10:59:59 VCS INFO V-16-6-15051 (TS49SDAS2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group
2009/04/09 10:59:59 VCS INFO V-16-6-15002 (TS49SDAS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart Ebill2ap-app successfully
2009/04/09 10:59:59 VCS INFO V-16-6-15004 (TS49SDAS2) hatrigger:Failed to send trigger for postonline; script doesn't exist
2009/04/09 15:46:39 VCS INFO V-16-1-50135 User root fired command: hagrp -clear Ebill2ap-app CS49PAPS2 from localhost
2009/04/09 15:46:39 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)
Appreciate your help.
Thank you.
rgds,
Ronny
Can anyone help me to analyse this VCS log? One of my applications failover suddenly and I need to find the reason behind this.
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10430 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:entryPointExit:Generating FFDC messages for resource [Web-ebill2app]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10443 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[ffdc:10027:Arch:GetOSType:Subroutine Arch::GetOSType is called
ffdc:10420:VCSagentFW:SetupEntry:Subroutine VCSagentFW::SetupEntry called with: Entry Point Name [monitor], Category ID [55008]
ffdc:10421:VCSagentFW:SetupLogging:Subroutine VCSagentFW::SetupLogging called with: Clean Reason [<undef>], Log Level [INFO]
ffdc:10425:VCSagentFW:SetupLogging:[monitor] Entered by resource instance [Web-ebill2app]
ffdc:20242:Resource found with State [2] and IState [0]
ffdc:20001:Subroutine <WebLogic9::ArgsValid> called with:
EntryPointName [monitor]
AdminURL []
BEA_HOME [/appl/bea]
WL_HOME [/appl/bea/weblogic92]
DomainName [ebilling2]
DomainDir [/appl/bea/user_projects/ebilling2]
ListenAddressPort [192.168.100.41:7002]
MonitorProgram []
nmListenAddressPort []
nmType [ssl]
ServerName [ebill2svr]
ServerRole [Administrative]
User [ebil2adm]
WLSUser [ebil2adm]
WLSPassword [drkPprGjkJhjEpiPfpMppJf]
ServerStartProgram [/appl/bea/user_projects/ebilling2/bin/startEBILLING2.sh]
ServerStopProgram [/appl/bea/user_projects/ebilling2/bin/stopEBILLING2.sh]
RequireAdminServer [0]
AdminServerMaxWait [60]
SecondLevelMonitor [0]
ffdc:20002:<WebLogic9::ArgsValid>:time is [1239246114]
ffdc:20003:<WebLogic9::ArgsValid>:This is non-NodeManaged based configuration.
ffdc:20243:All arguments validated successfully!
ffdc:20246:Subroutine <main::MonitorEntryPoint> called
ffdc:10439:VCSagentFW:GetVCSObjectHash:Subroutine [VCSagentFW::GetVCSObjectHash] called with ObjectName=[hatype], ObjectInstanceName=[WebLogic9]
ffdc:10209:Sys:RunWithEnvCmdWithOutputWithTimeOut:Subroutine Sys::RunWithEnvCmdWithOutputWithTimeOut called with: EnvFile [], Command [/opt/VRTSvcs/bin/hatype], Arguments [-display WebLogic9], User [root], Timeout [10], OutFile [<undef>], FromDir [<undef>]
ffdc:10211:Sys:RunWithEnvCmdWithOutputWithTimeOut:Environment file not set
ffdc:10218:Sys:RunWithEnvCmdWithOutputWithTimeOut:Going to run command line [/opt/VRTSvcs/bin/hatype -display WebLogic9], as User [root]
ffdc:10221:Sys:RunWithEnvCmdWithOutputWithTimeOut:Now going to run complete command line [/opt/VRTSvcs/bin/hatype -display WebLogic9]
ffdc:10445:VCSagentFW:messageEngineLog:[#Type Attribute Value
WebLogic9 AMF Mode 3 MonitorInterval 60 OfflineMonitorInterval 300 RegisterRetryLimit 3
WebLogic9 ActionTimeout 30
WebLogic9 AgentClass TS
WebLogic9 AgentDirectory /opt/VRTSagents/ha/bin/WebLogic9
WebLogic9 AgentFailedOn
WebLogic9 AgentFile /opt/VRTSvcs/bin/ScriptAgent
WebLogic9 AgentPriority 0
WebLogic9 AgentReplyTimeout 130
WebLogic9 AgentStartTimeout 60
WebLogic9 ArgList ResLogLevel State IState AdminURL BEA_HOME WL_HOME DomainName DomainDir ListenAddressPort MonitorProgram nmListenAddressPort nmType ServerName ServerRole User WLSUser WLSPassword ServerStartProgram ServerStopProgram RequireAdminServer AdminServerMaxWait SecondLevelMonitor]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10443 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[WebLogic9 AttrChangedTimeout 60
WebLogic9 CleanTimeout 60
WebLogic9 CloseTimeout 60
WebLogic9 ConfInterval 600
WebLogic9 FaultOnMonitorTimeouts 4
WebLogic9 FireDrill 0
WebLogic9 InfoInterval 0
WebLogic9 InfoTimeout 30
WebLogic9 LogDbg
WebLogic9 LogFileSize 33554432
WebLogic9 MonitorInterval 60
WebLogic9 MonitorStatsParam Frequency 0 ExpectedValue 100 ValueThreshold 100 AvgThreshold 40
WebLogic9 MonitorTimeout 60
WebLogic9 NumThreads 10
WebLogic9 OfflineMonitorInterval 300
WebLogic9 OfflineProcScanInterval 60
WebLogic9 OfflineTimeout 300
WebLogic9 OnlineRetryLimit 0
WebLogic9 OnlineTimeout 300
WebLogic9 OnlineWaitLimit 2
WebLogic9 OpenTimeout 60
WebLogic9 Operations OnOff
WebLogic9 ProcScanInterval 60
WebLogic9 ProcScanStatus 0
WebLogic9 RestartLimit 0
WebLogic9 ScriptClass TS
WebLogic9 ScriptPriority 0
WebLogic9 SourceFile ./WebLogic9Types.cf
WebLogic9 SupportedActions
WebLogic9 ToleranceLimit 0
]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AMF] [Mode 3 MonitorInterval 60 OfflineMonitorInterval 300 RegisterRetryLimit 3]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ActionTimeout] [30]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentClass] [TS]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentDirectory] [/opt/VRTSagents/ha/bin/WebLogic9]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentFailedOn] []
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentFile] [/opt/VRTSvcs/bin/ScriptAgent]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentPriority] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentReplyTimeout] [130]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AgentStartTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ArgList] [ResLogLevel State IState AdminURL BEA_HOME WL_HOME DomainName DomainDir ListenAddressPort MonitorProgram nmListenAddressPort nmType ServerName ServerRole User WLSUser WLSPassword ServerStartProgram ServerStopProgram RequireAdminServer AdminServerMaxWait SecondLevelMonitor]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [AttrChangedTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [CleanTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [CloseTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ConfInterval] [600]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [FaultOnMonitorTimeouts] [4]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [FireDrill] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [InfoInterval] [0]]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10443 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [InfoTimeout] [30]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [LogDbg] []
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [LogFileSize] [33554432]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [MonitorInterval] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [MonitorStatsParam] [Frequency 0 ExpectedValue 100 ValueThreshold 100 AvgThreshold 40]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [MonitorTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [NumThreads] [10]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OfflineMonitorInterval] [300]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OfflineProcScanInterval] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OfflineTimeout] [300]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OnlineRetryLimit] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OnlineTimeout] [300]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OnlineWaitLimit] [2]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [OpenTimeout] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [Operations] [OnOff]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ProcScanInterval] [60]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ProcScanStatus] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [RestartLimit] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ScriptClass] [TS]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ScriptPriority] [0]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [SourceFile] [./WebLogic9Types.cf]
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [SupportedActions] []
ffdc:10440:VCSagentFW:GetVCSObjectHash:Attribute/Value pair: [ToleranceLimit] [0]
ffdc:20082:Subroutine <WebLogic9::GetResourceState> called with arguments:
State [2]
iState [0]
EntryPointName [monitor]
BEA_HOME [/appl/bea]
WL_HOME [/appl/bea/weblogic92]
ListenAddressPort [192.168.100.41:7002]
nmListenAddressPort []
ServerName [ebill2svr]
ServerRole [Administrative]
DomainName [ebilling2]
DomainDir [/appl/bea/user_projects/ebilling2]
nmType [ssl]
WLSUser [ebil2adm]
WLSPassword [drkPprGjkJhjEpiPfpMppJf]
User [ebil2adm]
SecondLevelMonitor [0]
MonitorProgramAndArgs []
ExpirationTime [1239246171]
ffdc:20100:Subroutine <WebLogic9::GetWLSState> called with:
BEA_HOME [/appl/bea]
WL_HOME [/appl/bea/weblogic92]
EntryPointName [monitor]
AdminURL []]
2009/04/09 10:58:56 VCS NOTICE V-16-55008-10444 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:messageEngineLog:[
User [ebil2adm]
WLSUser [ebil2adm]
WLSPassword [drkPprGjkJhjEpiPfpMppJf]
ServerName [ebill2svr]
ServerRole [Administrative]
DomainName [ebilling2]
DomainDir [/appl/bea/user_projects/ebilling2]
ListenAddressPort [192.168.100.41:7002]
nmListenAddressPort []
ExpirationTime [1239246171]
MonitorProgram []
State [2]
IState [0]
ffdc:20143:Subroutine <WebLogic9::ExtractIP> called with: HostPort [192.168.100.41:7002]
ffdc:20146:Subroutine <WebLogic9::CheckIfIPExistsLocally> called with: IP [192.168.100.41]
ffdc:10027:Arch:GetOSType:Subroutine Arch::GetOSType is called
ffdc:20147:<WebLogic9::CheckIfIPExistsLocally>:IP [192.168.100.41] is plumbed.
ffdc:10001:Comms:CheckPort:Subroutine Comms::CheckPort called with: Host [192.168.100.41], Port [7002]
ffdc:10002:Comms:CheckPort:Checking Host:[192.168.100.41] on Port:[7002], TimeoutSeconds:[<undef>]
ffdc:10004:Comms:CheckPort:Checking Host:[192.168.100.41] on Port:[7002], TimeoutSeconds:[<undef>] (Failed to Connect)
ffdc:20103:<WebLogic9::GetWLSState>:IP [192.168.100.41] not listening on [7002].
ffdc:20085:<WebLogic9::GetResourceState>:Found no WLS processes for Server [ebill2svr]
ffdc:10427:VCSagentFW:entryPointExit:Subroutine VCSagentFW::entryPointExit called with: Exit code [100]
ffdc:10428:VCSagentFW:entryPointExit:Exiting entry point [monitor] with exit code [100]
ffdc:10430:VCSagentFW:entryPointExit:Generating FFDC messages for resource [Web-ebill2app]
]
2009/04/09 10:58:57 VCS NOTICE V-16-55008-10438 (CS49PAPS2) WebLogic9:Web-ebill2app:monitor:VCSagentFW:logOutputFromCommand:Command line [/bin/env] produced exit code [0] and generated output [CLUSTER_HOME=/opt/VRTSvcs
CLUSTER_LOGDBG=
HOME=/root
HZ=
LANG=C
LC_ALL=
LOGNAME=root
MAIL=/var/mail/root
MANPATH=/usr/man:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/usr/sbin:/usr/ccs/bin:/opt/VRTS/bin:/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/sbin:/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin
PWD=/var/VRTSvcs/diag/had
SHELL=/sbin/sh
SHLVL=1
TERM=vt100
TZ=Asia/Kuala_Lumpur
VAD_LOG_AGENT_NAME=WebLogic9
VAD_LOG_CATEGORY=55008
VAD_LOG_RESOURCE_NAME=Web-ebill2app
VAD_LOG_SCRIPT_NAME=monitor
VCS_AGFW=1
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG=/var/VRTSvcs
VCS_LOG_AGENT_NAME=WebLogic9
VCS_LOG_CATEGORY=55008
VCS_LOG_RESOURCE_NAME=Web-ebill2app
VCS_LOG_SCRIPT_NAME=monitor
_=/opt/VRTS/bin/hastart]
2009/04/09 10:58:57 VCS ERROR V-16-2-13067 (CS49PAPS2) Agent is calling clean for resource(Web-ebill2app) because the resource became OFFLINE unexpectedly, on its own.
2009/04/09 10:58:58 VCS NOTICE V-16-55008-10424 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:VCSagentFW:SetupLogging:[clean] Entered by resource instance [Web-ebill2app] with clean reason [4][Unexpected Offline]
2009/04/09 10:58:58 VCS NOTICE V-16-55008-20080 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:<WebLogic9::Kill>:WebLogic Server process not found.
2009/04/09 10:58:58 VCS INFO V-16-2-13068 (CS49PAPS2) Resource(Web-ebill2app) - clean completed successfully.
2009/04/09 10:59:00 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for resfault; script doesn't exist
2009/04/09 10:59:00 VCS INFO V-16-1-10305 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS INFO V-16-1-10305 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:02 VCS INFO V-16-1-10305 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10305 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:12 VCS ERROR V-16-1-10205 Group Ebill2ap-app is faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10446 Group Ebill2ap-app is offline on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating CS49PAPS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS INFO V-16-1-50010 Group Ebill2ap-app is online or faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating TS49SDAS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS INFO V-16-1-10298 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:13 VCS INFO V-16-6-15002 (CS49PAPS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline CS49PAPS2 Ebill2ap-app successfully
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for lvmvg_postoffline; script doesn't exist
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2009/04/09 10:59:13 VCS WARNING V-16-10001-1014 (TS49SDAS2) DiskGroup:Dg-ebill2app:online:Diskgroups will be imported without reservations
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1009 (TS49SDAS2) DiskGroup:Dg-ebill2app:online:vxdg import succeeded on Disk Group ebill2apdg
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1010 (TS49SDAS2) DiskGroup:Dg-ebill2app:online:Volumes in Disk Group ebill2apdg are started. Any mirrors are updated in background
2009/04/09 10:59:17 VCS INFO V-16-1-10298 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:17 VCS NOTICE V-16-10001-11501 (TS49SDAS2) Volume:Vol-ebill2app:online:Volume ebill2apvol is started. Any mirrors are updated in background
2009/04/09 10:59:18 VCS INFO V-16-1-10298 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:19 VCS INFO V-16-1-10298 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:20 VCS WARNING V-16-55008-20158 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>:Server Process is not running.
2009/04/09 10:59:26 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:31 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:36 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:41 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:46 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:51 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20162 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<WebLogic9::DelayEntryPoint>: First Level Check , server responding at [192.168.100.41:7002].
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20255 (TS49SDAS2) WebLogic9:Web-ebill2app:online:<main::OnlineEntryPoint>:Resource [Web-ebill2app] started up successfully.
2009/04/09 10:59:58 VCS INFO V-16-1-10298 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:58 VCS NOTICE V-16-1-10447 Group Ebill2ap-app is online on system TS49SDAS2
2009/04/09 10:59:58 VCS NOTICE V-16-1-10448 Group Ebill2ap-app failed over to system TS49SDAS2
2009/04/09 10:59:59 VCS INFO V-16-6-15051 (TS49SDAS2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group
2009/04/09 10:59:59 VCS INFO V-16-6-15002 (TS49SDAS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart Ebill2ap-app successfully
2009/04/09 10:59:59 VCS INFO V-16-6-15004 (TS49SDAS2) hatrigger:Failed to send trigger for postonline; script doesn't exist
2009/04/09 15:46:39 VCS INFO V-16-1-50135 User root fired command: hagrp -clear Ebill2ap-app CS49PAPS2 from localhost
2009/04/09 15:46:39 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)
Appreciate your help.
Thank you.
rgds,
Ronny
Labels:
- Labels:
-
business continuity
-
Cluster Server
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-09-2009 10:58 PM
Hi Ronny,
I received the following feedback from our Agent Engineering:
"From the log it appears that the WebLogic Server java process stopped running. The monitor entry point detected that the Server is no longer listening on the IP [192.168.100.41] and port [7002].
The susequent clean entry point detected that the process is not running and hence it did not kill any process (no matching process found). The Weblogic server logs may provide some information.
There is a section in WebLogic agent install config guide for trouble shooting where in you could find the WebLogic server log directories and log files location. This may provide some information.
If the WebLogic Server is facing problems, access the log files of the WebLogic Server to further investigate the problem. The log files are located at:
- For Node Managers:
<WL_HOME>/common/nodemanager/nodemanager.log
- For Administrative Servers:
<DomainDir>/servers/<ServerName>/<ServerName>.log
<DomainDir>/servers/<ServerName>/<ServerName>.out
- For Managed Servers:
<DomainDir>/servers/<ServerName>/<ServerName>.log
<DomainDir>/servers/<ServerName>/<ServerName>.out
<DomainDir>/servers/<ServerName>/access.log"
I hope this helps.
Regards
Manuel
