Problem with VRTS . oracle alram

What version?

Reply with output of:

rpm -aqi |grep VRTS

You may have encountered a known issue with an agent, but we would need more data.

 

Regards,

Ted

 

 

MonitorTimeout can be adjusted - the default timeout is 60 seconds.

Please consider this advice in VCS Admin Guide:

You can also adjust how often VCS monitors various functions by modifying their associated attributes. The attributes MonitorTimeout, OnlineTimeOut, and OfflineTimeout indicate the maximum time (in seconds) within which the monitor, online, and offline functions must complete or else be terminated. The default for the MonitorTimeout attribute is 60 seconds. The defaults for the OnlineTimeout and OfflineTimeout attributes is 300 seconds.

For best results, Symantec recommends measuring the time it takes to bring a resource online, take it offline, and monitor before modifying the defaults. Issue an online or offline command to measure the time it takes for each action. To measure how long it takes to monitor a resource, fault the resource and issue a probe, or bring the resource online outside of VCS control and issue a probe.

HI Ted,

Ive attached the  version of VRTS in the file.

Marianne .i dont see any place that says  The defaults for the OnlineTimeout and OfflineTimeout attributes is 300 seconds. expet doing the test of online and offline the resource . i dont understand what you mean .

also is seems that after this error  the resource change to FAULTED state .

 

Hello,

 

First I would ask, what type of resource is this mdsuOracleLog_lv ?  is it a logical Volume ? or Oracle resource ?  you can tell this from main.cf ..

Secondly, from logs it appears that most of the resources are showing same messages, i.e monitor procedure failing.... so what I understand here is, multiple agents are facing same issue so that is not a single agent issue...  however it appears to me that entire server itself is busy..

Wht is the CPU/mem/io usage when u see these messages ?

 

what Marianne is saying above is about adjusting MonitorTimeout (not onlinetimeout or offlinetimeout which is 300 sec) .... MonitorInterval & MonitorTimeout are 60 sec ... you can tune this to a higher value so that agent gets more time to complete monitor cycle..  but as I mentioned above I believe server is struggling for resources.... check if server is busy..

 

Gaurav

Hi Gaurav,

this is for your first questions

    LVMLogicalVolume mdsuOracleLog_lv (
        LogicalVolume = mdsuOracleLog_lv
        VolumeGroup = mdsu_vg

Ive also attached some error ive got recently .

i will check about the cpu and memory . to see if there was overload on the system .

can you please tell me what evrey parmeter mean and waht he is doing ?

Oracle       OnlineTimeout           300

Oracle       OfflineTimeout          300

Oracle       OfflineMonitorInterval  300

Oracle       MonitorTimeout          60

Oracle       MonitorInterval         60

Hi ,

THX for the guide . also  i notice that LVMLogicalVolume was Terminate . and i know that if the LVMLogicalVolume become dont  it will cause the failover to occur ,i didnt find anything in the sar files regarding to cpu and memory problem . my only problem is to decide why the failover occuer at first place . and why  the APPDB  become FAULTED state . my only lead now is because a overload , but i didnt find anything regard this .

2010/09/11 10:23:07 VCS WARNING V-16-1-10023 Agent LVMLogicalVolume not sending alive messages since Sat 11 Sep 2010 10:20:52 AM MVT

2010/09/11 10:23:07 VCS NOTICE V-16-1-53026 Agent LVMLogicalVolume ipm connection still valid

2010/09/11 10:23:07 VCS NOTICE V-16-1-53027 Waiting one more try for ipm connection for agent LVMLogicalVolume to go away

2010/09/11 10:23:18 VCS WARNING V-16-1-10023 Agent LVMLogicalVolume not sending alive messages since Sat 11 Sep 2010 10:20:52 AM MVT

2010/09/11 10:23:18 VCS NOTICE V-16-1-53026 Agent LVMLogicalVolume ipm connection still valid

2010/09/11 10:23:18 VCS NOTICE V-16-1-53030 Termination request sent to LVMLogicalVolume agent process with pid 18032

2010/09/11 10:23:18 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/LVMLogicalVolume/LVMLogicalVolumeAgent for resource type LVMLogicalVolume

I see that you are using 5.0MP3RP2 version..

I was just looking at VCS release note for 5.1  here:

http://sfdoccentral.symantec.com/sf/5.1/linux/pdf/vcs_notes.pdf

 

I found this on page 50/51

=====================================================================

Known issues (Means this issue is still open & not fixed yet)

Poor agent performance and inability to heartbeat to the engine

If the system has more than 200 configured resources, the agent may not get enough CPUcycles to function properly. This can prevent the agent from producing a heartbeat synchronously with the engine. If you notice poor agent performance and an agent's inability to heartbeat to the engine, check for the following symptoms.

Navigate to /var/VRTSvcs/diag/agents/ and look for files that resemble:

FFDC_AGFWMain_729_agent_type.log

FFDC_AGFWTimer_729_agent_type.log core
FFDC_AGFWSvc_729_agent_type.log

agent_typeAgent_stack_729.txt

Where agent_type is the type of agent, for example Application or FileOnOff. If you find these files, perform the next step.
Navigate to /var/VRTSvcs/log/ and check the engine_*.log file for messages that resemble:

2009/10/06 15:31:58 VCS WARNING V-16-1-10023 Agent agent_type not sending alive messages since Tue Oct 06 15:29:27 2009
2009/10/06 15:31:58 VCS NOTICE V-16-1-53026 Agent agent_type ipm connection still valid
2009/10/06 15:31:58 VCS NOTICE V-16-1-53030 Termination request sent to agent_type agent process with pid 729

Workaround: If you see that both of the above criteria are true, increase the value of the AgentReplyTimeout attribute value. (Up to 300 seconds or as necessary.)
[1853285]

===================================================================================

As you can see, messages looks similar to what you are seeing, perform the above check to see if you see any core/log files, If yes, then perform the above mentioned workaround...

As this is an open issue, might be fixed with some patch in upcoming version...

 

Gaurav

Hi ,

 

THX , but for some reason i dont have the path

/var/VRTSvcs/diag/agents/

but still its exacly ther errors i've got .

i will check the workaround to see if it helps .

thx again .

So I searched on the "system messages" about extents you sent and found this:

http://oracle.su/docs/11g/server.112/e10595/schema002.htm

If I read this correctly, as long as the DB has a query waiting for it, additional queries / db transactions wouldn't take place.

So- if our Oracle monitor wants to touch or read a table in the db, then it won't be able to and you would get timeouts... because our monitor cycle can't complete.

If someone else has another take on these  messages and what they mean with the new information I provided, then certainly let me know. I am intrigued by this scenario...

Also- please provide the output of both of these commands:

ls -l /etc/VRTSvcs/conf/types.cf

ls -l /etc/VRTSvcs/conf/config/types.cf

Sometimes types.cf doesn't get updated in the config directory and can cause all kind of unexpected behaviors.