Oracle agent: "online procedure did not complete within the expected time" but Oracle instance is UP

Question

When I'm trying to put online an Oracle resource via Cluster Manager, even after the instance is really running (processes running), the resource is NOT online and VCS shows the following messages, running the 'clean' entry and SHUTING DOWN an active instance. Apparently not problems of system resources (CPU, speed of disks,...). Version SFHA 5.1 SP1 RP2P2. Any idea?
&nbsp;
2012/02/29 19:39:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource ora_pprod10 (Owner: Unspecified, Group: pprod10) on System caspio
	2012/02/29 19:44:04 VCS WARNING V-16-2-13012 (caspio) Resource(ora_pprod10): online procedure did not complete within the expected time.
	2012/02/29 19:44:04 VCS ERROR V-16-2-13065 (caspio) Agent is calling clean for resource(ora_pprod10) because online did not complete within the expected time.
	2012/02/29 19:44:06 VCS NOTICE V-16-20002-25 (caspio) Oracle:ora_pprod10:clean:Oracle shutdown returned the output
	+--------------------------------------------------------------------+
	LD_LIBRARY_PATH - /database/product/10.2.0.4/lib:/usr/lib:
	ORACLE instance shut down.
	+====================================================================+
	2012/02/29 19:44:06 VCS INFO V-16-2-13068 (caspio) Resource(ora_pprod10) - clean completed successfully.
	2012/02/29 19:44:06 VCS INFO V-16-2-13071 (caspio) Resource(ora_pprod10): reached OnlineRetryLimit(0).
	2012/02/29 19:44:06 VCS ERROR V-16-1-10303 Resource ora_pprod10 (Owner: Unspecified, Group: pprod10) is FAULTED (timed out) on sys caspio
	2012/02/29 19:44:37 VCS INFO V-16-1-10307 Resource ora_pprod10 (Owner: Unspecified, Group: pprod10) is offline on caspio (Not initiated by VCS)
&nbsp;
Thanks in advance.

marianne · Answer

Something wrong with Oracle resource definition?
Please post ora_pprod10 resource definition in main.cf.

robur · Answer

From main.cf:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Oracle ora_pprod10 (
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Pfile = "/database/product/10.2.0.4/dbs/initpprod10.ora"
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Owner = oracle10
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; User = vcs
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Table = sys_alive
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Home = "/database/product/10.2.0.4"
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; StartUpOpt = STARTUP
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Sid = pprod10
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Pword = HVNtKVkNInJNkTUvKTk
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AgentDebug = 1
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LevelTwoMonitorFreq = 1
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )
&nbsp;
From OracleTypes.cf:
type Oracle (
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; static boolean IntentionalOffline = 0
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; static str AgentDirectory = "/opt/VRTSagents/ha/bin/Oracle"
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; static keylist SupportedActions = { VRTS_GetInstanceName, VRTS_GetRunningServices, DBRestrict, DBUndoRestrict, DBResume, DBSuspend, D
	BTbspBackup, "home.vfd", "owner.vfd", getid, "pfile.vfd" }
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; static int RestartLimit = 2
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; static str ArgList[] = { Sid, Owner, Home, Pfile, StartUpOpt, ShutDownOpt, DBAUser, DBAPword, EnvFile, AutoEndBkup, DetailMonitor, Us
	er, Pword, Table, MonScript, AgentDebug, Encoding, MonitorOption }
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; static int ContainerOpts{} = { RunInContainer=1, PassCInfo=0 }
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; static str IMFRegList[] = { Home, Owner, Sid, MonitorOption }
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str Pfile
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str Owner
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str User
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str Table
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str EnvFile
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int DetailMonitor
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str Home
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str StartUpOpt = STARTUP_FORCE
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int MonitorOption
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str MonScript = "./bin/Oracle/SqlTest.pl"
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; boolean AutoEndBkup = 1
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str Encoding
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str Sid
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str DBAPword
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str Pword
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str DBAUser
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; str ShutDownOpt = IMMEDIATE
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; boolean AgentDebug = 0
	)
	
	&nbsp;

mikebounds · Answer

Problem is probably that you specified Pfile as Oracle usually uses an SPfile so if this is the case then you should leave pfile attribute blank. &nbsp;If this is not the issue then I would trying unsetting the optional attributes:
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; User = vcs
	&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Table = sys_alive
	&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Pword = HVNtKVkNInJNkTUvKTk
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; LevelTwoMonitorFreq = 1
If it works without setting these, then there is a problem with the second level monitoring.
If still having problems, then also give output of "ps -ef | grep pmon"
Mike

robur · Answer

Processes are running after 'startup' entry is executed, but Monitor (with DetailMonitor=1 and LevelTwoMonitorFreq=1) doesn't seem to realize about this.
At the moment, the resource of Oracle type named "ora_pprod10" with DetailMonitor=0 works fine.
"Pfile" parameter is defined in the same way and DetailMonitor=1 over the other 18 Oracle resources and all works fine.
###################################3
# ps -ef | grep pmon
oracle10 29874&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:19 /database/product/10.2.0.4/bin/tnslsnr listener_pprod10 -inherit
	&nbsp;
# ps -ef | grep pprod10
oracle10 27088&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:20 ora_qmnc_pprod10
	oracle10 27041&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2:40 ora_ckpt_pprod10
	oracle10 27033&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:11 ora_psp0_pprod10
	oracle10 27045&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 ora_reco_pprod10
	oracle10 27035&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:13 ora_mman_pprod10
	oracle10 27081&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:06 ora_arc0_pprod10
	oracle10 27037&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1:11 ora_dbw0_pprod10
	oracle10 27051&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 8:07 ora_mmnl_pprod10
	oracle10 27043&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:32 ora_smon_pprod10
	oracle10 27371&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 ora_q001_pprod10
	oracle10 27039&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3:46 ora_lgwr_pprod10
	oracle10 27031&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3:21 ora_pmon_pprod10
	oracle10 27049&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:52 ora_mmon_pprod10
	oracle10 27047&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6:11 ora_cjq0_pprod10
	oracle10 15361&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0 11:17:42 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 ora_q000_pprod10
	oracle10&nbsp; 1518&nbsp; 1517&nbsp;&nbsp; 0&nbsp;&nbsp; Mar 01 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 23:47 oraclepprod10 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
	oracle10 27083&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:11 ora_arc1_pprod10
	oracle10&nbsp; 3680&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0 05:06:07 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:01 ora_q004_pprod10
	oracle10 21923&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0 10:50:20 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:00 ora_q002_pprod10
	oracle10 12072&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Mar 01 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0:09 oraclepprod10 (LOCAL=NO)
	...
# ps -ef | grep -i oracle|grep -i agent
	&nbsp;&nbsp;&nbsp; root&nbsp; 8829&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp; 0&nbsp;&nbsp; Feb 29 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 31:41 /opt/VRTSagents/ha/bin/Oracle/OracleAgent -type Oracle -agdir /opt/VRTSagents/h
# ptree 8829
	8829&nbsp; /opt/VRTSagents/ha/bin/Oracle/OracleAgent -type Oracle -agdir /opt/VRTSagents/h
	&nbsp;

&nbsp;

marianne · Answer

Additional parameters needed for Level Two Monitoring.
See Setting up detail monitoring for VCS agents for Oracle in https://sort.symantec.com/public/documents/sfha/5.1sp1/solaris/productguides/pdf/vcs_oracle_agent_51sp1_sol.pdf
&nbsp;

Forum Discussion

Oracle agent: "online procedure did not complete within the expected time" but Oracle instance is UP

8 Replies

Related Content

Catalog migration - procedure

New Virtual Machine The operation timed out. 195"

VMware browse client list timing out with "socket time out error"

Catalog migration - procedure

Same kilobyte and backup time when backup "Full Backup" or "Diff Incr Backup"