VCS 6.0.1 on Solaris x64 apache resource not comin...

Rick_Smith1 · ‎11-12-2013

2013/11/12 15:31:44 VCS INFO V-16-1-10298 Resource res_mysql (Owner: Unspecified, Group: testgrp) is online on solaris10-1 (VCS initiated)

2013/11/12 15:31:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource res_apache (Owner: Unspecified, Group: testgrp) on System solaris10-1

2013/11/12 15:32:36 VCS NOTICE V-16-10061-20284 (solaris10-1) Apache:res_apache:online:KillPIDandAllChildren:Proc:[SIGTERM] delivered to:

PID CMD

----------------

[12992] [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout]

----------------

Total: [1]

2013/11/12 15:32:59 VCS NOTICE V-16-10061-20284 (solaris10-1) Apache:res_apache:online:KillPIDandAllChildren:Proc:[SIGKILL] delivered to:

PID CMD

----------------

[12992] [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout]

----------------

Total: [1]

2013/11/12 15:33:20 VCS ERROR V-16-10061-20460 (solaris10-1) Apache:res_apache:online:Sys:RunWithEnvCmdWithOutputWithTimeOut:The command [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout] did not complete within [10] seconds

2013/11/12 15:33:41 VCS ERROR V-16-10061-20376 (solaris10-1) Apache:res_apache:online:<Apache::GetMonitorTimeout> The command line [/opt/VRTSvcs/bin/hatype-value Apache MonitorTimeout] did not complete within the allotted

amount of time ( [10] seconds )

2013/11/12 15:34:02 VCS ERROR V-16-10061-20312 (solaris10-1) Apache:res_apache:online:<Apache::ArgsValid> SecondLevelMonitorTimeOut must be less than MonitorTimeOut.

2013/11/12 15:34:09 VCS ERROR V-16-2-13027 (solaris10-1) Resource(res_apache) - monitor procedure did not complete within the expected time.

2013/11/12 15:34:35 VCS NOTICE V-16-10061-20284 (solaris10-1) Apache:res_apache:monitor:KillPIDandAllChildren:Proc:[SIGTERM] delivered to:

PID CMD

----------------

[13080] [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout]

----------------

Total: [1]

2013/11/12 15:34:58 VCS NOTICE V-16-10061-20284 (solaris10-1) Apache:res_apache:monitor:KillPIDandAllChildren:Proc:[SIGKILL] delivered to:

PID CMD

----------------

[13080] [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout]

----------------

Total: [1]

2013/11/12 15:35:29 VCS ERROR V-16-10061-20460 (solaris10-1) Apache:res_apache:monitor:Sys:RunWithEnvCmdWithOutputWithTimeOut:The command [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout] did not complete within [10] seconds

2013/11/12 15:36:01 VCS NOTICE V-16-10061-20284 (solaris10-1) Apache:res_apache:monitor:KillPIDandAllChildren:Proc:[SIGTERM] delivered to:

PID CMD

----------------

[13140] [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout]

----------------

Total: [1]

2013/11/12 15:36:07 VCS ERROR V-16-2-13066 (solaris10-1) Agent is calling clean for resource(res_apache) because the resource is not up even after online completed.

2013/11/12 15:36:11 VCS ERROR V-16-10061-20376 (solaris10-1) Apache:res_apache:monitor:<Apache::GetMonitorTimeout> The command line [/opt/VRTSvcs/bin/hatype-value Apache MonitorTimeout] did not complete within the allotted

amount of time ( [10] seconds )

2013/11/12 15:36:53 VCS NOTICE V-16-10061-20284 (solaris10-1) Apache:res_apache:monitor:KillPIDandAllChildren:Proc:[SIGKILL] delivered to:

PID CMD

----------------

[13140] [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout]

----------------

Total: [1]

bash-3.2# cat /etc/VRTSvcs/conf/config/main.cf

include "OracleASMTypes.cf"

include "types.cf"

include "Db2udbTypes.cf"

include "MySQLTypes51.cf"

include "OracleTypes.cf"

include "SybaseTypes.cf"

cluster ourcluster (

UserNames = { admin = aPQiPKpMQlQQoYQkPN }

ClusterAddress = "10.0.0.43"

Administrators = { admin }

)

system solaris10-1 (

)

system solaris10-2 (

)

group ClusterService (

SystemList = { solaris10-1 = 0, solaris10-2 = 1 }

AutoStartList = { solaris10-1, solaris10-2 }

OnlineRetryLimit = 3

OnlineRetryInterval = 120

)

IP webip (

Device = e1000g0

Address = "10.0.0.43"

NetMask = "255.255.255.0"

)

NIC csgnic (

Device = e1000g0

)

webip requires csgnic

// resource dependency tree

//

// group ClusterService

// {

// IP webip

// {

// NIC csgnic

// }

group testgrp (

SystemList = { solaris10-1 = 0, solaris10-2 = 1 }

ContainerInfo = { Name = testzone, Type = Zone, Enabled = 1 }

AutoStartList = { solaris10-1, solaris10-2 }

Administrators = { admin }

)

Apache res_apache (

ResLogLevel = TRACE

httpdDir = "/apps2/apache2/bin"

PidFile = "/apps2/apache2/logs/httpd.pid"

HostName = testzone

User = root

ConfigFile = "/apps2/apache2/conf/httpd.conf"

)

MySQL res_mysql (

Critical = 0

ResLogLevel = TRACE

MySQLAdminPasswd = mypwd

BaseDir = "/opt/mysql/mysql"

DataDir = "/opt/mysql/mysql/data"

MyCnf = "/opt/mysql/mysql/my.cnf"

HostName = testzone

)

Zone vcszone (

)

Zpool vcspool (

PoolName = spool

AltRootPath = "/"

ZoneResName = vcszone

)

res_apache requires res_mysql

res_mysql requires vcszone

vcszone requires vcspool

// resource dependency tree

//

// group testgrp

// {

// Apache res_apache

// {

// MySQL res_mysql

// {

// Zone vcszone

// {

// Zpool vcspool

// }

bash-3.2#

kjbss · ‎11-13-2013

The problem is that the command:

/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout

is taking more than 10 seconds to execute. That is crazy-long for such a simple command -- on a normal system this command completes in 0.01 seconds:

timex /opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout
60

real           0.01
user           0.00
sys            0.00

So you need to manually run that command and see what you get.

If it is hanging, you need to investigate why. You may need to restart some parts of the VCS service-set; though it might simply be easier to restart VCS on that server.

Is there more than one VCS node in the cluster? If so, try that command on another node; that should indicate whether the problem is node-specific, of "cluster-wide"...

-HTH

VOX

VCS 6.0.1 on Solaris x64 apache resource not coming up