VCS 5.0 Apache agent: I cannot get the monitor entry point to work

[antonella_arche]

Hi all,

I'm trying to use the Veritas Apache agent bundled in VCS 5.0 but I cannot get the monitor entry point to work.

This is the way I have configured the agent:
       Apache ctm_apache (
               Critical = 0
               ResLogLevel = TRACE
               httpdDir = "/opt/CiscoTransportManagerServer/webServer/bin"
               EnvFile = "/opt/CiscoTransportManagerServer/webServer/bin/envvars"
               HostName = "10.58.65.12"
               Port = 8051
               ConfigFile = "/opt/CiscoTransportManagerServer/webServer/conf/httpd.conf"
               )

And when I put it online I get the httpd processes running with no problems:
#/usr/ucb/ps -auwwx | grep httpd
root     18197  0.0  0.0 1256 1048 console  S 16:38:52  0:00 grep httpd
root      8811  0.0  0.1 5584 3864 ?        S 15:06:02  0:00 /opt/CiscoTransportManagerServer/webServer/bin/httpd -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf -k start
nobody    8814  0.0  0.1 5728 4064 ?        S 15:06:02  0:00 /opt/CiscoTransportManagerServer/webServer/bin/httpd -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf -k start
nobody    8815  0.0  0.1 5712 4024 ?        S 15:06:02  0:00 /opt/CiscoTransportManagerServer/webServer/bin/httpd -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf -k start
nobody    8816  0.0  0.1 5712 4024 ?        S 15:06:02  0:00 /opt/CiscoTransportManagerServer/webServer/bin/httpd -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf -k start
nobody    8817  0.0  0.1 5712 4024 ?        S 15:06:02  0:00 /opt/CiscoTransportManagerServer/webServer/bin/httpd -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf -k start
nobody    8818  0.0  0.1 5712 4024 ?        S 15:06:02  0:00 /opt/CiscoTransportManagerServer/webServer/bin/httpd -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf -k start
nobody   10549  0.0  0.1 5712 4024 ?        S 15:08:57  0:00 /opt/CiscoTransportManagerServer/webServer/bin/httpd -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf -k start

but after onlining, once the Apache agent checks the processes in its monitor entry point it looks like it cannot find them online:
V-16-2-13066
Agent is calling clean for resource(ctm_apache) because the resource is not up even after online completed.

I took an extract from the engine_A.log (please note lines in bold):

2007/05/31 15:06:02 VCS INFO V-16-55005-10408 (saphira) Apache:ctm_apache:online:Sys:RunWithEnvCmdWithOutputWithTimeOut:Subroutine Sys::RunW
ithEnvCmdWithOutputWithTimeOut called with: EnvFile [/opt/CiscoTransportManagerServer/webServer/bin/envvars], Command [/opt/CiscoTransportMa
nagerServer/webServer/bin/httpd], Arguments [ -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf   -k start], User [root], Timeou
t [SCALAR(0x4b2414)], OutFile [<undef>], FromDir [<undef>]
2007/05/31 15:06:02 VCS INFO V-16-55005-10413 (saphira) Apache:ctm_apache:online:Sys:RunWithEnvCmdWithOutputWithTimeOut:User [root] has sh d
erived native shell [/sbin/sh]
2007/05/31 15:06:02 VCS INFO V-16-55005-10417 (saphira) Apache:ctm_apache:online:Sys:RunWithEnvCmdWithOutputWithTimeOut:Going to run command
 line [USER=root; HOME=/; export USER HOME ; . /opt/CiscoTransportManagerServer/webServer/bin/envvars ;/opt/CiscoTransportManagerServer/webS
erver/bin/httpd  -f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf   -k start], as User [root]
2007/05/31 15:06:02 VCS INFO V-16-55005-10418 (saphira) Apache:ctm_apache:online:Sys:RunWithEnvCmdWithOutputWithTimeOut:Sourcing the login e
nvironment of user [root]
2007/05/31 15:06:02 VCS INFO V-16-55005-10212 (saphira) Apache:ctm_apache:online:VCSagentFW:messageEngineLog:[Sun Microsystems Inc.     SunO
S 5.10  Generic January 2005
stty: : No such device or address
]
2007/05/31 15:06:02 VCS INFO V-16-55005-10195 (saphira) Apache:ctm_apache:online:VCSagentFW:entryPointExit:Exiting entry point [online] with
 exit code [10]
2007/05/31 15:06:13 VCS NOTICE V-16-55005-10098 (saphira) Apache:ctm_apache:monitor:LogInt:SetDebugLevel:Information, Set Logging Level To [
TRACE]
2007/05/31 15:06:13 VCS INFO V-16-55005-10192 (saphira) Apache:ctm_apache:monitor:VCSagentFW:SetupLogging:[monitor] Entered by resource inst
ance [ctm_apache]
2007/05/31 15:06:13 VCS INFO V-16-55005-20057 (saphira) Apache:ctm_apache:monitor:Subroutine <Apache::GetMonitorTimeout> called
2007/05/31 15:06:13 VCS INFO V-16-55005-10408 (saphira) Apache:ctm_apache:monitor:Sys:RunWithEnvCmdWithOutputWithTimeOut:Subroutine Sys::Run
WithEnvCmdWithOutputWithTimeOut called with: EnvFile [], Command [/opt/VRTSvcs/bin/hatype], Arguments [-value Apache MonitorTimeout], User [
root], Timeout [10], OutFile [<undef>], FromDir [<undef>]
2007/05/31 15:06:13 VCS INFO V-16-55005-10410 (saphira) Apache:ctm_apache:monitor:Sys:RunWithEnvCmdWithOutputWithTimeOut:Environment file no
t set
2007/05/31 15:06:13 VCS INFO V-16-55005-10417 (saphira) Apache:ctm_apache:monitor:Sys:RunWithEnvCmdWithOutputWithTimeOut:Going to run comman
d line [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout], as User [root]
2007/05/31 15:06:13 VCS INFO V-16-55005-10212 (saphira) Apache:ctm_apache:monitor:VCSagentFW:messageEngineLog:[60
]
2007/05/31 15:06:13 VCS INFO V-16-55005-20058 (saphira) Apache:ctm_apache:monitor:<Apache::GetMonitorTimeout> GetMonitorTimeout is [60
]
2007/05/31 15:06:13 VCS INFO V-16-55005-20001 (saphira) Apache:ctm_apache:monitor:Subroutine <Apache::ArgsValid> called with:
 EntryPointName          [monitor]
 HttpDir                 [/opt/CiscoTransportManagerServer/webServer/bin]
2007/05/31 15:06:13 VCS INFO V-16-55005-10417 (saphira) Apache:ctm_apache:monitor:Sys:RunWithEnvCmdWithOutputWithTimeOut:Going to run comman
d line [/opt/VRTSvcs/bin/hatype -value Apache MonitorTimeout], as User [root]
2007/05/31 15:06:13 VCS INFO V-16-55005-10212 (saphira) Apache:ctm_apache:monitor:VCSagentFW:messageEngineLog:[60
]
2007/05/31 15:06:13 VCS INFO V-16-55005-20058 (saphira) Apache:ctm_apache:monitor:<Apache::GetMonitorTimeout> GetMonitorTimeout is [60
]
2007/05/31 15:06:13 VCS INFO V-16-55005-20001 (saphira) Apache:ctm_apache:monitor:Subroutine <Apache::ArgsValid> called with:
 EntryPointName          [monitor]
 HttpDir                 [/opt/CiscoTransportManagerServer/webServer/bin]
 EnvFile                 [/opt/CiscoTransportManagerServer/webServer/bin/envvars]
 SharedObjDir            []
 MonitorGetTimeout       [60
]
 Host                    [10.58.65.12]
 Port                    [8051]
 User                    []
 SecondLevelMonitor      [0]
 SecondLevelTimeout      [30]
 ConfigFile              [/opt/CiscoTransportManagerServer/webServer/conf/httpd.conf]
 DirectiveAfter          []
 DirectiveBefore         []
2007/05/31 15:06:13 VCS INFO V-16-55005-20003 (saphira) Apache:ctm_apache:monitor:<Apache::ArgsValid> All mandatory fields are set
2007/05/31 15:06:13 VCS INFO V-16-55005-20061 (saphira) Apache:ctm_apache:monitor:All arguments validated successfully.
2007/05/31 15:06:13 VCS INFO V-16-55005-20025 (saphira) Apache:ctm_apache:monitor:Subroutine <Apache::GetResourceState> called:
 httpDir                 [/opt/CiscoTransportManagerServer/webServer/bin]
 ConfigFile              [/opt/CiscoTransportManagerServer/webServer/conf/httpd.conf]
2007/05/31 15:06:13 VCS INFO V-16-55005-20030 (saphira) Apache:ctm_apache:monitor:Subroutine <Apache::GetProcState> called:
 ConfigFile               [/opt/CiscoTransportManagerServer/webServer/conf/httpd.conf]
 rliPIDsWithPPIDOfOne     [ARRAY(0x4ec21c)]
 rliPIDsWithoutPPIDOfOne  [ARRAY(0x4ec240)]
2007/05/31 15:06:13 VCS INFO V-16-55005-10256 (saphira) Apache:ctm_apache:monitor:Proc:GetProcessListHash:Subroutine Proc::GetProcessListHas
h is called
2007/05/31 15:06:13 VCS INFO V-16-55005-10257 (saphira) Apache:ctm_apache:monitor:Proc:GetProcessListHash:Calling ps command [/bin/ps] with
options [-e -o 'uid,pid,ppid,args']
2007/05/31 15:06:13 VCS INFO V-16-55005-10251 (saphira) Apache:ctm_apache:monitor:Proc:GetDefunctProcessListHash:Subroutine Proc::GetDefunct
ProcessListHash is called
2007/05/31 15:06:13 VCS INFO V-16-55005-10252 (saphira) Apache:ctm_apache:monitor:Proc:GetDefunctProcessListHash:Calling ps command [/bin/ps
] with options [-ef]
2007/05/31 15:06:13 VCS INFO V-16-55005-10226 (saphira) Apache:ctm_apache:monitor:Arch:GetOSType:Subroutine Arch::GetOSType is called
2007/05/31 15:06:13 VCS INFO V-16-55005-10255 (saphira) Apache:ctm_apache:monitor:Proc:GetDefunctProcessListHash:Got [0] defunct processes
2007/05/31 15:06:13 VCS INFO V-16-55005-10261 (saphira) Apache:ctm_apache:monitor:Proc:GetProcessListHash:Got [170] processes
2007/05/31 15:06:13 VCS INFO V-16-55005-20032 (saphira) Apache:ctm_apache:monitor:<Apache::GetProcState> Call to get process list returned [
170] processes
2007/05/31 15:06:13 VCS INFO V-16-55005-10269 (saphira) Apache:ctm_apache:monitor:Proc:FilterProcs:Subroutine Proc::FilterProcs is called wi
th: Filter [-f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf]
2007/05/31 15:06:13 VCS INFO V-16-55005-10271 (saphira) Apache:ctm_apache:monitor:Proc:FilterProcs:Got [0] matches
2007/05/31 15:06:13 VCS INFO V-16-55005-20033 (saphira) Apache:ctm_apache:monitor:<Apache::GetProcState> Matches for process [httpd] with fi
lter [-f /opt/CiscoTransportManagerServer/webServer/conf/httpd.conf] got [0] matches
2007/05/31 15:06:13 VCS INFO V-16-55005-20027 (saphira) Apache:ctm_apache:monitor:<Apache::GetResourceState> GetResourceState found that res
ource was offline
2007/05/31 15:06:13 VCS INFO V-16-55005-10195 (saphira) Apache:ctm_apache:monitor:VCSagentFW:entryPointExit:Exiting entry point [monitor] wi
th exit code [100]


Any suggestion on what maybe wrong and cause this?

Thanks,
Antonella

[Gene_Henriksen]

Antonella,

A couple of ideas:
1) The Host name is listed as an IP address. The Bundled Agents Reference Guide says to fill this in only if you are enabling second level monitoring. I don't have the monitor script handy, nor time to read thru it, but it the doc fuster says it should be in the form web1.veritas.com.

2) Toward the bottom, it looks like it is trying to get a match on a specific string from the ps command. Sometimes ps truncates long process names. If this is happening, the result is what you are seeing. Try creating a symbolic link to the bin and conf to shorten the path like /Apache/bin as a link to the bin directory.

[antonella_arche]

Thanks Gene,

I tried your second suggestion and it does work! I will try now also the second level monitoring.

Thanks,
Antonella

[Gene_Henriksen]

Be sure the Host name is correct and that the port is filled in if it is not 80 (default).

Glad I could help.

[antonella_arche]

Hi Gene,

just for info: I tried and second level monitoring works also leaving the ip as HostName. I guess it really depends on how the Apache web server is configured.

Great! it seems we will finally be able to replace our proprietary Apache agent with the Veritas provide one.

Thanks again,
Antonella