Monitoring Cluster Resources

[neilECM]

Hi,

Working on a new project using Veritas Storage Foundation 5.0x on Linux. Environment consists of 2 machines or nodes within the cluster each with redundant software installed for failover purposes.

Plan to use the Veritas HA agent for WebSphere to monitor the WebSphere process we need.

I have some other components or software processes that will require monitoring as well. In order to monitor these processes, I imagine that I will have to develop or write scripts to monitor these software processes. Should a process fail, it will have to have the ability fail over to the other node.  Have been reading the Veritas Cluster Server Agents Developers Guide as a start. I plan to use scripts to write the custom agents.

From what I gather thus far, each process which needs monitoring requires basic entry points:

online

offline

monitor

clean

Right now we are in the planning phase of the project still so we have not installed the veritas cluster software yet. For the additional components or software processes I need to monitor, they need to have the ability to failover to the other node just as the WebSphere veritas agent will do for us.

Not quite sure how the monitoring entry point works. For example, say I start process A with the command: startProcessA.sh   I can then grep for process A to see that it's running.

What code is used in the monitoring entry point to check to see if process A is running?

Are there any templates or example of agents for monitoring processes for failover purposes?

Thank you.

[M__Braun · Accepted solution]

Most applications can be handled by the Application Agent. See the following example from the Bundled Agents Guide:

ftp://exftpp.symantec.com/pub/support/products/ClusterServer_UNIX/306957.pdf

Application samba_app (
User = "root"
StartProgram = "/usr/sbin/samba start"
StopProgram = "/usr/sbin/samba stop"
PidFiles = { "/var/lock/samba/smbd.pid" }
MonitorProcesses = { "nmbd" }
)

TheAgent Builder is another option:

ftp://exftpp.symantec.com/pub/support/products/ClusterServer_UNIX/317352.pdf

I hope this helps.

Regards

Manuel

[neilECM]

Thank you this is just what I was looking for.

I see I can use MonitorProcesses to determine if a certain process or application is running.

Also discussed is MonitorProgram as an additional option.  I imagine if I wrote a script for the monitoring program it would grep for the process and return either 100 or 110 to determine if the process is running?

Is there any advantage to using  MonitorProgram?  Will MonitorProcesses more or less provide the same functionality?

Thank you.

[M__Braun]

> I imagine if I wrote a script for the monitoring program it would grep for the process and return either 100 or 110 to determine if the process is running?

Correct. Here is an example:

#!/bin/ksh
#
# Application-Agent monitorscript
#
RESNAME="$1"
PS=/usr/bin/ps
GREP=/usr/bin/grep
CAT=/usr/bin/cat
AWK=/usr/bin/awk
WC=/usr/bin/wc
ECHO=/usr/bin/echo
HALOG=/opt/VRTSvcs/bin/halog
PGREP=/usr/bin/pgrep
ONLINE_SMBD=99
ONLINE_NMBD=99

ONLINE_NMBD=`pgrep -u root nmbd | $WC -l`
ONLINE_SMBD=`pgrep -u root smbd | $WC -l`

if [ $ONLINE_NMBD -eq 1 ]
   then
    if [ $ONLINE_SMBD -ge 1 ]
       then
        exit 110                ; online status
       else
            exit 100                ; offline status
    fi
   else   
        exit 100             ; offline status
fi

exit 100

> Is there any advantage to using  MonitorProgram?  Will MonitorProcesses more or less provide the same functionality?

Yes. the functionality is comparable. Scripting just gives more flexibility.

Regards

Manuel

[TomerG]

One advantage to the MonitorProgram is that you can make it do what VCS often terms "detailed monitoring". For example, instead of just checking processes, you might also, or instead, do a test to see that the application is actually running and responding (since process running may be hung, so process running doesn't actually tell you if the app is alive). 

The Oracle and other database agents often have something similar as an optional secondary level of monitoring that does SQL queries to ensure the database is actually working.

If you don't care about that level of monitoring, then just use the MonitorProcesses (or PidFiles, if your app uses Process ID files like Samba and Apache do).

[oogunmadewa]

Hello Tomer,

I saw your posting and thought you may be of help...

I need help completing my configuration of VCS Application Agent Builder.
I'm not sure how to move forward after creating agents.

I have installed the ACC Library and the Builder RPMs.
I have created agent for my application running on 2 nodes using the following command:
./agentbuilder FOGLIGHT_MS_HA -base vcs5 -platform linux ssh -verbose -system rocsedvvi001.usv-sba.com -system rocsedvvi002.usv-sba.com

The following were created on both nodes:
AGENT_HOME=/opt/VRTSagents/ha/bin/FOGLIGHT_MS_HA

CONTENTS OF AGENT_HOME:
=========================
lrwxrwxrwx 1 root root     7 Apr  7 13:06 clean -> monitor
-r-x------ 1 root root 92995 Apr  7 13:05 FOGLIGHT_MS_HA.pm
-r-x------ 1 root root 10860 Apr  7 13:05 FOGLIGHT_MS_HA.xml
-r-x------ 1 root root 20624 Apr  7 13:05 monitor
lrwxrwxrwx 1 root root     7 Apr  7 13:06 offline -> monitor
lrwxrwxrwx 1 root root     7 Apr  7 13:06 online -> monitor

APPLICATION INFORMATION:
=======================

# ps -ef|grep -i oaadmfl
oaadmfl  14228     1  0 Mar23 ?        00:00:49 Foglight High Availability 5.2.4: FoglightHA Daemon
oaadmfl  14247 14228  1 Mar23 ?        05:10:09 foglight 5.2.4: Foglight

ApplicationHome=/app/quest/Foglight/
ApplicationBin=/app/quest/Foglight/bin
ApplicationStop=# ./fmsShutdownHA.sh
ApplicationStart=fmsStartupHA.sh

ApplicationURL=https://<ServerName>:8444/console/page/main

Thanks!
 

[TomerG]

 Sorry oog... don't know what the question is. If this is a separate question, you might want to post in a new thread.

[Satish_K__Pagar]

Hi Oog,

You need to create a resource group and a resource with the type(FOGLIGHT_MS_HA) that you created using the agentbuilder tool. The agent builder guide available at http://seer.entsupport.symantec.com/docs/317352.htm which has some sample configurations which should help you clustering your app.

Hope that helps,
Satish/