Application Resource failing while doing SG Fail-over test

Hi Ashish,

Can you paste whatever is there in engine_A.log & resource definition from main.cf ?

Also, is the monitor script same for Application resource on both the nodes ? It is quite possible that monitor script on Active node is diffferent than to passive node which is why resource is not getting online on node B.

VCS needs exit codes to be defined as 110 (for successful) & 100 (unsuccesful) for monitoring a resource. Compare the monitor program on both the nodes & see the results.

G

Hi Gaurav,

For checking the functionality of the statrting script i had copied the same from active one to standby.. its exit status was 110 as you mentioned, when it started. For your reference, i'm attaching the engine logs, when i started the failopver testing to standby node and the main.cf entried for the SG on that node.

Regards,

Ashish C 

Hi Ashish,

This typically happens when there are setup differences between the two nodes.

Please check and compare the setup on both the nodes (where the application resource is coming online and where it is not coming online). You will have to check the following files for differences on the two nodes - MonitorProgram / MonitorProcesses / PidFiles (all that you have configured).

HTH.

Hi Ashish,

From the engine log, it seems that the application resource was never tried to bring online on the system - PmsProd26. Online was fired only for the resources upto IP resource.

If online was tried for the Application resource on PmsProd26, then can you please provide the relevant section of engine logs?

Thanks.

Hi Setu,

I've checked the monitoring scripts on bothe the nodes, it looks same. Attaching the same for your reference. I didn't tried with replacing the script on PmsProd26, my standby node with the active node.

** Is there any issues, if the PATH for vsc and the application are not set in ./root/.bash_profile  ??

It was not there previously, just now only set the same for root.

Regards,

Ashish

Hi Ashish,

The diff of both your monitor scripts reveals that there is some difference in the commands executed during online. Please copy the working file on not-working node and check the result of online operation.

There might be issues if PATH for VCS or application are not set in /root/.bash_profile if your scripts does not use the complete path at any place.

Thanks,

Setu.

Which node is the active node - PmsProd25 or PmsProd26.

Which version of VCS are you using?

Mike

Hi Mike,

PmsProd25 is active node.

VCS version is 

Engine Version    5.1
Join Version      5.1.10.0
Build Date        Fri 01 Oct 2010 12:00:00 PM IST
PSTAMP            5.1.100.000-5.1SP1GA-2010-09-30_23.30.00

OS version is

Red Hat Enterprise Linux Server release 5.6 (Tikanga)

Kernal Version is 

Linux PmsProd26.IDEACONNECT.COM 2.6.18-238.el5 #1 SMP Sun Dec 19 14:22:44 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

Regards,

Ashish C

In 5.1 there is the following attribute for the Application agent:

UseSUDash
When the value of this attribute is 0, the agent performs an su User command before it executes the StartProgram, the StopProgram, the MonitorProgram, or the CleanProgram agent functions.
When the value of this attribute is 1, the agent performs an su - User command before it executes the StartProgram, the StopProgram, the MonitorProgram or the CleanProgram agent functions.
Type and dimension: boolean-scalar
Default: 0

You have not set this which means it takes the default of 0, so the profile will not be run so you will need to include any environment variables you need in your scripts, or change the attribute to 1.

If this does not fix your issue, then please explain your issue further as the logs do not match your description as in your opening post you say you:

"made the resources online one by one on the secondary node", but the logs do not show you onlining resource HPPI_Appl on PmsProd26

Also the logs show:

2014/03/21 16:23:32 VCS ERROR V-16-2-13067 (PmsProd25) Agent is calling clean for resource(HPPI_Appl) because the resource became OFFLINE unexpectedly, on its own.

Was this a test you did killing the processes on PmsProd25, or do you have an issue on PmsProd25 too?

Mike

Looking at script on both the nodes

in the "online" loop

on PmsProd25

#############  # FUNCTION: Called by main() when ($MODE = "Run_OVPI.ksh")
OVPI_online()  # RETURN: 0 if success, else 1.
#############  # Start piweb, and optionally trendtimer if not WAS-ONLY/
{
 typeset Res

 if [ ${OVPI_SCENARIO} -ne ${WAS_ONLY} ] ; then

    # Don't start trendtimer if another copy of it is already running:
    #
    Res=`ps -ef | grep "trendtimer" | grep -v grep`
    if [ ! "$Res" ] ; then
       #
       # Implement same mechanism used by "/etc/init.d/ovpi_timer start" :
       #
       logit I "OVPI_online() STARTING trendtimer ...."

       /bin/su - $PIUSER -c "${PIHOME}/bin/trendtimer -s \
         ${PIHOME}/lib/trendtimer.sched"  >StartTtimerOUT  2>StartTtimerERR
       sleep 5
       /bin/su - $PIUSER -c "${PIHOME}/bin/trendtimer -s \
         ${PIHOME}/lib/trendtimer_IDEA.sched"  >StartTtimerOUT  2>StartTtimerERR
       sleep 5
    fi
 fi
 #THIS CODE CHECKS THE APPROPRIATE OS AND STARTS PIWEB ACCORDINGLY #
 logit I "OVPI_online() STARTING piweb ...."
 startjboss
 logit I "OVPI_online() WAITING 3 seconds after piweb start ...."
 sleep 3

 logit I "OVPI_online() EXIT 0"
 return 110
}

While on PmsProd26

#############  # FUNCTION: Called by main() when ($MODE = "Run_OVPI.ksh")
OVPI_online()  # RETURN: 0 if success, else 1.
#############  # Start piweb, and optionally trendtimer if not WAS-ONLY/
{
 typeset Res

 if [ ${OVPI_SCENARIO} -ne ${WAS_ONLY} ] ; then

    # Don't start trendtimer if another copy of it is already running:
    #
    Res=`ps -ef | grep "trendtimer" | grep -v grep`
    if [ ! "$Res" ] ; then
       #
       # Implement same mechanism used by "/etc/init.d/ovpi_timer start" :
       #
       logit I "OVPI_online() STARTING trendtimer ...."

       /bin/su - $PIUSER -c "${PIHOME}/bin/trendtimer -s \
         ${PIHOME}/lib/trendtimer.sched"  >StartTtimerOUT  2>StartTtimerERR
       sleep 5
    fi
 fi
 #THIS CODE CHECKS THE APPROPRIATE OS AND STARTS PIWEB ACCORDINGLY #
 logit I "OVPI_online() STARTING piweb ...."
 startjboss
 logit I "OVPI_online() WAITING 3 seconds after piweb start ...."
 sleep 3

 logit I "OVPI_online() EXIT 0"
 return 110
}

As you can see above, node 25 is bringing up "trendtimer" twice .. one for

trendtimer.sched

& second time for

trendtimer_IDEA.sched

This should same on both the nodes right ?

G