Service group concurrency violation

The message 

2014/12/26 19:37:03 VCS INFO V-16-1-10299 Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) is online on mapibm625 (Not initiated by VCS)

means as it says the resource App_saposcol came online outside of VCS

This is not allowed (Concurrency Violation) so VCS brought it offline

2014/12/26 19:37:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) on System mapibm625

So you wouldn't see it offfline when you ran your commands as VCS brought it down within 1second of seeing it online.

So your issue is either:

1. Someone brought the app online outside of VCS control within the 5 mins prior to 19:37:03
2. The App_saposcol  resource is incorrectly reporting the resource is online - this is unlikely if you are using SAP agent, but if you are using Application agent then if you are using MonitorProcesses, then you should check the process is unique (for example the process is not the same as a test instance the app team maybe starting) or if using MonitorProgram, then you should check the script you are using.

Mike

When i gone with hares -display it shows,

App_saposcol        ConfidenceLevel       mapibm625  0
App_saposcol        ConfidenceLevel       mapibm626  100
App_saposcol        ConfidenceMsg         mapibm625
App_saposcol        ConfidenceMsg         mapibm626
App_saposcol        Flags                 mapibm625
App_saposcol        Flags                 mapibm626
App_saposcol        IState                mapibm625  not waiting
App_saposcol        IState                mapibm626  not waiting
App_saposcol        MonitorMethod         mapibm625  IMF
App_saposcol        MonitorMethod         mapibm626  Traditional
App_saposcol        Probed                mapibm625  1
App_saposcol        Probed                mapibm626  1
App_saposcol        Start                 mapibm625  0
App_saposcol        Start                 mapibm626  1
App_saposcol        State                 mapibm625  OFFLINE
App_saposcol        State                 mapibm626  ONLINE
App_saposcol        CleanProgram          global     /opt/VRTSvcs/bin/Saposcol/clean
App_saposcol        ComputeStats          global     0
App_saposcol        ContainerInfo         global     Type               Name            Enabled
App_saposcol        EnvFile               global
App_saposcol        MonitorProcesses      global
App_saposcol        MonitorProgram        global     /opt/VRTSvcs/bin/Saposcol/monitor
App_saposcol        PidFiles              global
App_saposcol        ResContainerInfo      global     Type               Name            Enabled
App_saposcol        ResourceInfo          global     State      Stale   Msg             TS
App_saposcol        ResourceRecipients    global
App_saposcol        StartProgram          global     /opt/VRTSvcs/bin/Saposcol/online
App_saposcol        StopProgram           global     /opt/VRTSvcs/bin/Saposcol/offline
App_saposcol        TriggerPath           global
App_saposcol        TriggerResRestart     global     0
App_saposcol        TriggerResStateChange global     0
App_saposcol        TriggersEnabled       global
App_saposcol        UseSUDash             global     0
App_saposcol        User                  global     root
App_saposcol        MonitorTimeStats      mapibm625  Avg        0       TS
App_saposcol        MonitorTimeStats      mapibm626  Avg        0       TS

=========================================================

I believe it has Moniter program configured,so what you suggests, what would be our next action plan to resolve this?

You need to debug /opt/VRTSvcs/bin/Saposcol/monitor (this is a custom script - not one provided by Symantec) to find out why it is reporting your application is online when you believe it is not online.

If your application is SAP, then you should look to see if you can use the Symantec SAP agents you can download from https://sort.symantec.com/agents. 

Mike

Its like this,

SU=/sbin/su
PS=/usr/bin/ps
SH=/usr/bin/sh
GREP=/usr/bin/grep
PGREP="/usr/bin/ps -ef | ${GREP}"
ECHO=/usr/bin/echo
SLEEP=/usr/bin/sleep
HALOG="/opt/VRTSvcs/bin/halog -add"
#

#
# This script is used by the application agent. It is done this way because
# the "MonitorProcesses" function looks for the exact string used to start it,
# unlike pgrep -f which can use an exact match of a partial string.
# It is preferred that VCS start it with the full path. If  it is started
# manually without the full path, VCS will not detect that it is running.
#

CMD=`/usr/bin/ps -ef | ${GREP} "saposcol" | ${GREP} -v grep | ${GREP} -v App_saposcol`
if [ "${CMD}" ]
then
        exit 110                # online (should be 110)
fi
exit 100                        # offline (should be 100)

=======================================================

I am unable to judge with this 110 or 100 value.

Wha does this command do,

CMD=`/usr/bin/ps -ef | ${GREP} "saposcol" | ${GREP} -v grep | ${GREP} -v App_saposcol`

Is that relates with the issue?

The custom script you have shown is not written well so this is your issue.  It is matching saposcol which is not specific enough as for instance if someone ran "vi saposcol" (or ANY file with saposcol somewhere in the filename), then this would be matched by the script and report to VCS that it was online.

You should use MonitorProcesses if you can:

run ps -ef   | grep saposcol

Copy the whole command in the CMD column output of VCS for the saposcol process (not the App_saposcol process if this exists) and use this for MoniorProcesses.  If you are unsure how to do this - please post the output of "ps -ef | grep saposcol" from the LIVE system.

Mike

In The second node mapibm626, when i tried with 

[root@mapibm626]:  # ps -ef |grep saposcol
  gtpadm 13893680        1   0   Dec 21      -  8:48 /usr/sap/hostctrl/exe/saposcol

as  saposcol shows online in mapibm626 and shows no o/p in mapibm625.