cancel
Showing results for 
Search instead for 
Did you mean: 

Multiple Service groups dependencies configuration.

hpoirier8
Level 3

hi,

I would like to expose my Cluster configuration problem based on Infoscale 7.4 installed on Linux  virtual environments (2 VMs), here to get some advices.
- Our application is splitted into three independant sub-applications : [Billing, Rating, Data ]. These 3 Sub-Applications are distributed on the 2 nodes "Active/Active" cluster as follows : [Data & Billing] on Node 1; Rating on Node2. The 3 sub-Applications are associated to a VIP.
- We will create 3 Service Groups (SG) : Data_SG; Billing_SG; Rating_SG.
- Dependencies between these 3 SG need to be set as follows : "Data_SG" will start first, THEN and ONLY when fully started, "Biling_SG" & "Rating_SG" will be started.
- When we start VCS, the SG are launched in a distributed mode, as mentioned above.
- In a distributed mode, "Billing_SG" leverages it own configuration File to start; "Rating_SG" leverages it own configuration file to start as well.
- In a Standalone Mode ("Billing_SG" and "Rating_SG" running on the same node), these 2 SG leverages a third configuration file which encapsulates the configuration of the 2 SG.
- So, if the "Billing_SG" OR the "Rating_SG" fails, they are supposed to try to restart on their original node to remain in a distributed configuration. But if it is not possible, both SG will be hosted on the same Cluster node, in StandAlone mode. In that case, there is a constraint which needs to be managed: we need to leverage the third configuration file as described just above. So it means that before switching the failed SG on the second node, it implies that we stop the remaining running SG, change the configuration files, and then restart the two SG on the same node.
I tried to illustrated different Used Cases in the Attached tab.
There is an other important constraint which is :
- If the "Data_SG" failed, we need to stop the "Billing_SG" and "Rating_SG" before trying to restart the "Data_SG", and then restart the "Billing_SG" and "Rating_SG" in their original location, where they were running before we stop them.

Based on these constraints, if you have any advice, It would be greetly appreciated
Regards

hubert

 

 

 

12 REPLIES 12

CliffordB
Level 4
Employee

Within InfoScale (VCS), we have the idea of per/node attributes.  By default, when you edit an attribute within a resrouce, it is for all nodes.  If needed, one can have different values for each node.   In your case, you would have a per/node start script or per/node config file for your application.

Hope that helps.

Cheers


_______________________________
My customers spend the weekends at home with family, not in the datacenter.

Hi Sir ,

Thanks for your reply.
I try to understand how I could implement your remarks in our configuration, i mean the one relating to the start Scripts ("you would have a per/node start script or per/node config file for your application").
Based on my previous explanations and the different Use Cases depicted in the document I attached to the initial discussion, Node1 is able to run one, two, or all Service Groups. the main problem is linked to the way to differenciate what is running on each node, even if the initial one is the one presented as "Use Case1" in my Attached document.
I tried to illustrate more precisely the configuration files used in the different use cases ...

Hope it clarifies a bit .

Thanks for your comments.
Rgds
Hubert

 

 

Hi ,

@CliffordB : To be more complete, I guess you refer to the following (hereafter an example) regarding attributes :

group billing_sg (
        SystemList = { node1 = 0, node2 = 1 }
        AutoStart = 0
        Parallel = 1
        PreOnline = 1
        )
        Application billing_app (
                StartProgram @node1 = "/apps/scripts/cluster/iSer_SRV_Node1_N.sh start"
                StartProgram @node2 = "/apps/scripts/cluster/iSer_SRV_Node2_N.sh start"
               ....
                )

is it something like that ?

When this application (Billing) is switching on Node2, the second one (Rating) is stopped first and both are restarted using the same script and same configuration file.

Hope it is more clear.

Rgds
hubert

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi Hubert,

What you need to do is build intelligence into your start up scripts. It should do something like this

Determine the number of active nodes in the cluster (can use output from gabconfig or something to figure this out)

if number of active nodes is less than 2, use combo configuration file, else use regular configuration file

You don't need to use local variable because your use case doesn't really require it. The service groups should also not be parallel (as shown in main.cf snippet)

Cheers

Riaan

 

hi,

I tried to find the appropriate configuration to be compliant with the requirements. I described it in the attached document.
The different Service Groups behaviors are described also. Now I need to elaborate the required intelligence to be compliant with the requirements.
I think I will have to elaborate the intelligence in a "preonline" trigger, set as Global or Local, which will be invoked in case of manual group online, group autostart, or group failover.

If someone has some other good ideas, any good idea, I would appreciate a lot.
Thanks
Rgds

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi Hubert,

Your requirements has 2 aspects that need to be addressed;

  1. Online/Start up of the billing & rating service groups using a specific configuration depending on the availability of resources i.e. how many nodes are online. If one node is down, or has an issue, then both services groups would need to be started on the same system and they need to use a different config file. This cannot be implemented in VCS without a custom script that modifies how the service groups startup. It could be implemented in preonline, but since you don't have any other resources in your groups, you could probably do it in the start script as well.
  2. Failover logic that will offline/online groups in a certain order. This can be done using service group dependencies or via VOM (if the dependencies in VCS are not adequate).

I would focus on point 1 first, and then worry about point 2.

Hi,

Thank for your contribution.

The solution could be something like that:
Two SGs dedicated to the Billing & Rating VIP, will be created:
- billing_VIP_sg,
- rating_VIP_sg.
These 2 SGs will contain only the VIP resource associated to each application; they will have the “AutoStart” property set to 1. These two SG will contain a “postonline” trigger in charge of starting their respective Application SG listed below, via the "hagrp -online <xxxxx_APP_SG -sys node" command :
- billing_APP_sg,
- rating_APP_sg.
The Application Service Groups will embed a “preonline” trigger containing the intelligence, able to manage the start method, according to the Use Cases defined previously. The Application SGs will have the “AutoStart” property set to 0.
is there some incompatibilities between the triggers "postonline" and "preonline" ?
do they trigger both ?

your remarks are welcome regarding this approach, to be able to be compliant with the initial problem.
Thanks for your cooperation, your advice.
Rgds

hubert

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi Hubert

I've spent quite a while checking the available options but the biggest problem is offlining the parent (upper level) SG when there has been no error in the child SG. Neither VCS nor VIOM can do that so the script has to.

I think this logic is required.

  1. Is this a normal startup i.e. is it starting on its autostart list node?
    1. If YES then check which configuration is in use on both SG's (in case we've failed over and busy failing back)
      1. If this is failback operation then
        1. offline both SG's
        2. fix config of both SGs (hares -modify app1 start "/x/y/normal" & hares -modify app2 start "/x/y/normal")
        3. hagrp -online SG1 -sys A
        4. hagrp -online SG1 -sys B
      2. If this is normal startup then
        1. start with normal config (hagrp -online SG1 -sys A or hagrp -online SG1 -sys B)
    2. If NO it needs to offline its neighbor because they will now share the same node and they both need to startup with the alternate configuration file
      1. offline the other SG (hagrp -offline SG2 -sys B or hagrp -offline SG1 -sys A)
      2. startup itself with alternate config
        1. update the start script to reflect alternate config (hares -modify app1 start "/x/y/alternate")
        2. hagrp -online SG1 -sys B (assuming A is where the issue occurred)
      3. startup other SG with alternate config
        1. update the start script to reflect alternate config (hares -modify app1 start "/x/y/alternate")
        2. hagrp -online SG2 -sys B (assuming A is where the issue occurred)

It's a little complicated :)

Hi,

Does it exist a way to know that VCS has just been started via "hastart" and the different group in "Autostart" mode just been started, versus a way to know that a SG  (or multiple) has just Switch-over or Fail-over ?

thanks for your cooperation

Rgds

H.

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

You could use hagrp -state -sys "system name" to check if the system is faulted or not. It's not so much about hastart or autostart, but rather whether the SG can run where it should, or if there has been a failure, and only one node is available.

Hi,

I am still working on this topics , but not everyday...

I am using the triggers available and add some perl code inside.

Is there somewhere some guide which explains how to write the code even if i already wrote some, or some examples, even complex one, which shows some tips and tricks ...

For example within a "resstatechange" trigger, I would like to know if there are some capabilities to enrich variables with the result of a cluster command like the following one or with basic system commands  :

my $groupState = `hagrp -state groupname -sys node1`

The previous command doesnt work at all but i need such command ..  how can I do that ?

Thanks for your help.

rgds

Hubert

hi,

Does anybody knows how to call a script (shell script) from a Veritas Cluster scripts ?
Is it like that :
VCSAG_SYSTEM("/opt/VRTSvcs/bin/triggers/billingAPPsg/ResStateChange.sh $ARGV[0] $ARGV[1] $ARGV[2] $ARGV[3]");
or like that :
SYSTEM("/opt/VRTSvcs/bin/triggers/billingAPPsg/ResStateChange.sh $ARGV[0] $ARGV[1] $ARGV[2] $ARGV[3]");

I read it was as the second method , starting with "system" but no way to trace what is happening inside the shell script.
I am looking for complex examples, to see what is it possible to do inside.

thanks for you cooperation and help.

Rgds

hubert