Forum Discussion

tanislavm's avatar
tanislavm
Level 6
11 years ago

Troubleshooting when an application hangs on an node within VCS

Hi, If an application hang on an node within vcs,i like to verify how I troubleshoot this issue. Should I use the stop script in main.cf to cleanly stop this application?Or kill -9 the application processes?Then start the group hagrp -online? thanks so much.

 

<<title edited by admin to add further descriptiveness>>

business continuity
Solaris
Storage Foundation
  • mikebounds's avatar
    mikebounds
    11 years ago

    If VCS detects the application is hung, then VCS will call a clean which will forcabily stop the application and then take further action depending on how you have configured VCS - so for instance if you have set RestartLimit on the resource type then VCS will restart the application, else in other configurations will failover the group to another system.

    If VCS does NOT detect the application is hung, then if you have set RestartLimit on the resource type then you could kill -9 the application processes and then VCS will restart, but if RestartLimit is not set and you don't application to failover to another system, then you could offline using VCS (hares -offline or use GUI) an this will try to gracefully stop the application and of this doesn't work, VCS will call a clean.  Alternatively, you freeze service group (hagrp -freeze or use GUI) and kill -9 the application processes (freezing group means VCS will not take action when it sees process dies) and then restart application manually or using VCS.

    Mike

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    If VCS detects the application is hung, then VCS will call a clean which will forcabily stop the application and then take further action depending on how you have configured VCS - so for instance if you have set RestartLimit on the resource type then VCS will restart the application, else in other configurations will failover the group to another system.

    If VCS does NOT detect the application is hung, then if you have set RestartLimit on the resource type then you could kill -9 the application processes and then VCS will restart, but if RestartLimit is not set and you don't application to failover to another system, then you could offline using VCS (hares -offline or use GUI) an this will try to gracefully stop the application and of this doesn't work, VCS will call a clean.  Alternatively, you freeze service group (hagrp -freeze or use GUI) and kill -9 the application processes (freezing group means VCS will not take action when it sees process dies) and then restart application manually or using VCS.

    Mike

  • tanislavm's avatar
    tanislavm
    Level 6
    11 years ago

    Hi Mike,

    Thanks so much.Could i start safely the application using hagrp -online,if the other group resources are online?Or better to stop whole group hagrp -offline and then start the group?

  • Gaurav_S's avatar
    Gaurav_S
    Moderator
    11 years ago

    Hi,

    you would do "hagrp -online" if the group is either offline or partially online. If in case the application resource had issue & application resource had faulted, if the apps resource was not critical, rest of resource within the service groups will still be online. In this state, the service group would be in partially online state. If you are sure that application fault has been fixed, then you can trigger hagrp -online to start the application resources. VCS will auto detect that rest of resources in the service group are already online & only application resource needs to be restarted.

    To answer your original question, I would agree with Mike on approach of freezing the service group, troubleshooting the application, unfreezing then.

     

    G

  • Gaurav_singh's avatar
    Gaurav_singh
    Level 4
    11 years ago

    @Mike.. I small doubt I have..Clould you please clear to me.

    a) 1st case: If i take this case, What configurations required to failover the application to the second node.. If                       application hangs down in a node.

    b) 2nd case: What restart limit need to put so that vcs restarts the apps?

    Kinldy assist..

     

    Thanks,

    Gary

     

     

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    If VCS detects the application issue, then if resource has Critical atribute of 1 (or any dependent resource is critical), then it will cause group to failover, else, it will not.

    If VCS doe not detect the application issue, then if only need to kill application once to fix it, then you could use a RestartLimit of 1, but if you need to restart applications more than once, then you need to set RestartLimit accordingly.  But I would freeze service group (see first post) and then kill application, rather than using RestartLimit as this is what freezing a group is really designed for.

    Mike