VCS ERROR V-16-10001-0

[kelvin_liang]

I met a error on VCS 5.0:   2010/05/28 16:22:39 VCS ERROR V-16-10001-0 (BErA) Tomcat:fmmweb:online:Could

After checking in the symantec, I can not find any other useful information about this error code "VCS ERROR V-16-10001-0".

Any one know it and and help? Thank you....

[Marianne]

Please share the rest of the error message:

Tomcat:fmmweb:online:Could ..............??

Please also share extent of problem/symptoms?

[vcs_man]

Hi Kelvin,

Which Type of Agent you are using for  Online the Resource?  Is the Tomcat Agent given by Symantec or you used Agent Builder to have the Agent Type.
Also, what is the VCS version and OS platform you are using?

When Is the issue occured when you online the Service Group or Resource itself?

Please provide us the exact error message along with following..

#/etc/VRTSvcs/conf/config/main.cf 
#pkginfo -l VRTSacclib   (assuming this as Solaris OS).


Thanks,
Mandar

[kelvin_liang]

Hello Mandar,

I think that the Type of Agent was given by Symantec, not by agent builder.
And the VCS Version was 5.0 and Solaris 10

And this problem was triggered by service group failover. When FMMgrp was faulty on B, VCS will trigger failover this resource group to A. And the error "V-16-10001-0" occured in Engine_A.log of A machine.

And do you know the meaning of error code "V-16-10001-0"

And the main.cf like the following:

# cat main.cf
include "types.cf"
include "CFSTypes.cf"
include "CVMTypes.cf"

cluster MediationCluster (
        UserNames = { vcsguest = cD2a90jzh1hgg, vcsop = j5rBONYy1OtL6,
                 vcsadm = "sEFEKH1CaHW5.",
                 root = hKKpKFkELdKQjFKrKI }
        Administrators = { vcsadm, root }
        )

system JIMCG303BErA (
        )

system JIMCG303BErB (
        )

group FMMgrp (
        SystemList = { JIMCG303BErA = 2, JIMCG303BErB = 1 }
        AutoStartList = { JIMCG303BErB, JIMCG303BErA }
        )

        IPMultiNIC FMMgrp_IP (
                Address = "192.168.101.216"
                NetMask = "255.255.255.192"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        ORACLE fmm (
                )

        Proxy FMMgrp_NIC_PROXY (
                TargetResName = MultiNICA
                )

        Tomcat fmmweb (
                )

        requires group SENTINELgrp online global soft
        FMMgrp_IP requires FMMgrp_NIC_PROXY
        fmm requires FMMgrp_IP
        fmmweb requires fmm


        // resource dependency tree
        //
        //      group FMMgrp
        //      {
        //      Tomcat fmmweb
        //          {
        //          ORACLE fmm
        //              {
        //              IPMultiNIC FMMgrp_IP
        //                  {
        //                  Proxy FMMgrp_NIC_PROXY
        //                  }
        //              }
        //          }
        //      }


group Mediator1 (
        SystemList = { JIMCG303BErA = 1, JIMCG303BErB = 2 }
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        IPMultiNIC Mediator1_BOSS (
                Address = "10.246.105.3"
                NetMask = "255.255.255.224"
                MultiNICResName = MultiNICA_BOSS
                IfconfigTwice = 1
                )

        IPMultiNIC Mediator1_IP (
                Address = "192.168.101.214"
                NetMask = "255.255.255.192"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        IPMultiNIC Mediator1_OM (
                Address = "10.202.19.134"
                NetMask = "255.255.255.224"
                MultiNICResName = MultiNICA_OM
                IfconfigTwice = 1
                )

        Mediator Server1 (
                )

        Proxy BOSS_NIC_PROXY (
                TargetResName = MultiNICA_BOSS
                )

        Proxy Mediator1_NIC_PROXY (
                TargetResName = MultiNICA
                )

        Proxy OM_NIC_PROXY (
                TargetResName = MultiNICA_OM
                )

        requires group ServerGroup1_DG online local firm
        Mediator1_BOSS requires BOSS_NIC_PROXY
        Mediator1_IP requires Mediator1_NIC_PROXY
        Mediator1_OM requires OM_NIC_PROXY
        Server1 requires Mediator1_IP


        // resource dependency tree
        //
        //      group Mediator1
        //      {
        //      IPMultiNIC Mediator1_BOSS
        //          {
        //          Proxy BOSS_NIC_PROXY
        //          }
        //      IPMultiNIC Mediator1_OM
        //          {
        //          Proxy OM_NIC_PROXY
        //          }
        //      Mediator Server1
        //          {
        //          IPMultiNIC Mediator1_IP
        //              {
        //              Proxy Mediator1_NIC_PROXY
        //              }
        //          }
        //      }


group Network (
        SystemList = { JIMCG303BErA = 1, JIMCG303BErB = 2 }
        Parallel = 1
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        MultiNICA MultiNICA (
                Device @JIMCG303BErA = { e1000g0 = "192.168.101.210",
                         nxge3 = "192.168.101.210" }
                Device @JIMCG303BErB = { e1000g0 = "192.168.101.211",
                         nxge3 = "192.168.101.211" }
                NetMask = "255.255.255.192"
                RouteOptions = "192.168.101.193"
                IfconfigTwice = 1
                NetworkHosts = { "192.168.101.193", "192.168.101.212" }
                )

        Phantom Phantom (
                )



        // resource dependency tree
        //
        //      group Network
        //      {
        //      MultiNICA MultiNICA
        //      Phantom Phantom
        //      }


group Network_BOSS (
        SystemList = { JIMCG303BErA = 1, JIMCG303BErB = 2 }
        Parallel = 1
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        MultiNICA MultiNICA_BOSS (
                Device @JIMCG303BErA = { e1000g1 = "10.246.105.9",
                         nxge0 = "10.246.105.9" }
                Device @JIMCG303BErB = { e1000g1 = "10.246.105.10",
                         nxge0 = "10.246.105.10" }
                NetMask = "255.255.255.224"
                RouteOptions = "10.246.105.14"
                IfconfigTwice = 1
                NetworkHosts = { "10.246.105.14", "10.246.105.7" }
                )

        Phantom Phantom_BOSS (
                )



        // resource dependency tree
        //
        //      group Network_BOSS
        //      {
        //      MultiNICA MultiNICA_BOSS
        //      Phantom Phantom_BOSS
        //      }


group Network_OM (
        SystemList = { JIMCG303BErA = 1, JIMCG303BErB = 2 }
        Parallel = 1
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        MultiNICA MultiNICA_OM (
                Device @JIMCG303BErA = { e1000g2 = "10.202.19.140",
                         nxge2 = "10.202.19.140" }
                Device @JIMCG303BErB = { e1000g2 = "10.202.19.141",
                         nxge2 = "10.202.19.141" }
                NetMask = "255.255.255.224"
                RouteOptions = "10.202.19.129"
                IfconfigTwice = 1
                NetworkHosts = { "10.202.19.141", "10.202.19.138" }
                )

        Phantom Phantom_OM (
                )



        // resource dependency tree
        //
        //      group Network_OM
        //      {
        //      MultiNICA MultiNICA_OM
        //      Phantom Phantom_OM
        //      }


group Oracle1 (
        SystemList = { JIMCG303BErA = 1, JIMCG303BErB = 2 }
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        IPMultiNIC Oracle1_IP (
                Address = "192.168.101.215"
                NetMask = "255.255.255.192"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        ORACLE bgw (
                )

        Proxy Oracle1_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ora_DG online local firm
        Oracle1_IP requires Oracle1_NIC_PROXY
        bgw requires Oracle1_IP


        // resource dependency tree
        //
        //      group Oracle1
        //      {
        //      ORACLE bgw
        //          {
        //          IPMultiNIC Oracle1_IP
        //              {
        //              Proxy Oracle1_NIC_PROXY
        //              }
        //          }
        //      }


group SENTINELgrp (
        SystemList = { JIMCG303BErA = 2, JIMCG303BErB = 1 }
        AutoStartList = { JIMCG303BErB, JIMCG303BErA }
        )

        IPMultiNIC SENTINELgrp_IP (
                Address = "192.168.101.217"
                NetMask = "255.255.255.192"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Proxy SENTINELgrp_NIC_PROXY (
                TargetResName = MultiNICA
                )

        Sentinel licserv (
                )

        requires group lic_DG online local firm
        SENTINELgrp_IP requires SENTINELgrp_NIC_PROXY
        SENTINELgrp_IP requires licserv


        // resource dependency tree
        //
        //      group SENTINELgrp
        //      {
        //      IPMultiNIC SENTINELgrp_IP
        //          {
        //          Proxy SENTINELgrp_NIC_PROXY
        //          Sentinel licserv
        //          }
        //      }


group ServerGroup1_DG (
        SystemList = { JIMCG303BErA = 0, JIMCG303BErB = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        CFSMount cfsmount1 (
                Critical = 0
                MountPoint = "/var/opt/BGw/ServerGroup1"
                BlockDevice = "/dev/vx/dsk/bgw1dg/vol01"
                MountOpt @JIMCG303BErA = "cluster"
                MountOpt @JIMCG303BErB = "cluster"
                NodeList = { JIMCG303BErA, JIMCG303BErB }
                )

        CVMVolDg cvmvoldg1 (
                Critical = 0
                CVMDiskGroup = bgw1dg
                CVMActivation @JIMCG303BErA = sw
                CVMActivation @JIMCG303BErB = sw
                )

        requires group cvm online local firm
        cfsmount1 requires cvmvoldg1


        // resource dependency tree
        //
        //      group ServerGroup1_DG
        //      {
        //      CFSMount cfsmount1
        //          {
        //          CVMVolDg cvmvoldg1
        //          }
        //      }


group cvm (
        SystemList = { JIMCG303BErA = 0, JIMCG303BErB = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        CFSfsckd vxfsckd (
                ActivationMode @JIMCG303BErA = { bgw1dg = sw, ora1dg = sw, lic1dg = sw,
                         fmm1dg = sw }
                ActivationMode @JIMCG303BErB = { bgw1dg = sw, ora1dg = sw, lic1dg = sw,
                         fmm1dg = sw }
                )

        CVMCluster cvm_clus (
                CVMClustName = MediationCluster
                CVMNodeId = { JIMCG303BErA = 0, JIMCG303BErB = 1 }
                CVMTransport = gab
                CVMTimeout = 200
                )

        CVMVxconfigd cvm_vxconfigd (
                Critical = 0
                CVMVxconfigdArgs = { syslog }
                )

        cvm_clus requires cvm_vxconfigd
        vxfsckd requires cvm_clus


        // resource dependency tree
        //
        //      group cvm
        //      {
        //      CFSfsckd vxfsckd
        //          {
        //          CVMCluster cvm_clus
        //              {
        //              CVMVxconfigd cvm_vxconfigd
        //              }
        //          }
        //      }


group lic_DG (
        SystemList = { JIMCG303BErA = 0, JIMCG303BErB = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        CFSMount cfsmount3 (
                Critical = 0
                MountPoint = "/var/opt/sentinel"
                BlockDevice = "/dev/vx/dsk/lic1dg/vol01"
                MountOpt @JIMCG303BErA = "cluster"
                MountOpt @JIMCG303BErB = "cluster"
                NodeList = { JIMCG303BErA, JIMCG303BErB }
                )

        CFSMount cfsmount4 (
                Critical = 0
                MountPoint = "/var/opt/mediation/fmmdb"
                BlockDevice = "/dev/vx/dsk/fmm1dg/vol01"
                MountOpt @JIMCG303BErA = "cluster"
                MountOpt @JIMCG303BErB = "cluster"
                NodeList = { JIMCG303BErA, JIMCG303BErB }
                )

        CVMVolDg cvmvoldg3 (
                Critical = 0
                CVMDiskGroup = lic1dg
                CVMActivation @JIMCG303BErA = sw
                CVMActivation @JIMCG303BErB = sw
                )

        CVMVolDg cvmvoldg4 (
                Critical = 0
                CVMDiskGroup = fmm1dg
                CVMActivation @JIMCG303BErA = sw
                CVMActivation @JIMCG303BErB = sw
                )

        requires group cvm online local firm
        cfsmount3 requires cvmvoldg3
        cfsmount4 requires cvmvoldg4


        // resource dependency tree
        //
        //      group lic_DG
        //      {
        //      CFSMount cfsmount3
        //          {
        //          CVMVolDg cvmvoldg3
        //          }
        //      CFSMount cfsmount4
        //          {
        //          CVMVolDg cvmvoldg4
        //          }
        //      }


group ora_DG (
        SystemList = { JIMCG303BErA = 0, JIMCG303BErB = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { JIMCG303BErA, JIMCG303BErB }
        )

        CFSMount cfsmount2 (
                Critical = 0
                MountPoint = "/var/opt/mediation/ora"
                BlockDevice = "/dev/vx/dsk/ora1dg/vol01"
                MountOpt @JIMCG303BErA = "cluster"
                MountOpt @JIMCG303BErB = "cluster"
                NodeList = { JIMCG303BErA, JIMCG303BErB }
                )

        CVMVolDg cvmvoldg2 (
                Critical = 0
                CVMDiskGroup = ora1dg
                CVMActivation @JIMCG303BErA = sw
                CVMActivation @JIMCG303BErB = sw
                )

        requires group cvm online local firm
        cfsmount2 requires cvmvoldg2


        // resource dependency tree
        //
        //      group ora_DG
        //      {
        //      CFSMount cfsmount2
        //          {
        //          CVMVolDg cvmvoldg2
        //          }
        //      }

[kelvin_liang]

Hi Marianne,

The error message "Tomcat:fmmweb:online:Could" in Engine_A.log is the all. I have shared all.........

[Gaurav_S]

can u paste what are contents of ...

a)  /var/VRTSvcs/log/tomcat_A.log    (log might be big, paste the contents during the time error was seen)...

Another thing I am wondering that there is no separate types.cf file imported for Tomcat resource.... is it build in types.cf only ...
Can you check  /etc/VRTSvcs/conf/config/types.cf  & let know the resource attributes for Tomcat ?

Gaurav

[Gaurav_S]

Also, can u tell, was this cluster working before or is a newly built cluster ? Have you seen this error for the first time ? also paste # ls -l /etc/VRTSvcs/conf/config/*.cf Gaurav

[Anoop_Kumar1]

I believe the resource is below one

        Tomcat fmmweb (
                )


Can you paste below command output ?

# hares -display fmmweb

[kelvin_liang · Accepted solution]

SOrry all, I leave for a long time.

The problem was caused by tomcat crashed unexpectly. After restart Tomcat, all the service was online.