cancel
Showing results for 
Search instead for 
Did you mean: 

VCS - Query regarding Cluster Resource behaviour if i do not do hastart on any of the node in the cluster

a_kjaiswal
Not applicable

Hi,

I'm using a 7 node cluster.

My query is that if i do hastart on first 6 nodes and in 1 nodes I do not do hastart, will my services runnning on first 6 nodes will come online ?

I'm seeing my paralllel group coming online but node others. Please Mediator1 and other groups present on first 2 nodes didnt came up. I have done hastart on node 0 and 1.

MM-BL0:/etc/VRTSvcs/conf/config# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  MM-BL0               RUNNING              0
A  MM-BL1               RUNNING              0
A  MM-BL2               UNKNOWN              0
A  MM-BL3               UNKNOWN              0
A  MM-BL4               UNKNOWN              0
A  MM-BL5               UNKNOWN              0
A  MM-BL6               UNKNOWN              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  FMMgrp          MM-BL0               Y          N               OFFLINE
B  FMMgrp          MM-BL1               Y          N               OFFLINE
B  FMMgrp          MM-BL2               Y          Y               OFFLINE
B  FMMgrp          MM-BL3               Y          Y               OFFLINE
B  FMMgrp          MM-BL4               Y          Y               OFFLINE
B  FMMgrp          MM-BL5               Y          Y               OFFLINE
B  FMMgrp          MM-BL6               Y          Y               OFFLINE
B  Mediator1       MM-BL0               Y          N               OFFLINE
B  Mediator1       MM-BL1               Y          N               OFFLINE
B  Mediator1       MM-BL2               Y          Y               OFFLINE
B  Mediator1       MM-BL3               Y          Y               OFFLINE
B  Mediator1       MM-BL4               Y          Y               OFFLINE
B  Mediator1       MM-BL5               Y          Y               OFFLINE
B  Mediator1       MM-BL6               Y          Y               OFFLINE
B  Mediator2       MM-BL0               Y          N               OFFLINE
B  Mediator2       MM-BL1               Y          N               OFFLINE
B  Mediator2       MM-BL2               Y          Y               OFFLINE
B  Mediator2       MM-BL3               Y          Y               OFFLINE
B  Mediator2       MM-BL4               Y          Y               OFFLINE
B  Mediator2       MM-BL5               Y          Y               OFFLINE
B  Mediator2       MM-BL6               Y          Y               OFFLINE
B  Mediator3       MM-BL0               Y          N               OFFLINE
B  Mediator3       MM-BL1               Y          N               OFFLINE
B  Mediator3       MM-BL2               Y          Y               OFFLINE
B  Mediator3       MM-BL3               Y          Y               OFFLINE
B  Mediator3       MM-BL4               Y          Y               OFFLINE
B  Mediator3       MM-BL5               Y          Y               OFFLINE
B  Mediator3       MM-BL6               Y          Y               OFFLINE
B  Mediator4       MM-BL0               Y          N               OFFLINE
B  Mediator4       MM-BL1               Y          N               OFFLINE
B  Mediator4       MM-BL2               Y          Y               OFFLINE
B  Mediator4       MM-BL3               Y          Y               OFFLINE
B  Mediator4       MM-BL4               Y          Y               OFFLINE
B  Mediator4       MM-BL5               Y          Y               OFFLINE
B  Mediator4       MM-BL6               Y          Y               OFFLINE
B  Mediator5       MM-BL0               Y          N               OFFLINE
B  Mediator5       MM-BL1               Y          N               OFFLINE
B  Mediator5       MM-BL2               Y          Y               OFFLINE
B  Mediator5       MM-BL3               Y          Y               OFFLINE
B  Mediator5       MM-BL4               Y          Y               OFFLINE
B  Mediator5       MM-BL5               Y          Y               OFFLINE
B  Mediator5       MM-BL6               Y          Y               OFFLINE
B  Mediator6       MM-BL0               Y          N               OFFLINE
B  Mediator6       MM-BL1               Y          N               OFFLINE
B  Mediator6       MM-BL2               Y          Y               OFFLINE
B  Mediator6       MM-BL3               Y          Y               OFFLINE
B  Mediator6       MM-BL4               Y          Y               OFFLINE
B  Mediator6       MM-BL5               Y          Y               OFFLINE
B  Mediator6       MM-BL6               Y          Y               OFFLINE
B  Network         MM-BL0               Y          N               ONLINE
B  Network         MM-BL1               Y          N               ONLINE
B  Network         MM-BL2               Y          Y               OFFLINE
B  Network         MM-BL3               Y          Y               OFFLINE
B  Network         MM-BL4               Y          Y               OFFLINE
B  Network         MM-BL5               Y          Y               OFFLINE
B  Network         MM-BL6               Y          Y               OFFLINE
B  Oracle1         MM-BL0               Y          N               OFFLINE
B  Oracle1         MM-BL1               Y          N               OFFLINE
B  Oracle1         MM-BL2               Y          Y               OFFLINE
B  Oracle1         MM-BL3               Y          Y               OFFLINE
B  Oracle1         MM-BL4               Y          Y               OFFLINE
B  Oracle1         MM-BL5               Y          Y               OFFLINE
B  Oracle1         MM-BL6               Y          Y               OFFLINE
B  SENTINELgrp     MM-BL0               Y          N               OFFLINE
B  SENTINELgrp     MM-BL1               Y          N               OFFLINE
B  SENTINELgrp     MM-BL2               Y          Y               OFFLINE
B  SENTINELgrp     MM-BL3               Y          Y               OFFLINE
B  SENTINELgrp     MM-BL4               Y          Y               OFFLINE
B  SENTINELgrp     MM-BL5               Y          Y               OFFLINE
B  SENTINELgrp     MM-BL6               Y          Y               OFFLINE
B  SNMPMasterAgent MM-BL0               Y          N               ONLINE
B  SNMPMasterAgent MM-BL1               Y          N               ONLINE
B  SNMPMasterAgent MM-BL2               Y          Y               OFFLINE
B  SNMPMasterAgent MM-BL3               Y          Y               OFFLINE
B  SNMPMasterAgent MM-BL4               Y          Y               OFFLINE
B  SNMPMasterAgent MM-BL5               Y          Y               OFFLINE
B  SNMPMasterAgent MM-BL6               Y          Y               OFFLINE
B  ServerGroup1_DG MM-BL0               Y          N               ONLINE
B  ServerGroup1_DG MM-BL1               Y          N               ONLINE
B  ServerGroup1_DG MM-BL2               Y          Y               OFFLINE
B  ServerGroup1_DG MM-BL3               Y          Y               OFFLINE
B  ServerGroup1_DG MM-BL4               Y          Y               OFFLINE
B  ServerGroup1_DG MM-BL5               Y          Y               OFFLINE
B  ServerGroup1_DG MM-BL6               Y          Y               OFFLINE
B  ServerGroup2_DG MM-BL0               Y          N               ONLINE
B  ServerGroup2_DG MM-BL1               Y          N               ONLINE
B  ServerGroup2_DG MM-BL2               Y          Y               OFFLINE
B  ServerGroup2_DG MM-BL3               Y          Y               OFFLINE
B  ServerGroup2_DG MM-BL4               Y          Y               OFFLINE
B  ServerGroup2_DG MM-BL5               Y          Y               OFFLINE
B  ServerGroup2_DG MM-BL6               Y          Y               OFFLINE
B  ServerGroup3_DG MM-BL0               Y          N               ONLINE
B  ServerGroup3_DG MM-BL1               Y          N               ONLINE
B  ServerGroup3_DG MM-BL2               Y          Y               OFFLINE
B  ServerGroup3_DG MM-BL3               Y          Y               OFFLINE
B  ServerGroup3_DG MM-BL4               Y          Y               OFFLINE
B  ServerGroup3_DG MM-BL5               Y          Y               OFFLINE
B  ServerGroup3_DG MM-BL6               Y          Y               OFFLINE
B  ServerGroup4_DG MM-BL0               Y          N               ONLINE
B  ServerGroup4_DG MM-BL1               Y          N               ONLINE
B  ServerGroup4_DG MM-BL2               Y          Y               OFFLINE
B  ServerGroup4_DG MM-BL3               Y          Y               OFFLINE
B  ServerGroup4_DG MM-BL4               Y          Y               OFFLINE
B  ServerGroup4_DG MM-BL5               Y          Y               OFFLINE
B  ServerGroup4_DG MM-BL6               Y          Y               OFFLINE
B  ServerGroup5_DG MM-BL0               Y          N               ONLINE
B  ServerGroup5_DG MM-BL1               Y          N               ONLINE
B  ServerGroup5_DG MM-BL2               Y          Y               OFFLINE
B  ServerGroup5_DG MM-BL3               Y          Y               OFFLINE
B  ServerGroup5_DG MM-BL4               Y          Y               OFFLINE
B  ServerGroup5_DG MM-BL5               Y          Y               OFFLINE
B  ServerGroup5_DG MM-BL6               Y          Y               OFFLINE
B  ServerGroup6_DG MM-BL0               Y          N               ONLINE
B  ServerGroup6_DG MM-BL1               Y          N               ONLINE
B  ServerGroup6_DG MM-BL2               Y          Y               OFFLINE
B  ServerGroup6_DG MM-BL3               Y          Y               OFFLINE
B  ServerGroup6_DG MM-BL4               Y          Y               OFFLINE
B  ServerGroup6_DG MM-BL5               Y          Y               OFFLINE
B  ServerGroup6_DG MM-BL6               Y          Y               OFFLINE
B  cvm             MM-BL0               Y          N               ONLINE
B  cvm             MM-BL1               Y          N               ONLINE
B  cvm             MM-BL2               Y          Y               OFFLINE
B  cvm             MM-BL3               Y          Y               OFFLINE
B  cvm             MM-BL4               Y          Y               OFFLINE
B  cvm             MM-BL5               Y          Y               OFFLINE
B  cvm             MM-BL6               Y          Y               OFFLINE
B  lic_DG          MM-BL0               Y          N               ONLINE
B  lic_DG          MM-BL1               Y          N               ONLINE
B  lic_DG          MM-BL2               Y          Y               OFFLINE
B  lic_DG          MM-BL3               Y          Y               OFFLINE
B  lic_DG          MM-BL4               Y          Y               OFFLINE
B  lic_DG          MM-BL5               Y          Y               OFFLINE
B  lic_DG          MM-BL6               Y          Y               OFFLINE
B  ora_DG          MM-BL0               Y          N               ONLINE
B  ora_DG          MM-BL1               Y          N               ONLINE
B  ora_DG          MM-BL2               Y          Y               OFFLINE
B  ora_DG          MM-BL3               Y          Y               OFFLINE
B  ora_DG          MM-BL4               Y          Y               OFFLINE
B  ora_DG          MM-BL5               Y          Y               OFFLINE
B  ora_DG          MM-BL6               Y          Y               OFFLINE

-- RESOURCES NOT PROBED
-- Group           Type                 Resource             System

D  FMMgrp          IPMultiNIC           FMMgrp_IP            MM-BL2
D  FMMgrp          IPMultiNIC           FMMgrp_IP            MM-BL3
D  FMMgrp          IPMultiNIC           FMMgrp_IP            MM-BL4
D  FMMgrp          IPMultiNIC           FMMgrp_IP            MM-BL5
D  FMMgrp          IPMultiNIC           FMMgrp_IP            MM-BL6
D  FMMgrp          ORACLE               fmm                  MM-BL2
D  FMMgrp          ORACLE               fmm                  MM-BL3
D  FMMgrp          ORACLE               fmm                  MM-BL4
D  FMMgrp          ORACLE               fmm                  MM-BL5
D  FMMgrp          ORACLE               fmm                  MM-BL6
D  FMMgrp          Proxy                FMMgrp_NIC_PROXY     MM-BL2
D  FMMgrp          Proxy                FMMgrp_NIC_PROXY     MM-BL3
D  FMMgrp          Proxy                FMMgrp_NIC_PROXY     MM-BL4
D  FMMgrp          Proxy                FMMgrp_NIC_PROXY     MM-BL5
D  FMMgrp          Proxy                FMMgrp_NIC_PROXY     MM-BL6
D  FMMgrp          Tomcat               fmmweb               MM-BL2
D  FMMgrp          Tomcat               fmmweb               MM-BL3
D  FMMgrp          Tomcat               fmmweb               MM-BL4
D  FMMgrp          Tomcat               fmmweb               MM-BL5
D  FMMgrp          Tomcat               fmmweb               MM-BL6
D  Mediator1       IPMultiNIC           Mediator1_IP         MM-BL2
D  Mediator1       IPMultiNIC           Mediator1_IP         MM-BL3
D  Mediator1       IPMultiNIC           Mediator1_IP         MM-BL4
D  Mediator1       IPMultiNIC           Mediator1_IP         MM-BL5
D  Mediator1       IPMultiNIC           Mediator1_IP         MM-BL6
D  Mediator1       Mediator             Server1              MM-BL2
D  Mediator1       Mediator             Server1              MM-BL3
D  Mediator1       Mediator             Server1              MM-BL4
D  Mediator1       Mediator             Server1              MM-BL5
D  Mediator1       Mediator             Server1              MM-BL6
D  Mediator1       Proxy                Mediator1_NIC_PROXY  MM-BL2
D  Mediator1       Proxy                Mediator1_NIC_PROXY  MM-BL3
D  Mediator1       Proxy                Mediator1_NIC_PROXY  MM-BL4
D  Mediator1       Proxy                Mediator1_NIC_PROXY  MM-BL5
D  Mediator1       Proxy                Mediator1_NIC_PROXY  MM-BL6
D  Mediator2       IPMultiNIC           Mediator2_IP         MM-BL2
D  Mediator2       IPMultiNIC           Mediator2_IP         MM-BL3
D  Mediator2       IPMultiNIC           Mediator2_IP         MM-BL4
D  Mediator2       IPMultiNIC           Mediator2_IP         MM-BL5
D  Mediator2       IPMultiNIC           Mediator2_IP         MM-BL6
D  Mediator2       Mediator             Server2              MM-BL2
D  Mediator2       Mediator             Server2              MM-BL3
D  Mediator2       Mediator             Server2              MM-BL4
D  Mediator2       Mediator             Server2              MM-BL5
D  Mediator2       Mediator             Server2              MM-BL6
D  Mediator2       Proxy                Mediator2_NIC_PROXY  MM-BL2
D  Mediator2       Proxy                Mediator2_NIC_PROXY  MM-BL3
D  Mediator2       Proxy                Mediator2_NIC_PROXY  MM-BL4
D  Mediator2       Proxy                Mediator2_NIC_PROXY  MM-BL5
D  Mediator2       Proxy                Mediator2_NIC_PROXY  MM-BL6
D  Mediator3       IPMultiNIC           Mediator3_IP         MM-BL2
D  Mediator3       IPMultiNIC           Mediator3_IP         MM-BL3
D  Mediator3       IPMultiNIC           Mediator3_IP         MM-BL4
D  Mediator3       IPMultiNIC           Mediator3_IP         MM-BL5
D  Mediator3       IPMultiNIC           Mediator3_IP         MM-BL6
D  Mediator3       Mediator             Server3              MM-BL2
D  Mediator3       Mediator             Server3              MM-BL3
D  Mediator3       Mediator             Server3              MM-BL4
D  Mediator3       Mediator             Server3              MM-BL5
D  Mediator3       Mediator             Server3              MM-BL6
D  Mediator3       Proxy                Mediator3_NIC_PROXY  MM-BL2
D  Mediator3       Proxy                Mediator3_NIC_PROXY  MM-BL3
D  Mediator3       Proxy                Mediator3_NIC_PROXY  MM-BL4
D  Mediator3       Proxy                Mediator3_NIC_PROXY  MM-BL5
D  Mediator3       Proxy                Mediator3_NIC_PROXY  MM-BL6
D  Mediator4       IPMultiNIC           Mediator4_IP         MM-BL2
D  Mediator4       IPMultiNIC           Mediator4_IP         MM-BL3
D  Mediator4       IPMultiNIC           Mediator4_IP         MM-BL4
D  Mediator4       IPMultiNIC           Mediator4_IP         MM-BL5
D  Mediator4       IPMultiNIC           Mediator4_IP         MM-BL6
D  Mediator4       Mediator             Server4              MM-BL2
D  Mediator4       Mediator             Server4              MM-BL3
D  Mediator4       Mediator             Server4              MM-BL4
D  Mediator4       Mediator             Server4              MM-BL5
D  Mediator4       Mediator             Server4              MM-BL6
D  Mediator4       Proxy                Mediator4_NIC_PROXY  MM-BL2
D  Mediator4       Proxy                Mediator4_NIC_PROXY  MM-BL3
D  Mediator4       Proxy                Mediator4_NIC_PROXY  MM-BL4
D  Mediator4       Proxy                Mediator4_NIC_PROXY  MM-BL5
D  Mediator4       Proxy                Mediator4_NIC_PROXY  MM-BL6
D  Mediator5       IPMultiNIC           Mediator5_IP         MM-BL2
D  Mediator5       IPMultiNIC           Mediator5_IP         MM-BL3
D  Mediator5       IPMultiNIC           Mediator5_IP         MM-BL4
D  Mediator5       IPMultiNIC           Mediator5_IP         MM-BL5
D  Mediator5       IPMultiNIC           Mediator5_IP         MM-BL6
D  Mediator5       Mediator             Server5              MM-BL2
D  Mediator5       Mediator             Server5              MM-BL3
D  Mediator5       Mediator             Server5              MM-BL4
D  Mediator5       Mediator             Server5              MM-BL5
D  Mediator5       Mediator             Server5              MM-BL6
D  Mediator5       Proxy                Mediator5_NIC_PROXY  MM-BL2
D  Mediator5       Proxy                Mediator5_NIC_PROXY  MM-BL3
D  Mediator5       Proxy                Mediator5_NIC_PROXY  MM-BL4
D  Mediator5       Proxy                Mediator5_NIC_PROXY  MM-BL5
D  Mediator5       Proxy                Mediator5_NIC_PROXY  MM-BL6
D  Mediator6       IPMultiNIC           Mediator6_IP         MM-BL2
D  Mediator6       IPMultiNIC           Mediator6_IP         MM-BL3
D  Mediator6       IPMultiNIC           Mediator6_IP         MM-BL4
D  Mediator6       IPMultiNIC           Mediator6_IP         MM-BL5
D  Mediator6       IPMultiNIC           Mediator6_IP         MM-BL6
D  Mediator6       Mediator             Server6              MM-BL2
D  Mediator6       Mediator             Server6              MM-BL3
D  Mediator6       Mediator             Server6              MM-BL4
D  Mediator6       Mediator             Server6              MM-BL5
D  Mediator6       Mediator             Server6              MM-BL6
D  Mediator6       Proxy                Mediator6_NIC_PROXY  MM-BL2
D  Mediator6       Proxy                Mediator6_NIC_PROXY  MM-BL3
D  Mediator6       Proxy                Mediator6_NIC_PROXY  MM-BL4
D  Mediator6       Proxy                Mediator6_NIC_PROXY  MM-BL5
D  Mediator6       Proxy                Mediator6_NIC_PROXY  MM-BL6
D  Network         MultiNICA            MultiNICA            MM-BL2
D  Network         MultiNICA            MultiNICA            MM-BL3
D  Network         MultiNICA            MultiNICA            MM-BL4
D  Network         MultiNICA            MultiNICA            MM-BL5
D  Network         MultiNICA            MultiNICA            MM-BL6
D  Network         Phantom              Phantom              MM-BL2
D  Network         Phantom              Phantom              MM-BL3
D  Network         Phantom              Phantom              MM-BL4
D  Network         Phantom              Phantom              MM-BL5
D  Network         Phantom              Phantom              MM-BL6
D  Oracle1         IPMultiNIC           Oracle1_IP           MM-BL2
D  Oracle1         IPMultiNIC           Oracle1_IP           MM-BL3
D  Oracle1         IPMultiNIC           Oracle1_IP           MM-BL4
D  Oracle1         IPMultiNIC           Oracle1_IP           MM-BL5
D  Oracle1         IPMultiNIC           Oracle1_IP           MM-BL6
D  Oracle1         ORACLE               bgw                  MM-BL2
D  Oracle1         ORACLE               bgw                  MM-BL3
D  Oracle1         ORACLE               bgw                  MM-BL4
D  Oracle1         ORACLE               bgw                  MM-BL5
D  Oracle1         ORACLE               bgw                  MM-BL6
D  Oracle1         Proxy                Oracle1_NIC_PROXY    MM-BL2
D  Oracle1         Proxy                Oracle1_NIC_PROXY    MM-BL3
D  Oracle1         Proxy                Oracle1_NIC_PROXY    MM-BL4
D  Oracle1         Proxy                Oracle1_NIC_PROXY    MM-BL5
D  Oracle1         Proxy                Oracle1_NIC_PROXY    MM-BL6
D  SENTINELgrp     IPMultiNIC           SENTINELgrp_IP       MM-BL2
D  SENTINELgrp     IPMultiNIC           SENTINELgrp_IP       MM-BL3
D  SENTINELgrp     IPMultiNIC           SENTINELgrp_IP       MM-BL4
D  SENTINELgrp     IPMultiNIC           SENTINELgrp_IP       MM-BL5
D  SENTINELgrp     IPMultiNIC           SENTINELgrp_IP       MM-BL6
D  SENTINELgrp     Proxy                SENTINELgrp_NIC_PROXY MM-BL2
D  SENTINELgrp     Proxy                SENTINELgrp_NIC_PROXY MM-BL3
D  SENTINELgrp     Proxy                SENTINELgrp_NIC_PROXY MM-BL4
D  SENTINELgrp     Proxy                SENTINELgrp_NIC_PROXY MM-BL5
D  SENTINELgrp     Proxy                SENTINELgrp_NIC_PROXY MM-BL6
D  SENTINELgrp     Sentinel             licserv              MM-BL2
D  SENTINELgrp     Sentinel             licserv              MM-BL3
D  SENTINELgrp     Sentinel             licserv              MM-BL4
D  SENTINELgrp     Sentinel             licserv              MM-BL5
D  SENTINELgrp     Sentinel             licserv              MM-BL6
D  SNMPMasterAgent SNMPMasterAgent      SNMPMasterAgent      MM-BL2
D  SNMPMasterAgent SNMPMasterAgent      SNMPMasterAgent      MM-BL3
D  SNMPMasterAgent SNMPMasterAgent      SNMPMasterAgent      MM-BL4
D  SNMPMasterAgent SNMPMasterAgent      SNMPMasterAgent      MM-BL5
D  SNMPMasterAgent SNMPMasterAgent      SNMPMasterAgent      MM-BL6
D  ServerGroup1_DG CFSMount             cfsmount2            MM-BL2
D  ServerGroup1_DG CFSMount             cfsmount2            MM-BL3
D  ServerGroup1_DG CFSMount             cfsmount2            MM-BL4
D  ServerGroup1_DG CFSMount             cfsmount2            MM-BL5
D  ServerGroup1_DG CFSMount             cfsmount2            MM-BL6
D  ServerGroup1_DG CVMVolDg             cvmvoldg2            MM-BL2
D  ServerGroup1_DG CVMVolDg             cvmvoldg2            MM-BL3
D  ServerGroup1_DG CVMVolDg             cvmvoldg2            MM-BL4
D  ServerGroup1_DG CVMVolDg             cvmvoldg2            MM-BL5
D  ServerGroup1_DG CVMVolDg             cvmvoldg2            MM-BL6
D  ServerGroup2_DG CFSMount             cfsmount7            MM-BL2
D  ServerGroup2_DG CFSMount             cfsmount7            MM-BL3
D  ServerGroup2_DG CFSMount             cfsmount7            MM-BL4
D  ServerGroup2_DG CFSMount             cfsmount7            MM-BL5
D  ServerGroup2_DG CFSMount             cfsmount7            MM-BL6
D  ServerGroup2_DG CVMVolDg             cvmvoldg7            MM-BL2
D  ServerGroup2_DG CVMVolDg             cvmvoldg7            MM-BL3
D  ServerGroup2_DG CVMVolDg             cvmvoldg7            MM-BL4
D  ServerGroup2_DG CVMVolDg             cvmvoldg7            MM-BL5
D  ServerGroup2_DG CVMVolDg             cvmvoldg7            MM-BL6
D  ServerGroup3_DG CFSMount             cfsmount8            MM-BL2
D  ServerGroup3_DG CFSMount             cfsmount8            MM-BL3
D  ServerGroup3_DG CFSMount             cfsmount8            MM-BL4
D  ServerGroup3_DG CFSMount             cfsmount8            MM-BL5
D  ServerGroup3_DG CFSMount             cfsmount8            MM-BL6
D  ServerGroup3_DG CVMVolDg             cvmvoldg8            MM-BL2
D  ServerGroup3_DG CVMVolDg             cvmvoldg8            MM-BL3
D  ServerGroup3_DG CVMVolDg             cvmvoldg8            MM-BL4
D  ServerGroup3_DG CVMVolDg             cvmvoldg8            MM-BL5
D  ServerGroup3_DG CVMVolDg             cvmvoldg8            MM-BL6
D  ServerGroup4_DG CFSMount             cfsmount9            MM-BL2
D  ServerGroup4_DG CFSMount             cfsmount9            MM-BL3
D  ServerGroup4_DG CFSMount             cfsmount9            MM-BL4
D  ServerGroup4_DG CFSMount             cfsmount9            MM-BL5
D  ServerGroup4_DG CFSMount             cfsmount9            MM-BL6
D  ServerGroup4_DG CVMVolDg             cvmvoldg9            MM-BL2
D  ServerGroup4_DG CVMVolDg             cvmvoldg9            MM-BL3
D  ServerGroup4_DG CVMVolDg             cvmvoldg9            MM-BL4
D  ServerGroup4_DG CVMVolDg             cvmvoldg9            MM-BL5
D  ServerGroup4_DG CVMVolDg             cvmvoldg9            MM-BL6
D  ServerGroup5_DG CFSMount             cfsmount10           MM-BL2
D  ServerGroup5_DG CFSMount             cfsmount10           MM-BL3
D  ServerGroup5_DG CFSMount             cfsmount10           MM-BL4
D  ServerGroup5_DG CFSMount             cfsmount10           MM-BL5
D  ServerGroup5_DG CFSMount             cfsmount10           MM-BL6
D  ServerGroup5_DG CVMVolDg             cvmvoldg10           MM-BL2
D  ServerGroup5_DG CVMVolDg             cvmvoldg10           MM-BL3
D  ServerGroup5_DG CVMVolDg             cvmvoldg10           MM-BL4
D  ServerGroup5_DG CVMVolDg             cvmvoldg10           MM-BL5
D  ServerGroup5_DG CVMVolDg             cvmvoldg10           MM-BL6
D  ServerGroup6_DG CFSMount             cfsmount11           MM-BL2
D  ServerGroup6_DG CFSMount             cfsmount11           MM-BL3
D  ServerGroup6_DG CFSMount             cfsmount11           MM-BL4
D  ServerGroup6_DG CFSMount             cfsmount11           MM-BL5
D  ServerGroup6_DG CFSMount             cfsmount11           MM-BL6
D  ServerGroup6_DG CVMVolDg             cvmvoldg11           MM-BL2
D  ServerGroup6_DG CVMVolDg             cvmvoldg11           MM-BL3
D  ServerGroup6_DG CVMVolDg             cvmvoldg11           MM-BL4
D  ServerGroup6_DG CVMVolDg             cvmvoldg11           MM-BL5
D  ServerGroup6_DG CVMVolDg             cvmvoldg11           MM-BL6
D  cvm             CFSfsckd             vxfsckd              MM-BL2
D  cvm             CFSfsckd             vxfsckd              MM-BL3
D  cvm             CFSfsckd             vxfsckd              MM-BL4
D  cvm             CFSfsckd             vxfsckd              MM-BL5
D  cvm             CFSfsckd             vxfsckd              MM-BL6
D  cvm             CVMCluster           cvm_clus             MM-BL2
D  cvm             CVMCluster           cvm_clus             MM-BL3
D  cvm             CVMCluster           cvm_clus             MM-BL4
D  cvm             CVMCluster           cvm_clus             MM-BL5
D  cvm             CVMCluster           cvm_clus             MM-BL6
D  cvm             CVMVxconfigd         cvm_vxconfigd        MM-BL2
D  cvm             CVMVxconfigd         cvm_vxconfigd        MM-BL3
D  cvm             CVMVxconfigd         cvm_vxconfigd        MM-BL4
D  cvm             CVMVxconfigd         cvm_vxconfigd        MM-BL5
D  cvm             CVMVxconfigd         cvm_vxconfigd        MM-BL6
D  lic_DG          CFSMount             cfsmount5            MM-BL2
D  lic_DG          CFSMount             cfsmount5            MM-BL3
D  lic_DG          CFSMount             cfsmount5            MM-BL4
D  lic_DG          CFSMount             cfsmount5            MM-BL5
D  lic_DG          CFSMount             cfsmount5            MM-BL6
D  lic_DG          CFSMount             cfsmount6            MM-BL2
D  lic_DG          CFSMount             cfsmount6            MM-BL3
D  lic_DG          CFSMount             cfsmount6            MM-BL4
D  lic_DG          CFSMount             cfsmount6            MM-BL5
D  lic_DG          CFSMount             cfsmount6            MM-BL6
D  lic_DG          CVMVolDg             cvmvoldg5            MM-BL2
D  lic_DG          CVMVolDg             cvmvoldg5            MM-BL3
D  lic_DG          CVMVolDg             cvmvoldg5            MM-BL4
D  lic_DG          CVMVolDg             cvmvoldg5            MM-BL5
D  lic_DG          CVMVolDg             cvmvoldg5            MM-BL6
D  lic_DG          CVMVolDg             cvmvoldg6            MM-BL2
D  lic_DG          CVMVolDg             cvmvoldg6            MM-BL3
D  lic_DG          CVMVolDg             cvmvoldg6            MM-BL4
D  lic_DG          CVMVolDg             cvmvoldg6            MM-BL5
D  lic_DG          CVMVolDg             cvmvoldg6            MM-BL6
D  ora_DG          CFSMount             cfsmount4            MM-BL2
D  ora_DG          CFSMount             cfsmount4            MM-BL3
D  ora_DG          CFSMount             cfsmount4            MM-BL4
D  ora_DG          CFSMount             cfsmount4            MM-BL5
D  ora_DG          CFSMount             cfsmount4            MM-BL6
D  ora_DG          CVMVolDg             cvmvoldg4            MM-BL2
D  ora_DG          CVMVolDg             cvmvoldg4            MM-BL3
D  ora_DG          CVMVolDg             cvmvoldg4            MM-BL4
D  ora_DG          CVMVolDg             cvmvoldg4            MM-BL5
D  ora_DG          CVMVolDg             cvmvoldg4            MM-BL6

MM-BL1:/etc/VRTSvcs/conf/config# cat main.cf
include "types.cf"
include "CFSTypes.cf"
include "CVMTypes.cf"

cluster MediatorCluster (
        UserNames = { vcsguest = cD2a90jzh1hgg, vcsop = j5rBONYy1OtL6,
                 vcsadm = "sEFEKH1CaHW5.",
                 root = PHbwhE5x2iM9I }
        Administrators = { vcsadm, root }
        UseFence = SCSI3
        )

system MM-BL0 (
        )

system MM-BL1 (
        )

system MM-BL2 (
        )

system MM-BL3 (
        )

system MM-BL4 (
        )

system MM-BL5 (
        )

system MM-BL6 (
        )

group FMMgrp (
        SystemList = { MM-BL0 = 1, MM-BL1 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL0 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC FMMgrp_IP (
                Address = "172.21.201.111"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        ORACLE fmm (
                )

        Proxy FMMgrp_NIC_PROXY (
                TargetResName = MultiNICA
                )

        Tomcat fmmweb (
                )

        requires group SENTINELgrp online global soft
        FMMgrp_IP requires FMMgrp_NIC_PROXY
        fmm requires FMMgrp_IP
        fmmweb requires fmm


        // resource dependency tree
        //
        //      group FMMgrp
        //      {
        //      Tomcat fmmweb
        //          {
        //          ORACLE fmm
        //              {
        //              IPMultiNIC FMMgrp_IP
        //                  {
        //                  Proxy FMMgrp_NIC_PROXY
        //                  }
        //              }
        //          }
        //      }


group Mediator1 (
        SystemList = { MM-BL1 = 1, MM-BL0 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL1 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC Mediator1_IP (
                Address = "172.21.201.113"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Mediator Server1 (
                )

        Proxy Mediator1_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ServerGroup1_DG online local firm
        Mediator1_IP requires Mediator1_NIC_PROXY
        Server1 requires Mediator1_IP


        // resource dependency tree
        //
        //      group Mediator1
        //      {
        //      Mediator Server1
        //          {
        //          IPMultiNIC Mediator1_IP
        //              {
        //              Proxy Mediator1_NIC_PROXY
        //              }
        //          }
        //      }


group Mediator2 (
        SystemList = { MM-BL2 = 1, MM-BL0 = 2, MM-BL1 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL2 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC Mediator2_IP (
                Address = "172.21.201.114"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Mediator Server2 (
                )

        Proxy Mediator2_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ServerGroup2_DG online local firm
        Mediator2_IP requires Mediator2_NIC_PROXY
        Server2 requires Mediator2_IP


        // resource dependency tree
        //
        //      group Mediator2
        //      {
        //      Mediator Server2
        //          {
        //          IPMultiNIC Mediator2_IP
        //              {
        //              Proxy Mediator2_NIC_PROXY
        //              }
        //          }
        //      }


group Mediator3 (
        SystemList = { MM-BL3 = 1, MM-BL0 = 2, MM-BL2 = 3, MM-BL1 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL3 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC Mediator3_IP (
                Address = "172.21.201.115"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Mediator Server3 (
                )

        Proxy Mediator3_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ServerGroup3_DG online local firm
        Mediator3_IP requires Mediator3_NIC_PROXY
        Server3 requires Mediator3_IP


        // resource dependency tree
        //
        //      group Mediator3
        //      {
        //      Mediator Server3
        //          {
        //          IPMultiNIC Mediator3_IP
        //              {
        //              Proxy Mediator3_NIC_PROXY
        //              }
        //          }
        //      }


group Mediator4 (
        SystemList = { MM-BL4 = 1, MM-BL0 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL1 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL4 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC Mediator4_IP (
                Address = "172.21.201.116"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Mediator Server4 (
                )

        Proxy Mediator4_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ServerGroup4_DG online local firm
        Mediator4_IP requires Mediator4_NIC_PROXY
        Server4 requires Mediator4_IP


        // resource dependency tree
        //
        //      group Mediator4
        //      {
        //      Mediator Server4
        //          {
        //          IPMultiNIC Mediator4_IP
        //              {
        //              Proxy Mediator4_NIC_PROXY
        //              }
        //          }
        //      }


group Mediator5 (
        SystemList = { MM-BL5 = 1, MM-BL0 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL1 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL5 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC Mediator5_IP (
                Address = "172.21.201.117"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Mediator Server5 (
                )

        Proxy Mediator5_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ServerGroup5_DG online local firm
        Mediator5_IP requires Mediator5_NIC_PROXY
        Server5 requires Mediator5_IP


        // resource dependency tree
        //
        //      group Mediator5
        //      {
        //      Mediator Server5
        //          {
        //          IPMultiNIC Mediator5_IP
        //              {
        //              Proxy Mediator5_NIC_PROXY
        //              }
        //          }
        //      }


group Mediator6 (
        SystemList = { MM-BL6 = 1, MM-BL0 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL1 = 7 }
        AutoStartList = { MM-BL6 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC Mediator6_IP (
                Address = "172.21.201.118"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Mediator Server6 (
                )

        Proxy Mediator6_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ServerGroup6_DG online local firm
        Mediator6_IP requires Mediator6_NIC_PROXY
        Server6 requires Mediator6_IP


        // resource dependency tree
        //
        //      group Mediator6
        //      {
        //      Mediator Server6
        //          {
        //          IPMultiNIC Mediator6_IP
        //              {
        //              Proxy Mediator6_NIC_PROXY
        //              }
        //          }
        //      }


group Network (
        SystemList = { MM-BL0 = 1, MM-BL1 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        MultiNICA MultiNICA (
                Device @MM-BL0 = { e1000g0 = "172.21.201.127",
                         e1000g3 = "172.21.201.127" }
                Device @MM-BL1 = { e1000g0 = "172.21.201.129",
                         e1000g3 = "172.21.201.129" }
                Device @MM-BL2 = { e1000g0 = "172.21.201.131",
                         e1000g3 = "172.21.201.131" }
                Device @MM-BL3 = { e1000g0 = "172.21.201.133",
                         e1000g3 = "172.21.201.133" }
                Device @MM-BL4 = { e1000g0 = "172.21.201.135",
                         e1000g3 = "172.21.201.135" }
                Device @MM-BL5 = { e1000g0 = "172.21.201.137",
                         e1000g3 = "172.21.201.137" }
                Device @MM-BL6 = { e1000g0 = "172.21.201.139",
                         e1000g3 = "172.21.201.139" }
                NetMask = "255.255.255.0"
                ArpDelay = 5
                RouteOptions = "127.0.0.1"
                IfconfigTwice = 1
                NetworkHosts = { "127.0.0.1", "127.0.0.1" }
                )

        Phantom Phantom (
                )

 

        // resource dependency tree
        //
        //      group Network
        //      {
        //      MultiNICA MultiNICA
        //      Phantom Phantom
        //      }


group Oracle1 (
        SystemList = { MM-BL0 = 1, MM-BL1 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL0 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC Oracle1_IP (
                Address = "172.21.201.110"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        ORACLE bgw (
                )

        Proxy Oracle1_NIC_PROXY (
                TargetResName = MultiNICA
                )

        requires group ora_DG online local firm
        Oracle1_IP requires Oracle1_NIC_PROXY
        bgw requires Oracle1_IP


        // resource dependency tree
        //
        //      group Oracle1
        //      {
        //      ORACLE bgw
        //          {
        //          IPMultiNIC Oracle1_IP
        //              {
        //              Proxy Oracle1_NIC_PROXY
        //              }
        //          }
        //      }


group SENTINELgrp (
        SystemList = { MM-BL0 = 1, MM-BL1 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6,
   MM-BL2 = 7 }
        AutoStartList = { MM-BL0 }
        FailOverPolicy = RoundRobin
        )

        IPMultiNIC SENTINELgrp_IP (
                Address = "172.21.201.125"
                NetMask = "255.255.255.0"
                MultiNICResName = MultiNICA
                IfconfigTwice = 1
                )

        Proxy SENTINELgrp_NIC_PROXY (
                TargetResName = MultiNICA
                )

        Sentinel licserv (
                )

        requires group lic_DG online local firm
        SENTINELgrp_IP requires SENTINELgrp_NIC_PROXY
        SENTINELgrp_IP requires licserv


        // resource dependency tree
        //
        //      group SENTINELgrp
        //      {
        //      IPMultiNIC SENTINELgrp_IP
        //          {
        //          Proxy SENTINELgrp_NIC_PROXY
        //          Sentinel licserv
        //          }
        //      }


group SNMPMasterAgent (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        SNMPMasterAgent SNMPMasterAgent (
                )

 

        // resource dependency tree
        //
        //      group SNMPMasterAgent
        //      {
        //      SNMPMasterAgent SNMPMasterAgent
        //      }


group ServerGroup1_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount2 (
                Critical = 0
                MountPoint = "/var/opt/BGw/ServerGroup1"
                BlockDevice = "/dev/vx/dsk/bgw1dg/vol01"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg2 (
                Critical = 0
                CVMDiskGroup = bgw1dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount2 requires cvmvoldg2


        // resource dependency tree
        //
        //      group ServerGroup1_DG
        //      {
        //      CFSMount cfsmount2
        //          {
        //          CVMVolDg cvmvoldg2
        //          }
        //      }


group ServerGroup2_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount7 (
                Critical = 0
                MountPoint = "/var/opt/BGw/ServerGroup2"
                BlockDevice = "/dev/vx/dsk/bgw2dg/vol01"
                MountOpt @MM-BL0 = "cluster"
                MountOpt @MM-BL1 = "cluster"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg7 (
                Critical = 0
                CVMDiskGroup = bgw2dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount7 requires cvmvoldg7


        // resource dependency tree
        //
        //      group ServerGroup2_DG
        //      {
        //      CFSMount cfsmount7
        //          {
        //          CVMVolDg cvmvoldg7
        //          }
        //      }


group ServerGroup3_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount8 (
                Critical = 0
                MountPoint = "/var/opt/BGw/ServerGroup3"
                BlockDevice = "/dev/vx/dsk/bgw3dg/vol01"
                MountOpt @MM-BL0 = "cluster"
                MountOpt @MM-BL1 = "cluster"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg8 (
                Critical = 0
                CVMDiskGroup = bgw3dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount8 requires cvmvoldg8


        // resource dependency tree
        //
        //      group ServerGroup3_DG
        //      {
        //      CFSMount cfsmount8
        //          {
        //          CVMVolDg cvmvoldg8
        //          }
        //      }


group ServerGroup4_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount9 (
                Critical = 0
                MountPoint = "/var/opt/BGw/ServerGroup4"
                BlockDevice = "/dev/vx/dsk/bgw4dg/vol01"
                MountOpt @MM-BL0 = "cluster"
                MountOpt @MM-BL1 = "cluster"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg9 (
                Critical = 0
                CVMDiskGroup = bgw4dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount9 requires cvmvoldg9


        // resource dependency tree
        //
        //      group ServerGroup4_DG
        //      {
        //      CFSMount cfsmount9
        //          {
        //          CVMVolDg cvmvoldg9
        //          }
        //      }


group ServerGroup5_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount10 (
                Critical = 0
                MountPoint = "/var/opt/BGw/ServerGroup5"
                BlockDevice = "/dev/vx/dsk/bgw5dg/vol01"
                MountOpt @MM-BL0 = "cluster"
                MountOpt @MM-BL1 = "cluster"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg10 (
                Critical = 0
                CVMDiskGroup = bgw5dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount10 requires cvmvoldg10


        // resource dependency tree
        //
        //      group ServerGroup5_DG
        //      {
        //      CFSMount cfsmount10
        //          {
        //          CVMVolDg cvmvoldg10
        //          }
        //      }


group ServerGroup6_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount11 (
                Critical = 0
                MountPoint = "/var/opt/BGw/ServerGroup6"
                BlockDevice = "/dev/vx/dsk/bgw6dg/vol01"
                MountOpt @MM-BL0 = "cluster"
                MountOpt @MM-BL1 = "cluster"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg11 (
                Critical = 0
                CVMDiskGroup = bgw6dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount11 requires cvmvoldg11


        // resource dependency tree
        //
        //      group ServerGroup6_DG
        //      {
        //      CFSMount cfsmount11
        //          {
        //          CVMVolDg cvmvoldg11
        //          }
        //      }


group cvm (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSfsckd vxfsckd (
                ActivationMode @MM-BL0 = { bgw1dg = sw, ora1dg = sw, lic1dg = sw,
                         fmm1dg = sw,
                         bgw2dg = sw,
                         bgw3dg = sw,
                         bgw4dg = sw,
                         bgw5dg = sw,
                         bgw6dg = sw }
                ActivationMode @MM-BL1 = { bgw1dg = sw, ora1dg = sw, lic1dg = sw,
                         fmm1dg = sw,
                         bgw2dg = sw,
                         bgw3dg = sw,
                         bgw4dg = sw,
                         bgw5dg = sw,
                         bgw6dg = sw }
                ActivationMode @MM-BL2 = { bgw1dg = sw, lic1dg = sw, fmm1dg = sw,
                         ora1dg = sw,
                         bgw2dg = sw,
                         bgw3dg = sw,
                         bgw4dg = sw,
                         bgw5dg = sw,
                         bgw6dg = sw }
                ActivationMode @MM-BL3 = { bgw1dg = sw, lic1dg = sw, fmm1dg = sw,
                         ora1dg = sw,
                         bgw2dg = sw,
                         bgw3dg = sw,
                         bgw4dg = sw,
                         bgw5dg = sw,
                         bgw6dg = sw }
                ActivationMode @MM-BL4 = { bgw1dg = sw, lic1dg = sw, fmm1dg = sw,
                         ora1dg = sw,
                         bgw2dg = sw,
                         bgw3dg = sw,
                         bgw4dg = sw,
                         bgw5dg = sw,
                         bgw6dg = sw }
                ActivationMode @MM-BL5 = { bgw1dg = sw, lic1dg = sw, fmm1dg = sw,
                         ora1dg = sw,
                         bgw2dg = sw,
                         bgw3dg = sw,
                         bgw4dg = sw,
                         bgw5dg = sw,
                         bgw6dg = sw }
                ActivationMode @MM-BL6 = { bgw1dg = sw, lic1dg = sw, fmm1dg = sw,
                         ora1dg = sw,
                         bgw2dg = sw,
                         bgw3dg = sw,
                         bgw4dg = sw,
                         bgw5dg = sw,
                         bgw6dg = sw }
                )

        CVMCluster cvm_clus (
                CVMClustName = MediatorCluster
                CVMNodeId = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                         MM-BL4 = 4,
                         MM-BL5 = 5,
                         MM-BL6 = 6 }
                CVMTransport = gab
                CVMTimeout = 200
                )

        CVMVxconfigd cvm_vxconfigd (
                Critical = 0
                CVMVxconfigdArgs = { syslog }
                )

        cvm_clus requires cvm_vxconfigd
        vxfsckd requires cvm_clus


        // resource dependency tree
        //
        //      group cvm
        //      {
        //      CFSfsckd vxfsckd
        //          {
        //          CVMCluster cvm_clus
        //              {
        //              CVMVxconfigd cvm_vxconfigd
        //              }
        //          }
        //      }


group lic_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount5 (
                Critical = 0
                MountPoint = "/var/opt/sentinel"
                BlockDevice = "/dev/vx/dsk/lic1dg/vol01"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CFSMount cfsmount6 (
                Critical = 0
                MountPoint = "/var/opt/mediation/fmmdb"
                BlockDevice = "/dev/vx/dsk/fmm1dg/vol01"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg5 (
                Critical = 0
                CVMDiskGroup = lic1dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        CVMVolDg cvmvoldg6 (
                Critical = 0
                CVMDiskGroup = fmm1dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount5 requires cvmvoldg5
        cfsmount6 requires cvmvoldg6


        // resource dependency tree
        //
        //      group lic_DG
        //      {
        //      CFSMount cfsmount5
        //          {
        //          CVMVolDg cvmvoldg5
        //          }
        //      CFSMount cfsmount6
        //          {
        //          CVMVolDg cvmvoldg6
        //          }
        //      }


group ora_DG (
        SystemList = { MM-BL0 = 0, MM-BL1 = 1, MM-BL2 = 2, MM-BL3 = 3,
                 MM-BL4 = 4,
                 MM-BL5 = 5,
                 MM-BL6 = 6 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        )

        CFSMount cfsmount4 (
                Critical = 0
                MountPoint = "/var/opt/mediation/ora"
                BlockDevice = "/dev/vx/dsk/ora1dg/vol01"
                MountOpt @MM-BL2 = "cluster"
                MountOpt @MM-BL3 = "cluster"
                MountOpt @MM-BL4 = "cluster"
                MountOpt @MM-BL5 = "cluster"
                MountOpt @MM-BL6 = "cluster"
                NodeList = { MM-BL0, MM-BL1, MM-BL2, MM-BL3, MM-BL4, MM-BL5,
                         MM-BL6 }
                )

        CVMVolDg cvmvoldg4 (
                Critical = 0
                CVMDiskGroup = ora1dg
                CVMActivation @MM-BL0 = sw
                CVMActivation @MM-BL1 = sw
                CVMActivation @MM-BL2 = sw
                CVMActivation @MM-BL3 = sw
                CVMActivation @MM-BL4 = sw
                CVMActivation @MM-BL5 = sw
                CVMActivation @MM-BL6 = sw
                )

        requires group cvm online local firm
        cfsmount4 requires cvmvoldg4


        // resource dependency tree
        //
        //      group ora_DG
        //      {
        //      CFSMount cfsmount4
        //          {
        //          CVMVolDg cvmvoldg4
        //          }
        //      }

 

 

Engine a log

2011/06/09 11:54:45 VCS NOTICE V-16-1-11022 VCS engine (had) started
2011/06/09 11:54:45 VCS INFO V-16-1-10196 Cluster logger started
2011/06/09 11:54:45 VCS NOTICE V-16-1-11050 VCS engine version=5.0
2011/06/09 11:54:45 VCS NOTICE V-16-1-11051 VCS engine join version=5.0.30.4
2011/06/09 11:54:45 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.0MP3RP3-11/27/09-09:17:00
2011/06/09 11:54:45 VCS NOTICE V-16-1-10114 Opening GAB library
2011/06/09 11:54:45 VCS NOTICE V-16-1-10619 'HAD' starting on: MM-BL0
2011/06/09 11:54:45 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms
2011/06/09 11:54:50 VCS INFO V-16-1-10077 Received new cluster membership
2011/06/09 11:54:50 VCS NOTICE V-16-1-10112 System (MM-BL0) - Membership: 0x1, DDNA: 0x7e
2011/06/09 11:54:50 VCS NOTICE V-16-1-10086 System MM-BL0 (Node '0') is in Regular Membership - Membership: 0x1
2011/06/09 11:54:50 VCS NOTICE V-16-1-10322 System MM-BL0 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2011/06/09 11:54:52 VCS NOTICE V-16-1-10032 VxFEN driver available. Local node id=0
2011/06/09 11:54:52 VCS NOTICE V-16-1-10322 System MM-BL0 (Node '0') changed state from LOCAL_BUILD to RUNNING
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSMount/CFSMountAgent for resource type CFSMount successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSfsckd/CFSfsckdAgent for resource type CFSfsckd successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMCluster/CVMClusterAgent for resource type CVMCluster successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVolDg/CVMVolDgAgent for resource type CVMVolDg successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVxconfigd/CVMVxconfigdAgent for resource type CVMVxconfigd successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IPMultiNIC/IPMultiNICAgent for resource type IPMultiNIC successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mediator/MediatorAgent for resource type Mediator successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/MultiNICA/MultiNICAAgent for resource type MultiNICA successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/ORACLE/ORACLEAgent for resource type ORACLE successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Phantom/PhantomAgent for resource type Phantom successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Proxy/ProxyAgent for resource type Proxy successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/SNMPMasterAgent/SNMPMasterAgentAgent for resource type SNMPMasterAgent successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Sentinel/SentinelAgent for resource type Sentinel successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Tomcat/TomcatAgent for resource type Tomcat successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Jun 09 11:54:52 2011
2011/06/09 11:54:52 VCS WARNING V-16-1-11155 LLT heartbeat link status changed. Previous status = 0xffffffff; Current status = 0x3.
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount5 (Owner: unknown, Group: lic_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount11 (Owner: unknown, Group: ServerGroup6_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount7 (Owner: unknown, Group: ServerGroup2_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount10 (Owner: unknown, Group: ServerGroup5_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount6 (Owner: unknown, Group: lic_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount4 (Owner: unknown, Group: ora_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount9 (Owner: unknown, Group: ServerGroup4_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cfsmount8 (Owner: unknown, Group: ServerGroup3_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS ERROR V-16-10001-1005 (MM-BL0) CVMCluster:???:monitor:node - state: out of cluster
reason: user initiated stop
2011/06/09 11:54:53 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MM-BL0
2011/06/09 11:54:53 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MM-BL0 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource vxfsckd (Owner: unknown, Group: cvm) is offline on MM-BL0 (First probe)
2011/06/09 11:54:53 VCS INFO V-16-1-10304 Resource cvm_clus (Owner: unknown, Group: cvm) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10297 Resource cvm_vxconfigd (Owner: unknown, Group: cvm) is online on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group cvm has been probed on system MM-BL0
2011/06/09 11:54:54 VCS NOTICE V-16-1-10442 Initiating auto-start online of group cvm on system MM-BL0
2011/06/09 11:54:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvm_clus (Owner: unknown, Group: cvm) on System MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource SNMPMasterAgent (Owner: unknown, Group: SNMPMasterAgent) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group SNMPMasterAgent has been probed on system MM-BL0
2011/06/09 11:54:54 VCS NOTICE V-16-1-10442 Initiating auto-start online of group SNMPMasterAgent on system MM-BL0
2011/06/09 11:54:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource SNMPMasterAgent (Owner: unknown, Group: SNMPMasterAgent) on System MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10297 Resource Phantom (Owner: unknown, Group: Network) is online on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL0
2011/06/09 11:54:54 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL1
2011/06/09 11:54:54 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL2
2011/06/09 11:54:54 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL3
2011/06/09 11:54:54 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL4
2011/06/09 11:54:54 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL5
2011/06/09 11:54:54 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL6
2011/06/09 11:54:54 VCS NOTICE V-16-1-10447 Group Network is online on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource licserv (Owner: unknown, Group: SENTINELgrp) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource fmmweb (Owner: unknown, Group: FMMgrp) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource fmm (Owner: unknown, Group: FMMgrp) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource bgw (Owner: unknown, Group: Oracle1) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg11 (Owner: unknown, Group: ServerGroup6_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group ServerGroup6_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group ServerGroup1_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg8 (Owner: unknown, Group: ServerGroup3_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group ServerGroup3_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg7 (Owner: unknown, Group: ServerGroup2_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group ServerGroup2_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg10 (Owner: unknown, Group: ServerGroup5_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group ServerGroup5_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group lic_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group ora_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource cvmvoldg9 (Owner: unknown, Group: ServerGroup4_DG) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS NOTICE V-16-1-10438 Group ServerGroup4_DG has been probed on system MM-BL0
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource Server3 (Owner: unknown, Group: Mediator3) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource Server6 (Owner: unknown, Group: Mediator6) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource Server4 (Owner: unknown, Group: Mediator4) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource Server5 (Owner: unknown, Group: Mediator5) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource Server2 (Owner: unknown, Group: Mediator2) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS INFO V-16-1-10304 Resource Server1 (Owner: unknown, Group: Mediator1) is offline on MM-BL0 (First probe)
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:Mediator2_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:Mediator5_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:Mediator6_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:SENTINELgrp_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:Mediator3_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:FMMgrp_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:Mediator4_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:54 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:Oracle1_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:55 VCS WARNING V-16-10001-4504 (MM-BL0) IPMultiNIC:Mediator1_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:54:55 VCS NOTICE V-16-1-10438 Group Network has been probed on system MM-BL0
2011/06/09 11:54:57 VCS INFO V-16-1-10298 Resource SNMPMasterAgent (Owner: unknown, Group: SNMPMasterAgent) is online on MM-BL0 (VCS initiated)
2011/06/09 11:54:57 VCS NOTICE V-16-1-10447 Group SNMPMasterAgent is online on system MM-BL0
2011/06/09 11:55:05 VCS INFO V-16-1-10077 Received new cluster membership
2011/06/09 11:55:05 VCS NOTICE V-16-1-10112 System (MM-BL0) - Membership: 0x3, DDNA: 0x7c
2011/06/09 11:55:05 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2011/06/09 11:55:05 VCS NOTICE V-16-1-10086 System MM-BL1 (Node '1') is in Regular Membership - Membership: 0x3
2011/06/09 11:55:05 VCS NOTICE V-16-1-10453 Node: 1 changed name from: 'MM-BL1' to: 'MM-BL1'
2011/06/09 11:55:05 VCS NOTICE V-16-1-10322 System MM-BL1 (Node '1') changed state from UNKNOWN to INITING
2011/06/09 11:55:05 VCS NOTICE V-16-1-10322 System MM-BL1 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT
2011/06/09 11:55:05 VCS NOTICE V-16-1-10322 System MM-BL1 (Node '1') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2011/06/09 11:55:05 VCS INFO V-16-1-10463 Sending snapshot to node: 1
2011/06/09 11:55:09 VCS NOTICE V-16-1-10322 System MM-BL1 (Node '1') changed state from REMOTE_BUILD to RUNNING
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount10 (Owner: unknown, Group: ServerGroup5_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount11 (Owner: unknown, Group: ServerGroup6_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount7 (Owner: unknown, Group: ServerGroup2_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount4 (Owner: unknown, Group: ora_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount6 (Owner: unknown, Group: lic_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount8 (Owner: unknown, Group: ServerGroup3_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount5 (Owner: unknown, Group: lic_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cfsmount9 (Owner: unknown, Group: ServerGroup4_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS ERROR V-16-10001-1005 (MM-BL1) CVMCluster:???:monitor:node - state: out of cluster
reason: user initiated stop
2011/06/09 11:55:10 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system MM-BL1
2011/06/09 11:55:10 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource vxfsckd (Owner: unknown, Group: cvm) is offline on MM-BL1 (First probe)
2011/06/09 11:55:10 VCS INFO V-16-1-10304 Resource cvm_clus (Owner: unknown, Group: cvm) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group ServerGroup1_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group ora_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg11 (Owner: unknown, Group: ServerGroup6_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group ServerGroup6_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group lic_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg9 (Owner: unknown, Group: ServerGroup4_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group ServerGroup4_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg7 (Owner: unknown, Group: ServerGroup2_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group ServerGroup2_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg10 (Owner: unknown, Group: ServerGroup5_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group ServerGroup5_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource cvmvoldg8 (Owner: unknown, Group: ServerGroup3_DG) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group ServerGroup3_DG has been probed on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10297 Resource cvm_vxconfigd (Owner: unknown, Group: cvm) is online on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group cvm has been probed on system MM-BL1
2011/06/09 11:55:11 VCS NOTICE V-16-1-10442 Initiating auto-start online of group cvm on system MM-BL1
2011/06/09 11:55:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvm_clus (Owner: unknown, Group: cvm) on System MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10297 Resource Phantom (Owner: unknown, Group: Network) is online on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL0
2011/06/09 11:55:11 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL1
2011/06/09 11:55:11 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL2
2011/06/09 11:55:11 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL3
2011/06/09 11:55:11 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL4
2011/06/09 11:55:11 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL5
2011/06/09 11:55:11 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group Network on node MM-BL6
2011/06/09 11:55:11 VCS NOTICE V-16-1-10447 Group Network is online on system MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource SNMPMasterAgent (Owner: unknown, Group: SNMPMasterAgent) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS NOTICE V-16-1-10438 Group SNMPMasterAgent has been probed on system MM-BL1
2011/06/09 11:55:11 VCS NOTICE V-16-1-10442 Initiating auto-start online of group SNMPMasterAgent on system MM-BL1
2011/06/09 11:55:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource SNMPMasterAgent (Owner: unknown, Group: SNMPMasterAgent) on System MM-BL1
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource fmmweb (Owner: unknown, Group: FMMgrp) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource licserv (Owner: unknown, Group: SENTINELgrp) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource fmm (Owner: unknown, Group: FMMgrp) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource bgw (Owner: unknown, Group: Oracle1) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource Server1 (Owner: unknown, Group: Mediator1) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource Server5 (Owner: unknown, Group: Mediator5) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource Server2 (Owner: unknown, Group: Mediator2) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource Server3 (Owner: unknown, Group: Mediator3) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource Server6 (Owner: unknown, Group: Mediator6) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS INFO V-16-1-10304 Resource Server4 (Owner: unknown, Group: Mediator4) is offline on MM-BL1 (First probe)
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:Mediator6_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:FMMgrp_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:SENTINELgrp_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:Oracle1_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:Mediator4_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:Mediator5_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:Mediator3_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:Mediator2_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:11 VCS WARNING V-16-10001-4504 (MM-BL1) IPMultiNIC:Mediator1_IP:monitor:MultiNICA MultiNICA not probed. Will go online after probe succeeds
2011/06/09 11:55:12 VCS NOTICE V-16-1-10438 Group Network has been probed on system MM-BL1
2011/06/09 11:55:14 VCS INFO V-16-10001-1003 (MM-BL0) CVMCluster:cvm_clus:online:CVMCluster role is - mode: enabled: cluster active - MASTER
master: MM-BL0
2011/06/09 11:55:14 VCS INFO V-16-1-10298 Resource SNMPMasterAgent (Owner: unknown, Group: SNMPMasterAgent) is online on MM-BL1 (VCS initiated)
2011/06/09 11:55:14 VCS NOTICE V-16-1-10447 Group SNMPMasterAgent is online on system MM-BL1
2011/06/09 11:55:16 VCS INFO V-16-1-10298 Resource cvm_clus (Owner: unknown, Group: cvm) is online on MM-BL0 (VCS initiated)
2011/06/09 11:55:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource vxfsckd (Owner: unknown, Group: cvm) on System MM-BL0
2011/06/09 11:55:18 VCS INFO V-16-1-10298 Resource vxfsckd (Owner: unknown, Group: cvm) is online on MM-BL0 (VCS initiated)
2011/06/09 11:55:18 VCS NOTICE V-16-1-10447 Group cvm is online on system MM-BL0
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group ora_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group ora_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group lic_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group lic_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup6_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group ServerGroup6_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup5_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group ServerGroup5_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup4_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group ServerGroup4_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup3_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group ServerGroup3_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup2_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group ServerGroup2_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup1_DG on node MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10227 One or more child group not okay with bringing group ServerGroup1_DG online on node MM-BL1. Ignoring Restart
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg11 (Owner: unknown, Group: ServerGroup6_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg10 (Owner: unknown, Group: ServerGroup5_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg9 (Owner: unknown, Group: ServerGroup4_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg8 (Owner: unknown, Group: ServerGroup3_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg7 (Owner: unknown, Group: ServerGroup2_DG) on System MM-BL0
2011/06/09 11:55:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) on System MM-BL0
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource Mediator2_IP (Owner: unknown, Group: Mediator2) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource Mediator1_IP (Owner: unknown, Group: Mediator1) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource Mediator4_IP (Owner: unknown, Group: Mediator4) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource Oracle1_IP (Owner: unknown, Group: Oracle1) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource Mediator6_IP (Owner: unknown, Group: Mediator6) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource Mediator3_IP (Owner: unknown, Group: Mediator3) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource FMMgrp_IP (Owner: unknown, Group: FMMgrp) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource SENTINELgrp_IP (Owner: unknown, Group: SENTINELgrp) is offline on MM-BL0 (First probe)
2011/06/09 11:55:25 VCS INFO V-16-1-10304 Resource Mediator5_IP (Owner: unknown, Group: Mediator5) is offline on MM-BL0 (First probe)
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg4:online:resource cvmvoldg4 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg5:online:resource cvmvoldg5 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg11:online:resource cvmvoldg11 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg6:online:resource cvmvoldg6 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg9:online:resource cvmvoldg9 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg10:online:resource cvmvoldg10 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg8:online:resource cvmvoldg8 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg2:online:resource cvmvoldg2 is online
2011/06/09 11:55:29 VCS INFO V-16-20007-1046 (MM-BL0) CVMVolDg:cvmvoldg7:online:resource cvmvoldg7 is online
2011/06/09 11:55:31 VCS INFO V-16-10001-1003 (MM-BL1) CVMCluster:cvm_clus:online:CVMCluster role is - mode: enabled: cluster active - SLAVE
master: MM-BL0
2011/06/09 11:55:34 VCS INFO V-16-1-10298 Resource cvm_clus (Owner: unknown, Group: cvm) is online on MM-BL1 (VCS initiated)
2011/06/09 11:55:34 VCS NOTICE V-16-1-10301 Initiating Online of Resource vxfsckd (Owner: unknown, Group: cvm) on System MM-BL1
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group Mediator6 has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10435 Group Mediator6 will not start automatically on System MM-BL0 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group SENTINELgrp has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10439 Group (SENTINELgrp) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system MM-BL0 after probe
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group Mediator2 has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10435 Group Mediator2 will not start automatically on System MM-BL0 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group Oracle1 has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10439 Group (Oracle1) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system MM-BL0 after probe
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group Mediator5 has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10435 Group Mediator5 will not start automatically on System MM-BL0 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group Mediator1 has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10435 Group Mediator1 will not start automatically on System MM-BL0 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group FMMgrp has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10439 Group (FMMgrp) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system MM-BL0 after probe
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group Mediator4 has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10435 Group Mediator4 will not start automatically on System MM-BL0 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:34 VCS NOTICE V-16-1-10438 Group Mediator3 has been probed on system MM-BL0
2011/06/09 11:55:34 VCS NOTICE V-16-1-10435 Group Mediator3 will not start automatically on System MM-BL0 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:36 VCS INFO V-16-1-10298 Resource vxfsckd (Owner: unknown, Group: cvm) is online on MM-BL1 (VCS initiated)
2011/06/09 11:55:36 VCS NOTICE V-16-1-10447 Group cvm is online on system MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group ora_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group ora_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group lic_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group lic_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group ServerGroup6_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup6_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group ServerGroup5_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup5_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group ServerGroup4_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup4_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group ServerGroup3_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup3_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group ServerGroup2_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup2_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-51033 Group ServerGroup1_DG is already active on system MM-BL0. Ignoring Restart
2011/06/09 11:55:36 VCS WARNING V-16-1-50046 Initiating online of parent group ServerGroup1_DG on node MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg11 (Owner: unknown, Group: ServerGroup6_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg10 (Owner: unknown, Group: ServerGroup5_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg9 (Owner: unknown, Group: ServerGroup4_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg8 (Owner: unknown, Group: ServerGroup3_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg7 (Owner: unknown, Group: ServerGroup2_DG) on System MM-BL1
2011/06/09 11:55:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) on System MM-BL1
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg4:online:resource cvmvoldg4 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg5:online:resource cvmvoldg5 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg6:online:resource cvmvoldg6 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg11:online:resource cvmvoldg11 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg10:online:resource cvmvoldg10 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg8:online:resource cvmvoldg8 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg9:online:resource cvmvoldg9 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg2:online:resource cvmvoldg2 is online
2011/06/09 11:55:38 VCS INFO V-16-20007-1046 (MM-BL1) CVMVolDg:cvmvoldg7:online:resource cvmvoldg7 is online
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource Oracle1_IP (Owner: unknown, Group: Oracle1) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource Mediator4_IP (Owner: unknown, Group: Mediator4) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource FMMgrp_IP (Owner: unknown, Group: FMMgrp) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource SENTINELgrp_IP (Owner: unknown, Group: SENTINELgrp) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource Mediator1_IP (Owner: unknown, Group: Mediator1) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource Mediator2_IP (Owner: unknown, Group: Mediator2) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource Mediator5_IP (Owner: unknown, Group: Mediator5) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource Mediator3_IP (Owner: unknown, Group: Mediator3) is offline on MM-BL1 (First probe)
2011/06/09 11:55:42 VCS INFO V-16-1-10304 Resource Mediator6_IP (Owner: unknown, Group: Mediator6) is offline on MM-BL1 (First probe)
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group Mediator1 has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10439 Group (Mediator1) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system MM-BL1 after probe
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group Oracle1 has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group Oracle1 will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group Mediator6 has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group Mediator6 will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group Mediator2 has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group Mediator2 will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group Mediator5 has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group Mediator5 will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group Mediator4 has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group Mediator4 will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group Mediator3 has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group Mediator3 will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group FMMgrp has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group FMMgrp will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:55:51 VCS NOTICE V-16-1-10438 Group SENTINELgrp has been probed on system MM-BL1
2011/06/09 11:55:51 VCS NOTICE V-16-1-10435 Group SENTINELgrp will not start automatically on System MM-BL1 as the system is not a part of AutoStartList attribute of the group.
2011/06/09 11:56:01 VCS INFO V-16-1-10298 Resource cvmvoldg7 (Owner: unknown, Group: ServerGroup2_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount7 (Owner: unknown, Group: ServerGroup2_DG) on System MM-BL0
2011/06/09 11:56:01 VCS INFO V-16-1-10298 Resource cvmvoldg9 (Owner: unknown, Group: ServerGroup4_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount9 (Owner: unknown, Group: ServerGroup4_DG) on System MM-BL0
2011/06/09 11:56:01 VCS INFO V-16-1-10298 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) on System MM-BL0
2011/06/09 11:56:01 VCS INFO V-16-1-10298 Resource cvmvoldg8 (Owner: unknown, Group: ServerGroup3_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount8 (Owner: unknown, Group: ServerGroup3_DG) on System MM-BL0
2011/06/09 11:56:01 VCS INFO V-16-1-10298 Resource cvmvoldg11 (Owner: unknown, Group: ServerGroup6_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount11 (Owner: unknown, Group: ServerGroup6_DG) on System MM-BL0
2011/06/09 11:56:01 VCS INFO V-16-1-10298 Resource cvmvoldg10 (Owner: unknown, Group: ServerGroup5_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:01 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount10 (Owner: unknown, Group: ServerGroup5_DG) on System MM-BL0
2011/06/09 11:56:02 VCS INFO V-16-1-10298 Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount6 (Owner: unknown, Group: lic_DG) on System MM-BL0
2011/06/09 11:56:02 VCS INFO V-16-1-10298 Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount4 (Owner: unknown, Group: ora_DG) on System MM-BL0
2011/06/09 11:56:02 VCS INFO V-16-1-10298 Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount5 (Owner: unknown, Group: lic_DG) on System MM-BL0
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount9  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount7  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount2  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount9:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster             Block Device /dev/vx/dsk/bgw4dg/vol01 : MountPoint /var/opt/BGw/ServerGroup4
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount7:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster             Block Device /dev/vx/dsk/bgw2dg/vol01 : MountPoint /var/opt/BGw/ServerGroup2
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount8  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount2:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/bgw1dg/vol01 : MountPoint /var/opt/BGw/ServerGroup1
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount11  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount8:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster             Block Device /dev/vx/dsk/bgw3dg/vol01 : MountPoint /var/opt/BGw/ServerGroup3
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount11:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster            Block Device /dev/vx/dsk/bgw6dg/vol01 : MountPoint /var/opt/BGw/ServerGroup6
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount10  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount10:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster            Block Device /dev/vx/dsk/bgw5dg/vol01 : MountPoint /var/opt/BGw/ServerGroup5
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount4  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount6  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount4:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/ora1dg/vol01 : MountPoint /var/opt/mediation/ora
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount6:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/fmm1dg/vol01 : MountPoint /var/opt/mediation/fmmdb
2011/06/09 11:56:02 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount5  Primary  MM-BL0  from localhost
2011/06/09 11:56:02 VCS INFO V-16-20011-5506 (MM-BL0) CFSMount:cfsmount5:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/lic1dg/vol01 : MountPoint /var/opt/sentinel
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount7  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount9  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount2  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount11  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount8  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount10  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount6  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount4  Primary  MM-BL0  from localhost
2011/06/09 11:56:03 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount5  Primary  MM-BL0  from localhost
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount5 (Owner: unknown, Group: lic_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount4 (Owner: unknown, Group: ora_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group ora_DG is online on system MM-BL0
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount10 (Owner: unknown, Group: ServerGroup5_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group ServerGroup5_DG is online on system MM-BL0
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount6 (Owner: unknown, Group: lic_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group lic_DG is online on system MM-BL0
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount8 (Owner: unknown, Group: ServerGroup3_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group ServerGroup3_DG is online on system MM-BL0
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount11 (Owner: unknown, Group: ServerGroup6_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group ServerGroup6_DG is online on system MM-BL0
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount9 (Owner: unknown, Group: ServerGroup4_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group ServerGroup4_DG is online on system MM-BL0
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount7 (Owner: unknown, Group: ServerGroup2_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group ServerGroup2_DG is online on system MM-BL0
2011/06/09 11:56:04 VCS INFO V-16-1-10298 Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) is online on MM-BL0 (VCS initiated)
2011/06/09 11:56:04 VCS NOTICE V-16-1-10447 Group ServerGroup1_DG is online on system MM-BL0
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg8 (Owner: unknown, Group: ServerGroup3_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount8 (Owner: unknown, Group: ServerGroup3_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg2 (Owner: unknown, Group: ServerGroup1_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg9 (Owner: unknown, Group: ServerGroup4_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount9 (Owner: unknown, Group: ServerGroup4_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg11 (Owner: unknown, Group: ServerGroup6_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount11 (Owner: unknown, Group: ServerGroup6_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg10 (Owner: unknown, Group: ServerGroup5_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount10 (Owner: unknown, Group: ServerGroup5_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg7 (Owner: unknown, Group: ServerGroup2_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount7 (Owner: unknown, Group: ServerGroup2_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg6 (Owner: unknown, Group: lic_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount6 (Owner: unknown, Group: lic_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg4 (Owner: unknown, Group: ora_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount4 (Owner: unknown, Group: ora_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-1-10298 Resource cvmvoldg5 (Owner: unknown, Group: lic_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount5 (Owner: unknown, Group: lic_DG) on System MM-BL1
2011/06/09 11:56:11 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount5:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/lic1dg/vol01 : MountPoint /var/opt/sentinel
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount8:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster             Block Device /dev/vx/dsk/bgw3dg/vol01 : MountPoint /var/opt/BGw/ServerGroup3
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount4:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/ora1dg/vol01 : MountPoint /var/opt/mediation/ora
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount9:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster             Block Device /dev/vx/dsk/bgw4dg/vol01 : MountPoint /var/opt/BGw/ServerGroup4
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount6:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/fmm1dg/vol01 : MountPoint /var/opt/mediation/fmmdb
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount2:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,            Block Device /dev/vx/dsk/bgw1dg/vol01 : MountPoint /var/opt/BGw/ServerGroup1
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount7:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster             Block Device /dev/vx/dsk/bgw2dg/vol01 : MountPoint /var/opt/BGw/ServerGroup2
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount10:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster            Block Device /dev/vx/dsk/bgw5dg/vol01 : MountPoint /var/opt/BGw/ServerGroup5
2011/06/09 11:56:12 VCS INFO V-16-20011-5506 (MM-BL1) CFSMount:cfsmount11:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster            Block Device /dev/vx/dsk/bgw6dg/vol01 : MountPoint /var/opt/BGw/ServerGroup6
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount8 (Owner: unknown, Group: ServerGroup3_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group ServerGroup3_DG is online on system MM-BL1
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount9 (Owner: unknown, Group: ServerGroup4_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group ServerGroup4_DG is online on system MM-BL1
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount10 (Owner: unknown, Group: ServerGroup5_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group ServerGroup5_DG is online on system MM-BL1
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount2 (Owner: unknown, Group: ServerGroup1_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group ServerGroup1_DG is online on system MM-BL1
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount11 (Owner: unknown, Group: ServerGroup6_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group ServerGroup6_DG is online on system MM-BL1
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount4 (Owner: unknown, Group: ora_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group ora_DG is online on system MM-BL1
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount6 (Owner: unknown, Group: lic_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount5 (Owner: unknown, Group: lic_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group lic_DG is online on system MM-BL1
2011/06/09 11:56:12 VCS INFO V-16-1-10298 Resource cfsmount7 (Owner: unknown, Group: ServerGroup2_DG) is online on MM-BL1 (VCS initiated)
2011/06/09 11:56:12 VCS NOTICE V-16-1-10447 Group ServerGroup2_DG is online on system MM-BL1
MM-BL0:/#
 

5 REPLIES 5

Wally_Heim
Level 6
Employee

Hi,

 

From you maiin.cf, Mediator1 will not attempt to be brought online in your situation if node MM-BL1 fails to startup when all nodes in the cluster are restarted at the same time.  This is controlled by the AutoStartList only having node MM-BL1 in it.

 

The AutoStartList is a list of servers that when they enter a running state will cause the service group to come online in the cluster if the group is not running at that time.  The service group is brought online on the nodes based on SystemList priority and other workload manament settings.

 

The simple solution to your situation is to included multiple or all nodes in your AutoStartList for all service groups.  This will make the cluster check if the service group is running when the additional nodes enter a running state and not be dependant on a single node to initiate the auto start of the service group.

 

 

Current configuration of the Mediator1 service group:

group Mediator1 (
        SystemList = { MM-BL1 = 1, MM-BL0 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL1 }
        FailOverPolicy = RoundRobin
        )

 

 

Suggested change to the Mediator1 service group:

group Mediator1 (
        SystemList = { MM-BL1 = 1, MM-BL0 = 2, MM-BL2 = 3, MM-BL3 = 4,
                 MM-BL4 = 5,
                 MM-BL5 = 6,
                 MM-BL6 = 7 }
        AutoStartList = { MM-BL1, MM-BL0, MM-BL2, MM-BL3, MM-BL4, MM-BL5, MM-BL6 }
        FailOverPolicy = RoundRobin
        )

 

Thanks,

Wally

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

When your cluster will startup depends on the amount of system listed in /etc/gabtab in the gaconfig -n "value" section. This indicates the minimum amount of systems that must be online and registered before the cluster is seeded.

ktandel
Level 4
Partner Accredited Certified

Hi ,

 As per Riaan you need to change /etc/gabtab for eg if you have 3 node cluster then default value is gabconfig -c -n3 that means at the time of cluster startup gab always check wheter 3 nodes are alive. so change gabtab value accordingly

mikebounds
Level 6
Partner Accredited

As other people have said, gabtab will control failover groups starting from a cold boot.  You could set gabtab to seed with 4 nodes, which is the recommended minimum for a 7 node cluster, however with a CFS cluster I would leave as 7.  You should not be doing a cold boot very often and when you do it most likely is user initiated and so it is not a problem to manually seed cluster using "gabconfig -cx" if one node does not boot.  There should not be a situation when you loose all 7 nodes together unintentionally as you should have dual independent power supplies.

With regards putting all nodes in the AutoStartList as Wally suggests, this will not work very well.  The reason for this is that your failover groups depend on parallel groups.  If you put all nodes in the AutoStartList, what happens here is that all failover group can be probed on all nodes, but their child parallel groups are not online yet on any nodes.  When the parallel group onlines on the first system (generally the node that finished booting first), all the failover groups online on that system, because they are probed on all systems and they cannot go online on other systems as child parallel group is not online on these systems.  Therefore your service groups are not evenly distributed across the systems.

The problem with one system in the AutoStartList, is that if you use "gabconfig -cx" when one system doesn't come up, then any service groups with that system in the AutoStartList, will not AutoStart.  A way round this is to put 2 systems in each AutoStart list, but this will mean you could get a slightly uneven distribution in a cold boot.

Mike

sam321
Level 4

gabtab contains the no of nodes required to join the cluster in order to start the cluster.You can start it using just one node also by using manual seeding