Forum Discussion

Home_224's avatar
Home_224
Level 6
7 years ago

Failed to start up the cluster service

2018/08/09 15:48:46 VCS INFO V-16-1-10305 Resource JavaMethodServer (Owner: unknown, Group: Documentum) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:46 VCS NOTICE V-16-1-10446 Group Documentum is offline on system devuaedms12
2018/08/09 15:48:46 VCS INFO V-16-6-15004 (devuaedms12) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2018/08/09 15:48:46 VCS INFO V-16-1-10305 Resource diskgroup_cfs4 (Owner: unknown, Group: Documentum_ENV_B) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:46 VCS INFO V-16-1-10305 Resource diskgroup_cfs3 (Owner: unknown, Group: Documentum_ENV_B) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:46 VCS NOTICE V-16-1-10446 Group Documentum_ENV_B is offline on system devuaedms12
2018/08/09 15:48:46 VCS NOTICE V-16-1-10300 Initiating Offline of Resource vxfsckd (Owner: unknown, Group: cvm) on System devuaedms12
2018/08/09 15:48:46 VCS INFO V-16-6-15004 (devuaedms12) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2018/08/09 15:48:48 VCS INFO V-16-1-10305 Resource vxfsckd (Owner: unknown, Group: cvm) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:48 VCS NOTICE V-16-1-10300 Initiating Offline of Resource qlogckd (Owner: unknown, Group: cvm) on System devuaedms12
2018/08/09 15:48:49 VCS INFO V-16-2-13001 (devuaedms12) Resource(qlogckd): Output of the completed operation (offline)
UX:vxfs qlogprint: INFO: V-3-22897: There are no QuickLog devices active
2018/08/09 15:48:50 VCS INFO V-16-1-10305 Resource qlogckd (Owner: unknown, Group: cvm) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvm_clus (Owner: unknown, Group: cvm) on System devuaedms12
2018/08/09 15:48:52 VCS ERROR V-16-10001-1005 (devuaedms12) CVMCluster:???:monitor:node - state: out of cluster
reason: user initiated stop
2018/08/09 15:48:53 VCS INFO V-16-1-10305 Resource cvm_clus (Owner: unknown, Group: cvm) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:53 VCS NOTICE V-16-1-10446 Group cvm is offline on system devuaedms12
2018/08/09 15:48:53 VCS NOTICE V-16-1-10010 Stopping all agents
2018/08/09 15:48:53 VCS NOTICE V-16-1-10322 System devuaedms12 (Node '1') changed state from LEAVING to EXITING
2018/08/09 15:48:53 VCS NOTICE V-16-1-10322 System devuaedms12 (Node '1') changed state from EXITING to EXITED

3 Replies

Sort By
  • Home_224's avatar
    Home_224
    Level 6
    7 years ago

    The server run on Solaris 9 and VCS 4.1 version, it is unable to start up the service group online 

     

    root@devuaedms12 # hastatus -sum

    -- SYSTEM STATE
    -- System State Frozen

    A devuaedms11 RUNNING 0
    A devuaedms12 RUNNING 0

    -- GROUP STATE
    -- Group System Probed AutoDisabled State

    B Documentum devuaedms11 Y N PARTIAL|FAULTED
    B Documentum devuaedms12 Y N PARTIAL
    B Documentum_ENV_B devuaedms11 Y N PARTIAL
    B Documentum_ENV_B devuaedms12 Y N STARTING|PARTIAL
    B MNICB_SITgroup devuaedms11 Y N ONLINE
    B MNICB_SITgroup devuaedms12 Y N ONLINE
    B cvm devuaedms11 Y N ONLINE
    B cvm devuaedms12 Y N ONLINE
    B documentum_bkup devuaedms11 Y N ONLINE
    B documentum_bkup devuaedms12 Y N OFFLINE

    -- RESOURCES FAILED
    -- Group Type Resource System

    C Documentum ContentServer ContentServer devuaedms11
    C Documentum_ENV_B ContentServer ContentServer_B devuaedms11
    C Documentum_ENV_B Proxy DocBroker_B devuaedms11
    C Documentum_ENV_B Proxy DocBroker_B devuaedms12

     

    Please advice if there is any advice so I can fix it 

    • Marianne's avatar
      Marianne
      Level 6
      7 years ago

      You are only showing us a piece of the log for "user initiated stop". 

      If something else happened prior to this, you need to check higher up in engine_A log.

      I see various errors in engine_A since 13 July, e.g. :

      2018/07/13 21:08:27 VCS ERROR V-16-2-13027 (devuaedms11) Resource(ContentServer_B) - monitor procedure did not complete within the expected time.

      2018/07/17 07:52:08 VCS INFO V-16-1-10307 Resource SITgroup (Owner: unknown, Group: MNICB_SITgroup) is offline on devuaedms12 (Not initiated by VCS)

      2018/07/17 07:52:09 VCS ERROR V-16-10001-5005 (devuaedms11) IPMultiNICB:mIP_bkup:online:Error in configuring IP address

      2018/07/17 07:52:11 VCS ERROR V-16-1-10205 Group MNICB_SITgroup is faulted on system devuaedms11

      2018/07/17 07:54:57 VCS ERROR V-16-2-13027 (devuaedms11) Resource(ContentServer) - monitor procedure did not complete within the expected time.
      2018/07/17 08:00:31 VCS ERROR V-16-2-13210 (devuaedms12) Agent is calling clean for resource(ContentServer) because 4 successive invocations of the monitor procedure did not complete within the expected time.

      2018/07/29 01:30:30 VCS ERROR V-16-2-13066 (devuaedms12) Agent is calling clean for resource(ContentServer) because the resource is not up even after online completed.


      Many more errors related to network and resources such as ContentServer.

      I feel that you need to perform a complete health check of the entire environment as EVERYTHING seems to be very old and out of support.

      • Gaurav_S's avatar
        Gaurav_S
        Moderator
        7 years ago

        Fully agree with Marianne, get us full logs, also, share the main.cf, what are the dependencies between the groups.