Highlighted

Failed to start up the cluster service

2018/08/09 15:48:46 VCS INFO V-16-1-10305 Resource JavaMethodServer (Owner: unknown, Group: Documentum) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:46 VCS NOTICE V-16-1-10446 Group Documentum is offline on system devuaedms12
2018/08/09 15:48:46 VCS INFO V-16-6-15004 (devuaedms12) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2018/08/09 15:48:46 VCS INFO V-16-1-10305 Resource diskgroup_cfs4 (Owner: unknown, Group: Documentum_ENV_B) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:46 VCS INFO V-16-1-10305 Resource diskgroup_cfs3 (Owner: unknown, Group: Documentum_ENV_B) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:46 VCS NOTICE V-16-1-10446 Group Documentum_ENV_B is offline on system devuaedms12
2018/08/09 15:48:46 VCS NOTICE V-16-1-10300 Initiating Offline of Resource vxfsckd (Owner: unknown, Group: cvm) on System devuaedms12
2018/08/09 15:48:46 VCS INFO V-16-6-15004 (devuaedms12) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2018/08/09 15:48:48 VCS INFO V-16-1-10305 Resource vxfsckd (Owner: unknown, Group: cvm) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:48 VCS NOTICE V-16-1-10300 Initiating Offline of Resource qlogckd (Owner: unknown, Group: cvm) on System devuaedms12
2018/08/09 15:48:49 VCS INFO V-16-2-13001 (devuaedms12) Resource(qlogckd): Output of the completed operation (offline)
UX:vxfs qlogprint: INFO: V-3-22897: There are no QuickLog devices active
2018/08/09 15:48:50 VCS INFO V-16-1-10305 Resource qlogckd (Owner: unknown, Group: cvm) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:50 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvm_clus (Owner: unknown, Group: cvm) on System devuaedms12
2018/08/09 15:48:52 VCS ERROR V-16-10001-1005 (devuaedms12) CVMCluster:???:monitor:node - state: out of cluster
reason: user initiated stop
2018/08/09 15:48:53 VCS INFO V-16-1-10305 Resource cvm_clus (Owner: unknown, Group: cvm) is offline on devuaedms12 (VCS initiated)
2018/08/09 15:48:53 VCS NOTICE V-16-1-10446 Group cvm is offline on system devuaedms12
2018/08/09 15:48:53 VCS NOTICE V-16-1-10010 Stopping all agents
2018/08/09 15:48:53 VCS NOTICE V-16-1-10322 System devuaedms12 (Node '1') changed state from LEAVING to EXITING
2018/08/09 15:48:53 VCS NOTICE V-16-1-10322 System devuaedms12 (Node '1') changed state from EXITING to EXITED

3 Replies

Re: Failed to start up the cluster service

The server run on Solaris 9 and VCS 4.1 version, it is unable to start up the service group online 

 

root@devuaedms12 # hastatus -sum

-- SYSTEM STATE
-- System State Frozen

A devuaedms11 RUNNING 0
A devuaedms12 RUNNING 0

-- GROUP STATE
-- Group System Probed AutoDisabled State

B Documentum devuaedms11 Y N PARTIAL|FAULTED
B Documentum devuaedms12 Y N PARTIAL
B Documentum_ENV_B devuaedms11 Y N PARTIAL
B Documentum_ENV_B devuaedms12 Y N STARTING|PARTIAL
B MNICB_SITgroup devuaedms11 Y N ONLINE
B MNICB_SITgroup devuaedms12 Y N ONLINE
B cvm devuaedms11 Y N ONLINE
B cvm devuaedms12 Y N ONLINE
B documentum_bkup devuaedms11 Y N ONLINE
B documentum_bkup devuaedms12 Y N OFFLINE

-- RESOURCES FAILED
-- Group Type Resource System

C Documentum ContentServer ContentServer devuaedms11
C Documentum_ENV_B ContentServer ContentServer_B devuaedms11
C Documentum_ENV_B Proxy DocBroker_B devuaedms11
C Documentum_ENV_B Proxy DocBroker_B devuaedms12

 

Please advice if there is any advice so I can fix it 

Re: Failed to start up the cluster service

You are only showing us a piece of the log for "user initiated stop". 

If something else happened prior to this, you need to check higher up in engine_A log.

I see various errors in engine_A since 13 July, e.g. :

2018/07/13 21:08:27 VCS ERROR V-16-2-13027 (devuaedms11) Resource(ContentServer_B) - monitor procedure did not complete within the expected time.

2018/07/17 07:52:08 VCS INFO V-16-1-10307 Resource SITgroup (Owner: unknown, Group: MNICB_SITgroup) is offline on devuaedms12 (Not initiated by VCS)

2018/07/17 07:52:09 VCS ERROR V-16-10001-5005 (devuaedms11) IPMultiNICB:mIP_bkupSmiley Surprisednline:Error in configuring IP address

2018/07/17 07:52:11 VCS ERROR V-16-1-10205 Group MNICB_SITgroup is faulted on system devuaedms11

2018/07/17 07:54:57 VCS ERROR V-16-2-13027 (devuaedms11) Resource(ContentServer) - monitor procedure did not complete within the expected time.
2018/07/17 08:00:31 VCS ERROR V-16-2-13210 (devuaedms12) Agent is calling clean for resource(ContentServer) because 4 successive invocations of the monitor procedure did not complete within the expected time.

2018/07/29 01:30:30 VCS ERROR V-16-2-13066 (devuaedms12) Agent is calling clean for resource(ContentServer) because the resource is not up even after online completed.


Many more errors related to network and resources such as ContentServer.

I feel that you need to perform a complete health check of the entire environment as EVERYTHING seems to be very old and out of support.

Re: Failed to start up the cluster service

Fully agree with Marianne, get us full logs, also, share the main.cf, what are the dependencies between the groups.