Hello everyone:
I´m having this issue that seems to be irregular, sometimes work and sometimes don´t
I installed Storage Foundation For Oracle RAC v5 MP3
OS Solaris 10 10/08 Generic_139555-08
Oracle RAC 10g
Two node Cluster
I tried to shutdown cluster server but to leave services online
so I typed on one node:
# hastop -all -force
But checking the status of the cluster, everything was going offline
Just tried the same on another similar system an worked fine, the only difference are the apps and the Maintenance Pack is version 1 on this system
Does anybody knows about a bug or something about this situation?
Thanks for any info
Here´s an output from the logs
2010/03/08 12:59:07 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_ALL_FORCE from localhost
2010/03/08 12:59:07 VCS NOTICE V-16-1-10322 System mxcpm9kh1101dc4 (Node '0') changed state from RUNNING to LEAVING
2010/03/08 12:59:07 VCS NOTICE V-16-1-10322 System mxcpm9kh1401dc4 (Node '1') changed state from RUNNING to LEAVING
2010/03/08 12:59:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource VCSWeb (Owner: unknown, Group: ClusterService) on System mxcpm9kh1101dc4
2010/03/08 12:59:07 VCS NOTICE V-16-1-10300 Initiating Offline of Resource VCSWeb (Owner: unknown, Group: ClusterService) on System mxcpm9kh1401dc4
2010/03/08 12:59:10 VCS INFO V-16-10001-11005 (mxcpm9kh1401dc4) VRTSWebApp:VCSWeb:offline:Output of completed operation - 'Web application cmc shutdown su
ccessfully'
2010/03/08 12:59:11 VCS INFO V-16-10001-11005 (mxcpm9kh1101dc4) VRTSWebApp:VCSWeb:offline:Output of completed operation - 'Web application cmc shutdown su
ccessfully'
2010/03/08 12:59:12 VCS INFO V-16-1-10305 Resource VCSWeb (Owner: unknown, Group: ClusterService) is offline on mxcpm9kh1401dc4 (VCS initiated)
2010/03/08 12:59:12 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system mxcpm9kh1401dc4
2010/03/08 12:59:12 VCS NOTICE V-16-1-10322 System mxcpm9kh1401dc4 (Node '1') changed state from LEAVING to EXITING
2010/03/08 12:59:12 VCS NOTICE V-16-1-10322 System mxcpm9kh1401dc4 (Node '1') changed state from EXITING to EXITED
2010/03/08 12:59:18 VCS INFO V-16-1-10077 Received new cluster membership
2010/03/08 12:59:18 VCS NOTICE V-16-1-10112 System (mxcpm9kh1101dc4) - Membership: 0x1, DDNA: 0x2
2010/03/08 12:59:18 VCS ERROR V-16-1-10113 System mxcpm9kh1401dc4 (Node '1') is in DDNA Membership - Membership: 0x1, Visible: 0x0
2010/03/08 12:59:18 VCS NOTICE V-16-1-10449 Group cvm autodisabled on node mxcpm9kh1401dc4 until it is probed
2010/03/08 12:59:18 VCS NOTICE V-16-1-10449 Group ora_arch autodisabled on node mxcpm9kh1401dc4 until it is probed
2010/03/08 12:59:18 VCS NOTICE V-16-1-10449 Group ora_crystwb autodisabled on node mxcpm9kh1401dc4 until it is probed
2010/03/08 12:59:18 VCS NOTICE V-16-1-10449 Group ora_danpro autodisabled on node mxcpm9kh1401dc4 until it is probed
2010/03/08 12:59:18 VCS NOTICE V-16-1-10449 Group ora_faxflow autodisabled on node mxcpm9kh1401dc4 until it is probed
2010/03/08 12:59:18 VCS NOTICE V-16-1-10449 Group ora_tandapro autodisabled on node mxcpm9kh1401dc4 until it is probed
2010/03/08 12:59:18 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on node mxcpm9kh1401dc4 until it is probed
2010/03/08 12:59:18 VCS NOTICE V-16-1-10446 Group cvm is offline on system mxcpm9kh1401dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10446 Group ora_arch is offline on system mxcpm9kh1401dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10446 Group ora_crystwb is offline on system mxcpm9kh1401dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10446 Group ora_danpro is offline on system mxcpm9kh1401dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10446 Group ora_faxflow is offline on system mxcpm9kh1401dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10446 Group ora_tandapro is offline on system mxcpm9kh1401dc4
2010/03/08 12:59:18 VCS INFO V-16-1-10305 Resource VCSWeb (Owner: unknown, Group: ClusterService) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:18 VCS NOTICE V-16-1-10446 Group ClusterService is offline on system mxcpm9kh1101dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10300 Initiating Offline of Resource CRYSTWB_ORA (Owner: unknown, Group: ora_crystwb) on System mxcpm9kh1101dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_ORA (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10300 Initiating Offline of Resource FAXFLOW_ORA (Owner: unknown, Group: ora_faxflow) on System mxcpm9kh1101dc4
2010/03/08 12:59:18 VCS NOTICE V-16-1-10300 Initiating Offline of Resource TANDAPRO_ORA (Owner: unknown, Group: ora_tandapro) on System mxcpm9kh1101dc4
2010/03/08 12:59:18 VCS INFO V-16-6-15002 (mxcpm9kh1101dc4) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline mxcpm9kh1101dc4 Cluster
Service successfully
2010/03/08 12:59:18 VCS INFO V-16-6-15004 (mxcpm9kh1101dc4) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2010/03/08 12:59:29 VCS NOTICE V-16-20002-25 (mxcpm9kh1101dc4) Oracle:FAXFLOW_ORA:offline:Oracle shutdown returned the output
+--------------------------------------------------------------------+
LD_LIBRARY_PATH - /usr/lib:/u01/sq/ora_1/oracle/db/10.2.0/lib:
Database closed.
Database dismounted.
ORACLE instance shut down.
+====================================================================+
2010/03/08 12:59:30 VCS INFO V-16-1-10305 Resource FAXFLOW_ORA (Owner: unknown, Group: ora_faxflow) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:30 VCS NOTICE V-16-1-10300 Initiating Offline of Resource FAXFLOW_MNT_u09 (Owner: unknown, Group: ora_faxflow) on System mxcpm9kh1101dc4
2010/03/08 12:59:31 VCS NOTICE V-16-20002-25 (mxcpm9kh1101dc4) Oracle:TANDAPRO_ORA:offline:Oracle shutdown returned the output
+--------------------------------------------------------------------+
LD_LIBRARY_PATH - /usr/lib:/u01/sq/ora_1/oracle/db/10.2.0/lib:
Database closed.
Database dismounted.
ORACLE instance shut down.
+====================================================================+
2010/03/08 12:59:31 VCS INFO V-16-1-10305 Resource TANDAPRO_ORA (Owner: unknown, Group: ora_tandapro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource TANDAPRO_MNT_u08 (Owner: unknown, Group: ora_tandapro) on System mxcpm9kh1101dc
4
2010/03/08 12:59:31 VCS INFO V-16-1-10305 Resource FAXFLOW_MNT_u09 (Owner: unknown, Group: ora_faxflow) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource FAXFLOW_DG (Owner: unknown, Group: ora_faxflow) on System mxcpm9kh1101dc4
2010/03/08 12:59:32 VCS INFO V-16-1-10305 Resource TANDAPRO_MNT_u08 (Owner: unknown, Group: ora_tandapro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource TANDAPRO_DG (Owner: unknown, Group: ora_tandapro) on System mxcpm9kh1101dc4
2010/03/08 12:59:33 VCS INFO V-16-1-10305 Resource FAXFLOW_DG (Owner: unknown, Group: ora_faxflow) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:33 VCS NOTICE V-16-1-10446 Group ora_faxflow is offline on system mxcpm9kh1101dc4
2010/03/08 12:59:34 VCS INFO V-16-6-15002 (mxcpm9kh1101dc4) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline mxcpm9kh1101dc4 ora_fax
flow successfully
2010/03/08 12:59:34 VCS INFO V-16-6-15004 (mxcpm9kh1101dc4) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2010/03/08 12:59:34 VCS INFO V-16-1-10305 Resource TANDAPRO_DG (Owner: unknown, Group: ora_tandapro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:34 VCS NOTICE V-16-1-10446 Group ora_tandapro is offline on system mxcpm9kh1101dc4
2010/03/08 12:59:35 VCS INFO V-16-6-15002 (mxcpm9kh1101dc4) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline mxcpm9kh1101dc4 ora_tan
dapro successfully
2010/03/08 12:59:35 VCS INFO V-16-6-15004 (mxcpm9kh1101dc4) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2010/03/08 12:59:36 VCS NOTICE V-16-20002-25 (mxcpm9kh1101dc4) Oracle:DANPRO_ORA:offline:Oracle shutdown returned the output
+--------------------------------------------------------------------+
LD_LIBRARY_PATH - /usr/lib:/u01/sq/ora_1/oracle/db/10.2.0/lib:
Database closed.
Database dismounted.
ORACLE instance shut down.
+====================================================================+
2010/03/08 12:59:36 VCS INFO V-16-1-10305 Resource DANPRO_ORA (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_danpro2 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc
4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u100 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u101 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u102 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u103 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u104 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u105 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u106 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u107 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u108 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u109 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u110 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u111 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u112 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u113 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u114 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u115 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u116 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u117 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u118 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_MNT_u119 (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:36 VCS NOTICE V-16-20002-25 (mxcpm9kh1101dc4) Oracle:CRYSTWB_ORA:offline:Oracle shutdown returned the output
+--------------------------------------------------------------------+
LD_LIBRARY_PATH - /usr/lib:/u01/sq/ora_1/oracle/db/10.2.0/lib:
Database closed.
Database dismounted.
ORACLE instance shut down.
+====================================================================+
2010/03/08 12:59:37 VCS INFO V-16-1-10305 Resource CRYSTWB_ORA (Owner: unknown, Group: ora_crystwb) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:37 VCS NOTICE V-16-1-10300 Initiating Offline of Resource CRYSTWB_MNT_u07 (Owner: unknown, Group: ora_crystwb) on System mxcpm9kh1101dc4
2010/03/08 12:59:38 VCS INFO V-16-1-10305 Resource DANPRO_MNT_danpro2 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:38 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO2_DG (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 12:59:38 VCS ERROR V-16-20011-5503 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u100:offline:Umount Failed : Mount Point : /u100
2010/03/08 12:59:38 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u100:offline:Attempting fuser TERM : Mount Point : /u100
2010/03/08 12:59:39 VCS ERROR V-16-20011-5503 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u119:offline:Umount Failed : Mount Point : /u119
2010/03/08 12:59:39 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u119:offline:Attempting fuser TERM : Mount Point : /u119
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u107 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u108 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u102 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u101 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u104 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u105 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u110 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u106 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u103 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u109 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u111 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u113 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u116 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u117 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u118 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO2_DG (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u114 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u112 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:40 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u115 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:42 VCS INFO V-16-1-10305 Resource CRYSTWB_MNT_u07 (Owner: unknown, Group: ora_crystwb) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:42 VCS NOTICE V-16-1-10300 Initiating Offline of Resource CRYSTWB_DG (Owner: unknown, Group: ora_crystwb) on System mxcpm9kh1101dc4
2010/03/08 12:59:44 VCS INFO V-16-1-10305 Resource CRYSTWB_DG (Owner: unknown, Group: ora_crystwb) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 12:59:44 VCS NOTICE V-16-1-10446 Group ora_crystwb is offline on system mxcpm9kh1101dc4
2010/03/08 12:59:44 VCS INFO V-16-6-15002 (mxcpm9kh1101dc4) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline mxcpm9kh1101dc4 ora_cry
stwb successfully
2010/03/08 12:59:44 VCS INFO V-16-6-15004 (mxcpm9kh1101dc4) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2010/03/08 12:59:47 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u100:offline:Attempting fuser TERM : Mount Point : /u100
2010/03/08 12:59:49 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u119:offline:Attempting fuser TERM : Mount Point : /u119
2010/03/08 12:59:54 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u100:offline:Attempting fuser KILL : Mount Point : /u100
2010/03/08 12:59:56 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u119:offline:Attempting fuser KILL : Mount Point : /u119
2010/03/08 13:00:01 VCS NOTICE V-16-20011-5504 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u100:offline:Force Umount : Mount Point : /u100
2010/03/08 13:00:02 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u100 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:02 VCS NOTICE V-16-20011-5504 (mxcpm9kh1101dc4) CFSMount:DANPRO_MNT_u119:offline:Force Umount : Mount Point : /u119
2010/03/08 13:00:02 VCS INFO V-16-1-10305 Resource DANPRO_MNT_u119 (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource DANPRO_DG (Owner: unknown, Group: ora_danpro) on System mxcpm9kh1101dc4
2010/03/08 13:00:04 VCS INFO V-16-1-10305 Resource DANPRO_DG (Owner: unknown, Group: ora_danpro) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:04 VCS NOTICE V-16-1-10446 Group ora_danpro is offline on system mxcpm9kh1101dc4
2010/03/08 13:00:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ORARC_MNT_U05 (Owner: unknown, Group: ora_arch) on System mxcpm9kh1101dc4
2010/03/08 13:00:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ORREDO_MNT_U02 (Owner: unknown, Group: ora_arch) on System mxcpm9kh1101dc4
2010/03/08 13:00:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ORREDO_MNT_U03 (Owner: unknown, Group: ora_arch) on System mxcpm9kh1101dc4
2010/03/08 13:00:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ORREDO_MNT_U04 (Owner: unknown, Group: ora_arch) on System mxcpm9kh1101dc4
2010/03/08 13:00:05 VCS INFO V-16-6-15002 (mxcpm9kh1101dc4) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline mxcpm9kh1101dc4 ora_dan
pro successfully
2010/03/08 13:00:05 VCS INFO V-16-6-15004 (mxcpm9kh1101dc4) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2010/03/08 13:00:05 VCS ERROR V-16-20011-5503 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U03:offline:Umount Failed : Mount Point : /u03
2010/03/08 13:00:05 VCS ERROR V-16-20011-5503 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U02:offline:Umount Failed : Mount Point : /u02
2010/03/08 13:00:05 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U03:offline:Attempting fuser TERM : Mount Point : /u03
2010/03/08 13:00:05 VCS ERROR V-16-20011-5503 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U04:offline:Umount Failed : Mount Point : /u04
2010/03/08 13:00:05 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U02:offline:Attempting fuser TERM : Mount Point : /u02
2010/03/08 13:00:05 VCS NOTICE V-16-20011-5510 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U04:offline:Attempting fuser TERM : Mount Point : /u04
2010/03/08 13:00:05 VCS INFO V-16-1-10305 Resource ORARC_MNT_U05 (Owner: unknown, Group: ora_arch) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:05 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ORARC_DG (Owner: unknown, Group: ora_arch) on System mxcpm9kh1101dc4
2010/03/08 13:00:07 VCS INFO V-16-1-10305 Resource ORARC_DG (Owner: unknown, Group: ora_arch) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:14 VCS NOTICE V-16-20011-5504 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U03:offline:Force Umount : Mount Point : /u03
2010/03/08 13:00:14 VCS NOTICE V-16-20011-5504 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U02:offline:Force Umount : Mount Point : /u02
2010/03/08 13:00:14 VCS NOTICE V-16-20011-5504 (mxcpm9kh1101dc4) CFSMount:ORREDO_MNT_U04:offline:Force Umount : Mount Point : /u04
2010/03/08 13:00:14 VCS INFO V-16-1-10305 Resource ORREDO_MNT_U02 (Owner: unknown, Group: ora_arch) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:15 VCS INFO V-16-1-10305 Resource ORREDO_MNT_U03 (Owner: unknown, Group: ora_arch) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:15 VCS INFO V-16-1-10305 Resource ORREDO_MNT_U04 (Owner: unknown, Group: ora_arch) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ORREDO_DG (Owner: unknown, Group: ora_arch) on System mxcpm9kh1101dc4
2010/03/08 13:00:17 VCS INFO V-16-1-10305 Resource ORREDO_DG (Owner: unknown, Group: ora_arch) is offline on mxcpm9kh1101dc4 (VCS initiated)
2010/03/08 13:00:17 VCS NOTICE V-16-1-10446 Group ora_arch is offline on system mxcpm9kh1101dc4
2010/03/08 13:00:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cssd (Owner: unknown, Group: cvm) on System mxcpm9kh1101dc4
2010/03/08 13:00:17 VCS INFO V-16-6-15002 (mxcpm9kh1101dc4) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline mxcpm9kh1101dc4 ora_arc
h successfully
2010/03/08 13:00:17 VCS INFO V-16-6-15004 (mxcpm9kh1101dc4) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2010/03/08 13:00:36 VCS INFO V-16-10001-3 (mxcpm9kh1101dc4) Application:cssd:offline:Executed /opt/VRTSvcs/rac/bin/cssd-offline
2010/03/08 13:00:36 VCS INFO V-16-2-13001 (mxcpm9kh1101dc4) Resource(cssd): Output of the completed operation (offline)
Shutting down Oracle Cluster Ready Services (CRS):