VCS volume group issue
I am facing an issue in VCS which is installed in Oracle Solaris 10 9/10 s10x_u9wos_14a X86 servers. The shared volume group suddenly disappeared from both servers and since then I am unable to use VCS. Could you please advise in how to solve this issue
engine_A.log:
2011/12/13 09:32:20 VCS WARNING V-16-10001-1060 (sahomxapp01u) DiskGroup:App_dg:clean:The command 'vxvol -g appdg stopall' failed. Doing a forced stop
2011/12/13 09:32:21 VCS INFO V-16-2-13716 (sahomxapp01u) Resource(App_dg): Output of the completed operation (clean)
==============================================
VxVM vxdg ERROR V-5-1-580 Disk group appdg: Flush failed: Disk group not found
VxVM vxdg ERROR V-5-1-581 Disk group appdg: No such disk group is imported
==============================================
2011/12/13 09:32:21 VCS INFO V-16-2-13068 (sahomxapp01u) Resource(App_dg) - clean completed successfully.
2011/12/13 09:32:21 VCS INFO V-16-2-13072 (sahomxapp01u) Resource(App_dg): Agent is retrying online (attempt number 1 of 1).
2011/12/13 09:32:21 VCS WARNING V-16-10001-1014 (sahomxapp01u) DiskGroup:App_dg:online:Diskgroups will be imported without reservations
2011/12/13 09:32:24 VCS WARNING V-16-10001-1016 (sahomxapp01u) DiskGroup:App_dg:online:vxdg import (clear flag) failed.
2011/12/13 09:32:24 VCS WARNING V-16-10001-1017 (sahomxapp01u) DiskGroup:App_dg:online:Trying force import for the diskgroup.
2011/12/13 09:32:24 VCS ERROR V-16-10001-1003 (sahomxapp01u) DiskGroup:App_dg:online:** ERROR: vxdg import (force) failed on Disk Group appdg
2011/12/13 09:32:29 VCS ERROR V-16-10001-1004 (sahomxapp01u) DiskGroup:App_dg:online:** ERROR: vxdg import failed on Disk Group appdg after vxdctl enable
2011/12/13 09:32:29 VCS INFO V-16-2-13716 (sahomxapp01u) Resource(App_dg): Output of the completed operation (online)
==============================================
VxVM vxdg ERROR V-5-1-10978 Disk group appdg: import failed:
No valid disk found containing disk group
2011/12/13 09:28:00 VCS INFO V-16-2-13716 (sahomxapp01u) Resource(App_dg): Output of the completed operation (clean)
==============================================
VxVM vxdg ERROR V-5-1-580 Disk group appdg: Flush failed: Disk group not found
VxVM vxdg ERROR V-5-1-581 Disk group appdg: No such disk group is imported
==============================================
2011/12/13 09:28:00 VCS INFO V-16-2-13068 (sahomxapp01u) Resource(App_dg) - clean completed successfully.
2011/12/13 09:28:00 VCS INFO V-16-2-13072 (sahomxapp01u) Resource(App_dg): Agent is retrying online (attempt number 1 of 1).
2011/12/13 09:28:00 VCS WARNING V-16-10001-1014 (sahomxapp01u) DiskGroup:App_dg:online:Diskgroups will be imported without reservations
2011/12/13 09:28:03 VCS WARNING V-16-10001-1016 (sahomxapp01u) DiskGroup:App_dg:online:vxdg import (clear flag) failed.
2011/12/13 09:28:03 VCS WARNING V-16-10001-1017 (sahomxapp01u) DiskGroup:App_dg:online:Trying force import for the diskgroup.
2011/12/13 09:28:03 VCS ERROR V-16-10001-1003 (sahomxapp01u) DiskGroup:App_dg:online:** ERROR: vxdg import (force) failed on Disk Group appdg
2011/12/13 09:28:07 VCS ERROR V-16-10001-1004 (sahomxapp01u) DiskGroup:App_dg:online:** ERROR: vxdg import failed on Disk Group appdg after vxdctl enable
2011/12/13 09:28:08 VCS INFO V-16-2-13716 (sahomxapp01u) Resource(App_dg): Output of the completed operation (online)
==============================================
VxVM vxdg ERROR V-5-1-10978 Disk group appdg: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group appdg: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group appdg: import failed:
No valid disk found containing disk group
==============================================
2011/12/13 09:30:09 VCS ERROR V-16-2-13066 (sahomxapp01u) Agent is calling clean for resource(App_dg) because the resource is not up even after online completed.
2011/12/13 09:30:09 VCS WARNING V-16-10001-1060 (sahomxapp01u) DiskGroup:App_dg:clean:The command 'vxvol -g appdg stopall' failed. Doing a forced stop
2011/12/13 09:30:10 VCS INFO V-16-2-13716 (sahomxapp01u) Resource(App_dg): Output of the completed operation (clean)
==============================================
VxVM vxdg ERROR V-5-1-580 Disk group appdg: Flush failed: Disk group not found
VxVM vxdg ERROR V-5-1-581 Disk group appdg: No such disk group is imported
==============================================
2011/12/13 09:30:10 VCS INFO V-16-2-13068 (sahomxapp01u) Resource(App_dg) - clean completed successfully.
2011/12/13 09:30:10 VCS INFO V-16-2-13071 (sahomxapp01u) Resource(App_dg): reached OnlineRetryLimit(1).
2011/12/13 09:30:11 VCS ERROR V-16-1-10303 Resource App_dg (Owner: Unspecified, Group: App_Group) is FAULTED (timed out) on sys sahomxapp01u
2011/12/13 09:30:11 VCS NOTICE V-16-1-10184 Group Retry: retrying group App_Group on System sahomxapp01u (try #2)
2011/12/13 09:30:11 VCS ERROR V-16-1-10205 Group App_Group is faulted on system sahomxapp01u
2011/12/13 09:30:11 VCS NOTICE V-16-1-10446 Group App_Group is offline on system sahomxapp01u
2011/12/13 09:30:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource App_dg (Owner: Unspecified, Group: App_Group) on System sahomxapp01u
2011/12/13 09:30:11 VCS WARNING V-16-10001-1014 (sahomxapp01u) DiskGroup:App_dg:online:Diskgroups will be imported without reservations
2011/12/13 09:30:14 VCS WARNING V-16-10001-1016 (sahomxapp01u) DiskGroup:App_dg:online:vxdg import (clear flag) failed.
2011/12/13 09:30:14 VCS WARNING V-16-10001-1017 (sahomxapp01u) DiskGroup:App_dg:online:Trying force import for the diskgroup.
2011/12/13 09:30:14 VCS ERROR V-16-10001-1003 (sahomxapp01u) DiskGroup:App_dg:online:** ERROR: vxdg import (force) failed on Disk Group appdg
2011/12/13 09:30:19 VCS ERROR V-16-10001-1004 (sahomxapp01u) DiskGroup:App_dg:online:** ERROR: vxdg import failed on Disk Group appdg after vxdctl enable
2011/12/13 09:30:19 VCS INFO V-16-2-13716 (sahomxapp01u) Resource(App_dg): Output of the completed operation (online)
==============================================
VxVM vxdg ERROR V-5-1-10978 Disk group appdg: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group appdg: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group appdg: import failed:
No valid disk found containing disk group
==============================================
2011/12/13 09:32:20 VCS ERROR V-16-2-13066 (sahomxapp01u) Agent is calling clean for resource(App_dg) because the resource is not up even after online completed.
2011/12/13 09:32:20 VCS WARNING V-16-10001-1060 (sahomxapp01u) DiskGroup:App_dg:clean:The command 'vxvol -g appdg stopall' failed. Doing a forced stop
2011/12/13 09:32:21 VCS INFO V-16-2-13716 (sahomxapp01u) Resource(App_dg): Output of the completed operation (clean)
==============================================
VxVM vxdg ERROR V-5-1-580 Disk group appdg: Flush failed: Disk group not found
VxVM vxdg ERROR V-5-1-581 Disk group appdg: No such disk group is imported
==============================================
(root_sahomxapp01u)@/var/VRTSvcs/log # vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:none - - online invalid
emc_clariion0_0 auto:cdsdisk - - error
emc_clariion0_1 auto:cdsdisk - - error
(root_sahomxapp01u)@/var/VRTSvcs/log #
(root_sahomxapp01u)@/var/VRTSvcs/log # vxprint -htr
NOTHING
- Did u mean that storage disappeared ? From the logs it appears that some thing happened to storage and it was not accessible.... Could be possible that veritas had issues during that time.... Is your storage back online now ? G