CVM mount point fails to be online when configuring with VCS 5.1SP1
HI there,
I've been configuring cluster nodes with CFS/CVM. This week while configuring the web service group I had problem enabling and onlining a CFS mount point in the the Web Service group. The error I got is as follows: Interesting is all my other service groups are configured in the same manner and work fine. I'm only having an issue with this server and service group.
==============================================================================================
Error Log:
011/11/25 10:47:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource webmnt7 (Owner: Unspecified, Group: webserver) on System sun11
2011/11/25 10:47:31 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt7:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster ,mntlock=VCS Block Device /dev/vx/dsk/webdg/webvol7 : MountPoint /opt2/sw
2011/11/25 10:47:31 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt7:online:Mount Error : Usage:
mount [-v | %-p]
mount [-F FSType] [-V] [current_options] [-o specific_options]
{special | mount_point}
mount [-F FSType] [-V] [current_options] [-o specific_options]
special mount_point
mount %-a [-F FSType ] [-V] [current_options] [-o specific_options]
[mount_point ...]
2011/11/25 10:47:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group webserver on all nodes
2011/11/25 10:47:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource webmnt8 (Owner: Unspecified, Group: webserver) on System sun11
2011/11/25 10:47:37 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt8:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster,mntlock=VCS Block Device /dev/vx/dsk/webdg/webvol8 : MountPoint /appl2
2011/11/25 10:47:37 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt8:online:Mount Error : UX:vxfs mount: ERROR: V-3-21264: /dev/vx/dsk/webdg/webvol8 is already mounted, /appl2 is busy,
or the allowable number of mount points has been exceeded.
2011/11/25 10:48:33 VCS ERROR V-16-2-13066 (sun11) Agent is calling clean for resource(webmnt7) because the resource is not up even after online completed.
2011/11/25 10:48:34 VCS INFO V-16-2-13068 (sun11) Resource(webmnt7) - clean completed successfully.
2011/11/25 10:48:34 VCS INFO V-16-2-13071 (sun11) Resource(webmnt7): reached OnlineRetryLimit(0).
2011/11/25 10:48:34 VCS ERROR V-16-1-10303 Resource webmnt7 (Owner: Unspecified, Group: webserver) is FAULTED (timed out) on sys sun11
2011/11/25 10:48:38 VCS ERROR V-16-2-13066 (sun11) Agent is calling clean for resource(webmnt8) because the resource is not up even after online completed.
2011/11/25 10:48:39 VCS INFO V-16-2-13068 (sun11) Resource(webmnt8) - clean completed successfully.
2011/11/25 10:48:39 VCS INFO V-16-2-13071 (sun11) Resource(webmnt8): reached OnlineRetryLimit(0).
2011/11/25 10:48:39 VCS ERROR V-16-1-10303 Resource webmnt8 (Owner: Unspecified, Group: webserver) is FAULTED (timed out) on sys sun11
2011/11/25 11:02:42 VCS INFO V-16-1-50135 User root fired command: hares -online vxfsckd sun11 from localhost
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun11
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun41
2011/11/25 11:02:42 VCS INFO V-16-1-50135 User root fired command: hares -online vxfsckd sun41 from localhost
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun11
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun41
2011/11/25 11:03:08 VCS INFO V-16-1-10307 Resource webmnt7 (Owner: Unspecified, Group: webserver) is offline on sun11 (Not initiated by VCS)
2011/11/25 11:03:12 VCS INFO V-16-1-10307 Resource webmnt8 (Owner: Unspecified, Group: webserver) is offline on sun11 (Not initiated by VCS)
2011/11/25 11:03:58 VCS NOTICE V-16-1-10167 Initiating manual offline of group webserver on system sun11
==========================================
Main.cf
group vrts_vea_cfs_int_cfsmount9 (
SystemList = { sun11 = 0, sun41 = 1 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { sun11, sun41 }
)
CFSMount cfsmount9 (
Critical = 0
MountPoint = "/appl2"
BlockDevice = "/dev/vx/dsk/webdg/webvol8"
MountOpt @sun11 = "cluster"
MountOpt @sun41 = "cluster"
NodeList = { sun11, sun41 }
)
CVMVolDg cvmvoldg5 (
Critical = 0
CVMDiskGroup = webdg
CVMVolume = { webvol8 }
CVMActivation @sun11 = sw
CVMActivation @sun41 = sw
)
requires group cvm online local firm
cfsmount9 requires cvmvoldg5
// resource dependency tree
//
// group vrts_vea_cfs_int_cfsmount9
// {
// CFSMount cfsmount9
// {
// CVMVolDg cvmvoldg5
// }
// }
group webserver (
SystemList = { sun41 = 0, sun11 = 1 }
)
CFSMount webmnt8 (
Critical = 0
MountPoint = "/appl2 "
BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
MountOpt = "cluster"
)
CFSMount webvol7 (
Critical = 0
MountPoint = "/opt2/sw"
BlockDevice = "/dev/vx/dsk/webdg/webvol7 "
MountOpt = "cluster "
)
CVMVolDg webdg (
Critical = 0
CVMDiskGroup = webdg
CVMVolume = { "webvol7 ", webvol8 }
CVMActivation = "sw "
)
IPMultiNICB web_ip (
Critical = 0
BaseResName = "nic "
Address = "10.202.16.179 "
NetMask = "255.255.252.0 "
)
Proxy nic_proxy (
Critical = 0
TargetResName = nic
)
web_ip requires nic_proxy
webvol7 requires webdg
// resource dependency tree
//
// group webserver
// {
// IPMultiNICB web_ip
// {
// Proxy nic_proxy
// }
// CFSMount webmnt8
// CFSMount webvol7
// {
// CVMVolDg webdg
// }
// }
Thanks
Sometimes, using the VCS GUI, its difficult to see when you've added an extra space character in an attribute for a resource. There are several extra space characters in the above configs, which should be removed:
MountPoint = "/appl2 "
BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
BlockDevice = "/dev/vx/dsk/webdg/webvol7 "
CVMVolume = { "webvol7 ", webvol8 }
I saw 4 more of them. And it appeared also that some of the CFSMount resources weren't dependent on the CVMVolDg resource.
