11-25-2011 05:13 AM
HI there,
I've been configuring cluster nodes with CFS/CVM. This week while configuring the web service group I had problem enabling and onlining a CFS mount point in the the Web Service group. The error I got is as follows: Interesting is all my other service groups are configured in the same manner and work fine. I'm only having an issue with this server and service group.
==============================================================================================
Error Log:
011/11/25 10:47:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource webmnt7 (Owner: Unspecified, Group: webserver) on System sun11
2011/11/25 10:47:31 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt7:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster ,mntlock=VCS Block Device /dev/vx/dsk/webdg/webvol7 : MountPoint /opt2/sw
2011/11/25 10:47:31 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt7:online:Mount Error : Usage:
mount [-v | %-p]
mount [-F FSType] [-V] [current_options] [-o specific_options]
{special | mount_point}
mount [-F FSType] [-V] [current_options] [-o specific_options]
special mount_point
mount %-a [-F FSType ] [-V] [current_options] [-o specific_options]
[mount_point ...]
2011/11/25 10:47:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group webserver on all nodes
2011/11/25 10:47:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource webmnt8 (Owner: Unspecified, Group: webserver) on System sun11
2011/11/25 10:47:37 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt8:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster,mntlock=VCS Block Device /dev/vx/dsk/webdg/webvol8 : MountPoint /appl2
2011/11/25 10:47:37 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt8:online:Mount Error : UX:vxfs mount: ERROR: V-3-21264: /dev/vx/dsk/webdg/webvol8 is already mounted, /appl2 is busy,
or the allowable number of mount points has been exceeded.
2011/11/25 10:48:33 VCS ERROR V-16-2-13066 (sun11) Agent is calling clean for resource(webmnt7) because the resource is not up even after online completed.
2011/11/25 10:48:34 VCS INFO V-16-2-13068 (sun11) Resource(webmnt7) - clean completed successfully.
2011/11/25 10:48:34 VCS INFO V-16-2-13071 (sun11) Resource(webmnt7): reached OnlineRetryLimit(0).
2011/11/25 10:48:34 VCS ERROR V-16-1-10303 Resource webmnt7 (Owner: Unspecified, Group: webserver) is FAULTED (timed out) on sys sun11
2011/11/25 10:48:38 VCS ERROR V-16-2-13066 (sun11) Agent is calling clean for resource(webmnt8) because the resource is not up even after online completed.
2011/11/25 10:48:39 VCS INFO V-16-2-13068 (sun11) Resource(webmnt8) - clean completed successfully.
2011/11/25 10:48:39 VCS INFO V-16-2-13071 (sun11) Resource(webmnt8): reached OnlineRetryLimit(0).
2011/11/25 10:48:39 VCS ERROR V-16-1-10303 Resource webmnt8 (Owner: Unspecified, Group: webserver) is FAULTED (timed out) on sys sun11
2011/11/25 11:02:42 VCS INFO V-16-1-50135 User root fired command: hares -online vxfsckd sun11 from localhost
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun11
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun41
2011/11/25 11:02:42 VCS INFO V-16-1-50135 User root fired command: hares -online vxfsckd sun41 from localhost
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun11
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun41
2011/11/25 11:03:08 VCS INFO V-16-1-10307 Resource webmnt7 (Owner: Unspecified, Group: webserver) is offline on sun11 (Not initiated by VCS)
2011/11/25 11:03:12 VCS INFO V-16-1-10307 Resource webmnt8 (Owner: Unspecified, Group: webserver) is offline on sun11 (Not initiated by VCS)
2011/11/25 11:03:58 VCS NOTICE V-16-1-10167 Initiating manual offline of group webserver on system sun11
==========================================
Main.cf
group vrts_vea_cfs_int_cfsmount9 (
SystemList = { sun11 = 0, sun41 = 1 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { sun11, sun41 }
)
CFSMount cfsmount9 (
Critical = 0
MountPoint = "/appl2"
BlockDevice = "/dev/vx/dsk/webdg/webvol8"
MountOpt @sun11 = "cluster"
MountOpt @sun41 = "cluster"
NodeList = { sun11, sun41 }
)
CVMVolDg cvmvoldg5 (
Critical = 0
CVMDiskGroup = webdg
CVMVolume = { webvol8 }
CVMActivation @sun11 = sw
CVMActivation @sun41 = sw
)
requires group cvm online local firm
cfsmount9 requires cvmvoldg5
// resource dependency tree
//
// group vrts_vea_cfs_int_cfsmount9
// {
// CFSMount cfsmount9
// {
// CVMVolDg cvmvoldg5
// }
// }
group webserver (
SystemList = { sun41 = 0, sun11 = 1 }
)
CFSMount webmnt8 (
Critical = 0
MountPoint = "/appl2 "
BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
MountOpt = "cluster"
)
CFSMount webvol7 (
Critical = 0
MountPoint = "/opt2/sw"
BlockDevice = "/dev/vx/dsk/webdg/webvol7 "
MountOpt = "cluster "
)
CVMVolDg webdg (
Critical = 0
CVMDiskGroup = webdg
CVMVolume = { "webvol7 ", webvol8 }
CVMActivation = "sw "
)
IPMultiNICB web_ip (
Critical = 0
BaseResName = "nic "
Address = "10.202.16.179 "
NetMask = "255.255.252.0 "
)
Proxy nic_proxy (
Critical = 0
TargetResName = nic
)
web_ip requires nic_proxy
webvol7 requires webdg
// resource dependency tree
//
// group webserver
// {
// IPMultiNICB web_ip
// {
// Proxy nic_proxy
// }
// CFSMount webmnt8
// CFSMount webvol7
// {
// CVMVolDg webdg
// }
// }
Thanks
Solved! Go to Solution.
01-17-2012 07:33 PM
Sometimes, using the VCS GUI, its difficult to see when you've added an extra space character in an attribute for a resource. There are several extra space characters in the above configs, which should be removed:
MountPoint = "/appl2 "
BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
BlockDevice = "/dev/vx/dsk/webdg/webvol7 "
CVMVolume = { "webvol7 ", webvol8 }
I saw 4 more of them. And it appeared also that some of the CFSMount resources weren't dependent on the CVMVolDg resource.
11-25-2011 08:46 AM
Have you tried to import the dg and mount the volume manually?
11-25-2011 12:22 PM
You don't need the""cluster" option in the MountOpt attribute of the CFSMount resource as the agent add this in for you, hence why you see cluster option being used twice in the logs - " -o cluster,cluster" - this is probably causing you your issue.
Mike
11-28-2011 02:47 AM
Hi Mike,
The same config works for the rest of my service groups. I'm stuck on this as this is the first time I'm having this issue. Please see the following for my Oracle resources.
======================================================================
2011/11/21 16:43:26 VCS INFO V-16-20011-5506 (sun11) CFSMount:ora_r02_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster,mntlock=VCS Block Device /dev/vx/dsk/oradg/oravol3 : MountPoint /data/oradata/r02
2011/11/21 16:43:26 VCS INFO V-16-20011-5506 (sun11) CFSMount:ora_r01_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster,mntlock=VCS Block Device /dev/vx/dsk/oradg/oravol2 : MountPoint /data/oradata/r01
...
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_arch_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_bkp_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_r02_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_r01_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_smprod (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_ecqrp (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_ecprd (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_autosys (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:45 VCS NOTICE V-16-20002-21 (sun11) Oracle:Oracle_autosys:online:Oracle startup returned the output
+--------------------------------------------------------------------+
LD_LIBRARY_PATH - /usr/lib:/opt1/sw/oracle/product/11.2.0/db/lib:
ORACLE instance started.
Total System Global Area 522092544 bytes
Fixed Size 2149672 bytes
Variable Size 398463704 bytes
Database Buffers 113246208 bytes
Redo Buffers 8232960 bytes
Database mounted.
Database opened.
=================================================
main.cf
group Oracle (
SystemList = { sun11 = 0, sun41 = 1 }
AutoStartList = { sun11, sun41 }
)
CFSMount ora_arch_mnt (
Critical = 0
MountPoint = "/data/oradata/archive"
BlockDevice = "/dev/vx/dsk/oradg/oravol4"
MountOpt = "cluster"
)
CFSMount ora_bkp_mnt (
Critical = 0
MountPoint = "/data/oradata/backup"
BlockDevice = "/dev/vx/dsk/oradg/oravol5"
MountOpt = "cluster"
)
....
group vrts_vea_cfs_int_cfsmount2 (
SystemList = { sun11 = 0, sun41 = 1 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { sun11, sun41 }
)
CFSMount cfsmount5 (
Critical = 0
MountPoint = "/data/oradata/archive"
BlockDevice = "/dev/vx/dsk/oradg/oravol4"
MountOpt @sun11 = "cluster"
MountOpt @sun41 = "cluster"
NodeList = { sun11, sun41 }
)
CFSMount cfsmount6 (
Critical = 0
MountPoint = "/data/oradata/backup"
BlockDevice = "/dev/vx/dsk/oradg/oravol5"
MountOpt @sun11 = "cluster"
MountOpt @sun41 = "cluster"
NodeList = { sun11, sun41 }
)
=================================
Anyway I'll try to remove the option and see if it works.
01-17-2012 07:33 PM
Sometimes, using the VCS GUI, its difficult to see when you've added an extra space character in an attribute for a resource. There are several extra space characters in the above configs, which should be removed:
MountPoint = "/appl2 "
BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
BlockDevice = "/dev/vx/dsk/webdg/webvol7 "
CVMVolume = { "webvol7 ", webvol8 }
I saw 4 more of them. And it appeared also that some of the CFSMount resources weren't dependent on the CVMVolDg resource.
01-27-2012 05:55 AM
CFSMount webmnt8 (
Critical = 0
MountPoint = "/appl2 "
BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
MountOpt = "cluster"
)