cancel
Showing results for 
Search instead for 
Did you mean: 

CVM mount point fails to be online when configuring with VCS 5.1SP1

SK_Ram
Level 3
Accredited

HI there,

I've been configuring cluster nodes with CFS/CVM. This week while configuring the web service group I had problem enabling and onlining a CFS mount point in the the Web Service group. The error I got is as follows: Interesting is all my other service groups are configured in the same manner and work fine. I'm only having an issue with this server and service group.

 

==============================================================================================

Error Log:

011/11/25 10:47:31 VCS NOTICE V-16-1-10301 Initiating Online of Resource webmnt7 (Owner: Unspecified, Group: webserver) on System sun11
2011/11/25 10:47:31 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt7:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster ,mntlock=VCS                 Block Device /dev/vx/dsk/webdg/webvol7  : MountPoint /opt2/sw
2011/11/25 10:47:31 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt7:online:Mount Error : Usage:
mount [-v | %-p]
mount [-F FSType] [-V] [current_options] [-o specific_options]
        {special | mount_point}
mount [-F FSType] [-V] [current_options] [-o specific_options]
        special mount_point
mount %-a [-F FSType ] [-V] [current_options] [-o specific_options]
        [mount_point ...]
2011/11/25 10:47:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group webserver on all nodes
2011/11/25 10:47:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource webmnt8 (Owner: Unspecified, Group: webserver) on System sun11
2011/11/25 10:47:37 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt8:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster,mntlock=VCS                 Block Device /dev/vx/dsk/webdg/webvol8  : MountPoint /appl2
2011/11/25 10:47:37 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt8:online:Mount Error : UX:vxfs mount: ERROR: V-3-21264: /dev/vx/dsk/webdg/webvol8 is already mounted, /appl2 is busy,
                 or the allowable number of mount points has been exceeded.
2011/11/25 10:48:33 VCS ERROR V-16-2-13066 (sun11) Agent is calling clean for resource(webmnt7) because the resource is not up even after online completed.
2011/11/25 10:48:34 VCS INFO V-16-2-13068 (sun11) Resource(webmnt7) - clean completed successfully.
2011/11/25 10:48:34 VCS INFO V-16-2-13071 (sun11) Resource(webmnt7): reached OnlineRetryLimit(0).
2011/11/25 10:48:34 VCS ERROR V-16-1-10303 Resource webmnt7 (Owner: Unspecified, Group: webserver) is FAULTED (timed out) on sys sun11
2011/11/25 10:48:38 VCS ERROR V-16-2-13066 (sun11) Agent is calling clean for resource(webmnt8) because the resource is not up even after online completed.
2011/11/25 10:48:39 VCS INFO V-16-2-13068 (sun11) Resource(webmnt8) - clean completed successfully.
2011/11/25 10:48:39 VCS INFO V-16-2-13071 (sun11) Resource(webmnt8): reached OnlineRetryLimit(0).
2011/11/25 10:48:39 VCS ERROR V-16-1-10303 Resource webmnt8 (Owner: Unspecified, Group: webserver) is FAULTED (timed out) on sys sun11
2011/11/25 11:02:42 VCS INFO V-16-1-50135 User root fired command: hares -online vxfsckd  sun11  from localhost
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun11
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun41
2011/11/25 11:02:42 VCS INFO V-16-1-50135 User root fired command: hares -online vxfsckd  sun41  from localhost
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun11
2011/11/25 11:02:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node sun41
2011/11/25 11:03:08 VCS INFO V-16-1-10307 Resource webmnt7 (Owner: Unspecified, Group: webserver) is offline on sun11 (Not initiated by VCS)
2011/11/25 11:03:12 VCS INFO V-16-1-10307 Resource webmnt8 (Owner: Unspecified, Group: webserver) is offline on sun11 (Not initiated by VCS)
2011/11/25 11:03:58 VCS NOTICE V-16-1-10167 Initiating manual offline of group webserver on system sun11

==========================================

Main.cf


group vrts_vea_cfs_int_cfsmount9 (
        SystemList = { sun11 = 0, sun41 = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { sun11, sun41 }
        )

        CFSMount cfsmount9 (
                Critical = 0
                MountPoint = "/appl2"
                BlockDevice = "/dev/vx/dsk/webdg/webvol8"
                MountOpt @sun11 = "cluster"
                MountOpt @sun41 = "cluster"
                NodeList = { sun11, sun41 }
                )

        CVMVolDg cvmvoldg5 (
                Critical = 0
                CVMDiskGroup = webdg
                CVMVolume = { webvol8 }
                CVMActivation @sun11 = sw
                CVMActivation @sun41 = sw
                )

        requires group cvm online local firm
        cfsmount9 requires cvmvoldg5


        // resource dependency tree
        //
        //      group vrts_vea_cfs_int_cfsmount9
        //      {
        //      CFSMount cfsmount9
        //          {
        //          CVMVolDg cvmvoldg5
        //          }
        //      }


group webserver (
        SystemList = { sun41 = 0, sun11 = 1 }
        )

        CFSMount webmnt8 (
                Critical = 0
                MountPoint = "/appl2 "
                BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
                MountOpt = "cluster"
                )

        CFSMount webvol7 (
                Critical = 0
                MountPoint = "/opt2/sw"
                BlockDevice = "/dev/vx/dsk/webdg/webvol7 "
                MountOpt = "cluster "
                )

        CVMVolDg webdg (
                Critical = 0
                CVMDiskGroup = webdg
                CVMVolume = { "webvol7 ", webvol8 }
                CVMActivation = "sw "
                )

        IPMultiNICB web_ip (
                Critical = 0
                BaseResName = "nic "
                Address = "10.202.16.179 "
                NetMask = "255.255.252.0 "
                )

        Proxy nic_proxy (
                Critical = 0
                TargetResName = nic
                )

        web_ip requires nic_proxy
        webvol7 requires webdg


        // resource dependency tree
        //
        //      group webserver
        //      {
        //      IPMultiNICB web_ip
        //          {
        //          Proxy nic_proxy
        //          }
        //      CFSMount webmnt8
        //      CFSMount webvol7
        //          {
        //          CVMVolDg webdg
        //          }
        //      }


Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

jstucki
Level 4

Sometimes, using the VCS GUI, its difficult to see when you've added an extra space character in an attribute for a resource.  There are several extra space characters in the above configs, which should be removed:

MountPoint = "/appl2 "

BlockDevice = "/dev/vx/dsk/webdg/webvol8 "

BlockDevice = "/dev/vx/dsk/webdg/webvol7 "

CVMVolume = { "webvol7 ", webvol8 }

I saw 4 more of them.  And it appeared also that some of the CFSMount resources weren't dependent on the CVMVolDg resource.

View solution in original post

5 REPLIES 5

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have you tried to import the dg and mount the volume manually?

mikebounds
Level 6
Partner Accredited

You don't need the""cluster" option in the MountOpt attribute of the CFSMount resource as the agent add this in for you, hence why you see cluster option being used twice in the logs - " -o cluster,cluster" - this is probably causing you your issue.

Mike

SK_Ram
Level 3
Accredited

Hi Mike,

 

The same config works for the rest of my service groups. I'm stuck on this as this is the first time I'm having this issue. Please see the following for my Oracle resources.

======================================================================

2011/11/21 16:43:26 VCS INFO V-16-20011-5506 (sun11) CFSMount:ora_r02_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster,mntlock=VCS                Block Device /dev/vx/dsk/oradg/oravol3 : MountPoint /data/oradata/r02
2011/11/21 16:43:26 VCS INFO V-16-20011-5506 (sun11) CFSMount:ora_r01_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,cluster,mntlock=VCS                Block Device /dev/vx/dsk/oradg/oravol2 : MountPoint /data/oradata/r01

...
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_arch_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_bkp_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_r02_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS INFO V-16-1-10298 Resource ora_r01_mnt (Owner: Unspecified, Group: Oracle) is online on sun11 (VCS initiated)
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_smprod (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_ecqrp (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_ecprd (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource Oracle_autosys (Owner: Unspecified, Group: Oracle) on System sun11
2011/11/21 16:43:45 VCS NOTICE V-16-20002-21 (sun11) Oracle:Oracle_autosys:online:Oracle startup returned the output
+--------------------------------------------------------------------+
LD_LIBRARY_PATH - /usr/lib:/opt1/sw/oracle/product/11.2.0/db/lib:
ORACLE instance started.

Total System Global Area  522092544 bytes
Fixed Size                  2149672 bytes
Variable Size             398463704 bytes
Database Buffers          113246208 bytes
Redo Buffers                8232960 bytes
Database mounted.
Database opened.

=================================================
main.cf

group Oracle (
        SystemList = { sun11 = 0, sun41 = 1 }
        AutoStartList = { sun11, sun41 }
        )

        CFSMount ora_arch_mnt (
                Critical = 0
                MountPoint = "/data/oradata/archive"
                BlockDevice = "/dev/vx/dsk/oradg/oravol4"
                MountOpt = "cluster"
                )

        CFSMount ora_bkp_mnt (
                Critical = 0
                MountPoint = "/data/oradata/backup"
                BlockDevice = "/dev/vx/dsk/oradg/oravol5"
                MountOpt = "cluster"
                )

 

....

 

group vrts_vea_cfs_int_cfsmount2 (
        SystemList = { sun11 = 0, sun41 = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { sun11, sun41 }
        )

CFSMount cfsmount5 (
                Critical = 0
                MountPoint = "/data/oradata/archive"
                BlockDevice = "/dev/vx/dsk/oradg/oravol4"
                MountOpt @sun11 = "cluster"
                MountOpt @sun41 = "cluster"
                NodeList = { sun11, sun41 }
                )

        CFSMount cfsmount6 (
                Critical = 0
                MountPoint = "/data/oradata/backup"
                BlockDevice = "/dev/vx/dsk/oradg/oravol5"
                MountOpt @sun11 = "cluster"
                MountOpt @sun41 = "cluster"
                NodeList = { sun11, sun41 }
                )

=================================

Anyway I'll try to remove the option and see if it works.
 

jstucki
Level 4

Sometimes, using the VCS GUI, its difficult to see when you've added an extra space character in an attribute for a resource.  There are several extra space characters in the above configs, which should be removed:

MountPoint = "/appl2 "

BlockDevice = "/dev/vx/dsk/webdg/webvol8 "

BlockDevice = "/dev/vx/dsk/webdg/webvol7 "

CVMVolume = { "webvol7 ", webvol8 }

I saw 4 more of them.  And it appeared also that some of the CFSMount resources weren't dependent on the CVMVolDg resource.

Anoop_Kumar1
Level 5

CFSMount webmnt8 (
                Critical = 0
                MountPoint = "/appl2 "
                BlockDevice = "/dev/vx/dsk/webdg/webvol8 "
                MountOpt = "cluster"
                )

 

 

2011/11/25 10:47:31 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt7:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster ,mntlock=VCS                 Block Device /dev/vx/dsk/webdg/webvol7  : MountPoint /opt2/sw
2011/11/25 10:47:31 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt7:online:Mount Error : Usage:
mount [-v | %-p]
mount [-F FSType] [-V] [current_options] [-o specific_options]
        {special | mount_point}
mount [-F FSType] [-V] [current_options] [-o specific_options]
        special mount_point
mount %-a [-F FSType ] [-V] [current_options] [-o specific_options]
        [mount_point ...]
2011/11/25 10:47:36 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group webserver on all nodes
 
 
- Abov errors are due to improper options selected to start the CFSMount resource. Please check bundled agent guide for required attribute and options to use it.
 
 
2011/11/25 10:47:36 VCS NOTICE V-16-1-10301 Initiating Online of Resource webmnt8 (Owner: Unspecified, Group: webserver) on System sun11
2011/11/25 10:47:37 VCS WARNING V-16-20011-5502 (sun11) CFSMount:webmnt8:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,cluster,mntlock=VCS                 Block Device /dev/vx/dsk/webdg/webvol8  : MountPoint /appl2
2011/11/25 10:47:37 VCS WARNING V-16-20011-5508 (sun11) CFSMount:webmnt8:online:Mount Error : UX:vxfs mount: ERROR: V-3-21264: /dev/vx/dsk/webdg/webvol8 is already mounted, /appl2 is busy,
                 or the allowable number of mount points has been exceeded.
                 
                 
 
-  Above error could be due to /appl2 directory. You can remove and recreate this directory to make mount successful.