Forum Discussion

Eric_Gao's avatar
Eric_Gao
Level 4
16 years ago

problem with cvm/cfs

After install license properly, I began the testing of cvm/cfs.

 

I had already created a shared disk group, and activation mode was set to sw (shared write),  while I tried to create the vxfs file system, it always reported error "cluster reservation failed for volume"

 

Does anyone know something about this?  thanks.

 

slave:/ #vxdg list mydg
Group:     mydg
dgid:      1168495239.16.slave
import-id: 33792.8
flags:     shared cds                                          ---> shared dg
version:   140
alignment: 8192 (bytes)
local-activation: shared-write
cluster-actv-modes: slave=sw test01=sw              ----> shared write
ssb:            on
autotagging:    on
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=default nlog=default
config:    seqno=0.1052 permlen=48144 free=48139 templen=3 loglen=7296
config disk c2t1d0s2 copy 1 len=48144 state=clean online
config disk c2t2d0s2 copy 1 len=48144 state=clean online
config disk c2t4d0s2 copy 1 len=48144 state=clean online
log disk c2t1d0s2 copy 1 len=7296
log disk c2t2d0s2 copy 1 len=7296
log disk c2t4d0s2 copy 1 len=7296
slave:/ #
slave:/ #mkfs -F vxfs /dev/vx/rdsk/mydg/vol01
UX:vxfs mkfs: ERROR: V-3-21018: cluster reservation failed for volume /dev/vx/rdsk/mydg/vol01
slave:/ #

  • problem is resolved.

     

    Basically this is because the partial installation of cluster file system.  Error "cluster reservation failed " is due to the abasence of glm module (group lock manager).  After I noticed that VRTSglm is missing, I first tried manuall installation of this package but still couldn't open port f, that being used by cfs.

     

    I deleted everything and re-installed storage foundation cluster file system and all went well afterwards.

     

    Also as I am using 3 iscsi disks (they are created from one server of the cluster, so basically there is no real shared storeage),  with dmp enabled, all 3 iscsi disks were probed into one disk with 3 paths,  that is becuase the solaris soft iscsi disks don't provide serial number, as a workaround, I tried to disable dmp but failed to do so. 

     

    After applied MP3, this bug was resolved and after disabled dmp, 3 iscsi disks were presented to vxvm properly.

     

    Now I got a cluster shared storage (3 iscsi disks actually), running on top of two server  slave (master), test01  (slave)

     

    slave:/ #vxdctl -c mode
    mode: enabled: cluster active - MASTER
    master: slave
    slave:/ #df -h | grep shared
    /dev/vx/dsk/mydg/vol01   2.0G    34M   1.8G     2%    /sharedfs
    slave:/ #
    slave:/ #
    slave:/ #
    slave:/ #vxdctl -c mode
    mode: enabled: cluster active - MASTER
    master: slave
    slave:/ #
    slave:/ #vxdg list
    NAME         STATE           ID
    mydg         enabled,shared,cds   1168567479.12.slave
    slave:/ #vxdg list mydg
    Group:     mydg
    dgid:      1168567479.12.slave
    import-id: 33792.8
    flags:     shared cds
    version:   140
    alignment: 8192 (bytes)
    local-activation: shared-write
    cluster-actv-modes: slave=sw test01=sw
    ssb:            on
    autotagging:    on
    detach-policy: global
    dg-fail-policy: dgdisable
    copies:    nconfig=default nlog=default
    config:    seqno=0.1044 permlen=48144 free=48140 templen=3 loglen=7296
    config disk c2t1d0s2 copy 1 len=48144 state=clean online
    log disk c2t1d0s2 copy 1 len=7296
    slave:/ #
    slave:/ #df -h
    Filesystem             size   used  avail capacity  Mounted on
    /dev/dsk/c0t1d0s0       19G   9.0G    10G    48%    /
    /devices                 0K     0K     0K     0%    /devices
    ctfs                     0K     0K     0K     0%    /system/contract
    proc                     0K     0K     0K     0%    /proc
    mnttab                   0K     0K     0K     0%    /etc/mnttab
    swap                   4.6G   1.6M   4.6G     1%    /etc/svc/volatile
    objfs                    0K     0K     0K     0%    /system/object
    fd                       0K     0K     0K     0%    /dev/fd
    swap                   4.6G   120K   4.6G     1%    /tmp
    swap                   4.6G    32K   4.6G     1%    /var/run
    swap                   4.6G     0K   4.6G     0%    /dev/vx/dmp
    swap                   4.6G     0K   4.6G     0%    /dev/vx/rdmp
    /dev/dsk/c0t1d0s3       19G   6.0G    13G    32%    /data
    /dev/odm                 0K     0K     0K     0%    /dev/odm
    /dev/vx/dsk/mydg/vol01
                           2.0G    34M   1.8G     2%    /sharedfs
    slave:/ #grep shared /etc/mnttab
    /dev/vx/dsk/mydg/vol01  /sharedfs       vxfs    rw,suid,delaylog,largefiles,qio,cluster,ioerror=mdisable,crw,dev=4f8b3b0        1168663491
    slave:/ #

     

    test01:/ #vxdg list
    NAME         STATE           ID
    mydg         enabled,shared,cds   1168567479.12.slave
    test01:/ #vxdg list mydg
    Group:     mydg
    dgid:      1168567479.12.slave
    import-id: 33792.8
    flags:     shared cds
    version:   140
    alignment: 8192 (bytes)
    local-activation: shared-write
    cluster-actv-modes: slave=sw test01=sw
    ssb:            on
    autotagging:    on
    detach-policy: global
    dg-fail-policy: dgdisable
    copies:    nconfig=default nlog=default
    config:    seqno=0.1044 permlen=0 free=0 templen=0 loglen=0
    test01:/ #
    test01:/ #df -h
    Filesystem             size   used  avail capacity  Mounted on
    /dev/dsk/c0t2d0s0       19G   6.2G    13G    33%    /
    /devices                 0K     0K     0K     0%    /devices
    ctfs                     0K     0K     0K     0%    /system/contract
    proc                     0K     0K     0K     0%    /proc
    mnttab                   0K     0K     0K     0%    /etc/mnttab
    swap                   4.3G   1.6M   4.3G     1%    /etc/svc/volatile
    objfs                    0K     0K     0K     0%    /system/object
    fd                       0K     0K     0K     0%    /dev/fd
    swap                   4.3G   120K   4.3G     1%    /tmp
    swap                   4.3G    32K   4.3G     1%    /var/run
    swap                   4.3G     0K   4.3G     0%    /dev/vx/dmp
    swap                   4.3G     0K   4.3G     0%    /dev/vx/rdmp
    /dev/odm                 0K     0K     0K     0%    /dev/odm
    /dev/vx/dsk/mydg/vol01
                           2.0G    34M   1.8G     2%    /sharedfs
    test01:/ #grep sharedfs /etc/mnttab
    /dev/vx/dsk/mydg/vol01  /sharedfs       vxfs    rw,suid,delaylog,largefiles,qio,cluster,ioerror=mdisable,crw,dev=4f8b3b0        1168663180
    test01:/ #

     

     

  • Hi,

     

    you can only create vx-objectson the master-node:

     

    Try this command:

    # vxdctl -c mode
    mode: enabled: cluster active - SLAVE
    master: <SYS>

     

    Greets

    Bjoern

     

  • yes of course I am operating on master node.

     

    my cluster consists of two servers, slave and test01.  slave is master and test01 is slave.

    Message Edited by Eric Gao on 02-24-2009 05:17 AM
  • problem is resolved.

     

    Basically this is because the partial installation of cluster file system.  Error "cluster reservation failed " is due to the abasence of glm module (group lock manager).  After I noticed that VRTSglm is missing, I first tried manuall installation of this package but still couldn't open port f, that being used by cfs.

     

    I deleted everything and re-installed storage foundation cluster file system and all went well afterwards.

     

    Also as I am using 3 iscsi disks (they are created from one server of the cluster, so basically there is no real shared storeage),  with dmp enabled, all 3 iscsi disks were probed into one disk with 3 paths,  that is becuase the solaris soft iscsi disks don't provide serial number, as a workaround, I tried to disable dmp but failed to do so. 

     

    After applied MP3, this bug was resolved and after disabled dmp, 3 iscsi disks were presented to vxvm properly.

     

    Now I got a cluster shared storage (3 iscsi disks actually), running on top of two server  slave (master), test01  (slave)

     

    slave:/ #vxdctl -c mode
    mode: enabled: cluster active - MASTER
    master: slave
    slave:/ #df -h | grep shared
    /dev/vx/dsk/mydg/vol01   2.0G    34M   1.8G     2%    /sharedfs
    slave:/ #
    slave:/ #
    slave:/ #
    slave:/ #vxdctl -c mode
    mode: enabled: cluster active - MASTER
    master: slave
    slave:/ #
    slave:/ #vxdg list
    NAME         STATE           ID
    mydg         enabled,shared,cds   1168567479.12.slave
    slave:/ #vxdg list mydg
    Group:     mydg
    dgid:      1168567479.12.slave
    import-id: 33792.8
    flags:     shared cds
    version:   140
    alignment: 8192 (bytes)
    local-activation: shared-write
    cluster-actv-modes: slave=sw test01=sw
    ssb:            on
    autotagging:    on
    detach-policy: global
    dg-fail-policy: dgdisable
    copies:    nconfig=default nlog=default
    config:    seqno=0.1044 permlen=48144 free=48140 templen=3 loglen=7296
    config disk c2t1d0s2 copy 1 len=48144 state=clean online
    log disk c2t1d0s2 copy 1 len=7296
    slave:/ #
    slave:/ #df -h
    Filesystem             size   used  avail capacity  Mounted on
    /dev/dsk/c0t1d0s0       19G   9.0G    10G    48%    /
    /devices                 0K     0K     0K     0%    /devices
    ctfs                     0K     0K     0K     0%    /system/contract
    proc                     0K     0K     0K     0%    /proc
    mnttab                   0K     0K     0K     0%    /etc/mnttab
    swap                   4.6G   1.6M   4.6G     1%    /etc/svc/volatile
    objfs                    0K     0K     0K     0%    /system/object
    fd                       0K     0K     0K     0%    /dev/fd
    swap                   4.6G   120K   4.6G     1%    /tmp
    swap                   4.6G    32K   4.6G     1%    /var/run
    swap                   4.6G     0K   4.6G     0%    /dev/vx/dmp
    swap                   4.6G     0K   4.6G     0%    /dev/vx/rdmp
    /dev/dsk/c0t1d0s3       19G   6.0G    13G    32%    /data
    /dev/odm                 0K     0K     0K     0%    /dev/odm
    /dev/vx/dsk/mydg/vol01
                           2.0G    34M   1.8G     2%    /sharedfs
    slave:/ #grep shared /etc/mnttab
    /dev/vx/dsk/mydg/vol01  /sharedfs       vxfs    rw,suid,delaylog,largefiles,qio,cluster,ioerror=mdisable,crw,dev=4f8b3b0        1168663491
    slave:/ #

     

    test01:/ #vxdg list
    NAME         STATE           ID
    mydg         enabled,shared,cds   1168567479.12.slave
    test01:/ #vxdg list mydg
    Group:     mydg
    dgid:      1168567479.12.slave
    import-id: 33792.8
    flags:     shared cds
    version:   140
    alignment: 8192 (bytes)
    local-activation: shared-write
    cluster-actv-modes: slave=sw test01=sw
    ssb:            on
    autotagging:    on
    detach-policy: global
    dg-fail-policy: dgdisable
    copies:    nconfig=default nlog=default
    config:    seqno=0.1044 permlen=0 free=0 templen=0 loglen=0
    test01:/ #
    test01:/ #df -h
    Filesystem             size   used  avail capacity  Mounted on
    /dev/dsk/c0t2d0s0       19G   6.2G    13G    33%    /
    /devices                 0K     0K     0K     0%    /devices
    ctfs                     0K     0K     0K     0%    /system/contract
    proc                     0K     0K     0K     0%    /proc
    mnttab                   0K     0K     0K     0%    /etc/mnttab
    swap                   4.3G   1.6M   4.3G     1%    /etc/svc/volatile
    objfs                    0K     0K     0K     0%    /system/object
    fd                       0K     0K     0K     0%    /dev/fd
    swap                   4.3G   120K   4.3G     1%    /tmp
    swap                   4.3G    32K   4.3G     1%    /var/run
    swap                   4.3G     0K   4.3G     0%    /dev/vx/dmp
    swap                   4.3G     0K   4.3G     0%    /dev/vx/rdmp
    /dev/odm                 0K     0K     0K     0%    /dev/odm
    /dev/vx/dsk/mydg/vol01
                           2.0G    34M   1.8G     2%    /sharedfs
    test01:/ #grep sharedfs /etc/mnttab
    /dev/vx/dsk/mydg/vol01  /sharedfs       vxfs    rw,suid,delaylog,largefiles,qio,cluster,ioerror=mdisable,crw,dev=4f8b3b0        1168663180
    test01:/ #