VxVM vxdg ERROR V-5-1-585 Unable to create shared disk group

This message is documented on SORT: https://sort.symantec.com/ecls/umi/V-5-1-585. 

Moving this post to the Storage Foundation forum for further discussion. 

Bruce,

Can you explain what process you went through to install and configure the product up to this point?  Did you use the "installsfrac" utility?  And did you use the "installsfrac -fencing" command (as described in the installation guide) to configure the fencing for you?  The reason I ask is that it appears that your fencing disks are imported on node 1.  They shouldn't be imported on any node.  They should be deported on both nodes. 

Did you perform the steps to test your fencing disks, by first identifying them by SCSI serial number (vxfenadm -i /dev/rdsk/c1t1d0s2) on both systems, and then using the vxfentsthdw utility?  Before trying to create shared disk groups, you should have the CFS/CVM/VCS products up and running, after having used the "installsfrac" utility to take care of getting I/O Fencing configured properly, and also getting the base products of CFS/CVM/VCS up and running properly, so that you can begin to create shared disk groups.

It doesn't appear that your configuration is ready to begin creating shared disk groups.  If this is your first time installing this product, read completely through the installation guide (maybe 2 or 3 times.....its a bit confusing the first time reading through it, and trying to understand what needs to happen when), and post any questions you have on this forum.  Get your questions answered before beginning the installation process.

-John

Bruce,

Are you able to create a private disk group using the same devce?

If so you will want to then check to see if the cvm service group is online.

Can you post the ouput of the following command:

gabconfig -a

Joe D

Hi,

Steps to configure I/O Fencing:

node1 * / # vxdisk -e list

DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
disk_0       auto:none      -            -           online invalid       c1t3d0s2         -
disk_1       auto:none      -            -           online invalid       c1t2d0s2         -
disk_2       auto:none      -            -           online invalid       c3t6d0s2         -
disk_3       auto:none      -            -           online invalid       c3t7d0s2         -
disk_4       auto:none      -            -           online invalid       c3t0d0s2         -
disk_5       auto:none      -            -           online invalid       c3t5d0s2         -
disk_6       auto:none      -            -           online invalid       c3t3d0s2         -
disk_7       auto:none      -            -           online invalid       c3t1d0s2         -
disk_8       auto:none      -            -           online invalid       c3t2d0s2         -
disk_9       auto:none      -            -           online invalid       c3t4d0s2         -
disk_10      auto:none      -            -           online invalid       c1t1d0s2         -
disk_11      auto:ZFS       -            -           ZFS                  c1t0d0s2         -

node2 * / # vxdisk -e list

DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
disk_0       auto:none      -            -           online invalid       c1t3d0s2         -
disk_1       auto:none      -            -           online invalid       c1t1d0s2         -
disk_2       auto:ZFS       -            -           ZFS                  c1t0d0s2         -
disk_3       auto:none      -            -           online invalid       c3t6d0s2         -
disk_4       auto:none      -            -           online invalid       c3t7d0s2         -
disk_5       auto:none      -            -           online invalid       c3t0d0s2         -
disk_6       auto:none      -            -           online invalid       c3t5d0s2         -
disk_7       auto:none      -            -           online invalid       c3t3d0s2         -
disk_8       auto:none      -            -           online invalid       c3t1d0s2         -
disk_9       auto:none      -            -           online invalid       c3t2d0s2         -
disk_10      auto:none      -            -           online invalid       c3t4d0s2         -
disk_11      auto:none      -            -           online invalid       c1t2d0s2         -

On node1, disk_3, disk_4 & disk_5 were used as coordinator disks:

node1 * / # vxdisk -f init disk_3
node1 * / # vxdisk -f init disk_4
node1 * / # vxdisk -f init disk_5

Steps to configure I/O Fencing:

node1 * / # vxfenadm -i /dev/rdsk/c3t6d0s2
Vendor id       : SEAGATE
Product id      : ST330055FSUN300G
Revision        : 3192
Serial Number   : 2000001D3874D6D0

node2 * / # vxfenadm -i /dev/rdsk/c3t6d0s2
Vendor id       : SEAGATE
Product id      : ST330055FSUN300G
Revision        : 3192
Serial Number   : 2000001D3874D6D0

node1 * / # vxfenadm -i /dev/rdsk/c3t7d0s2
Vendor id       : SEAGATE
Product id      : ST330055FSUN300G
Revision        : 3093
Serial Number   : 2000001D38AC4B15

node2 * / # vxfenadm -i /dev/rdsk/c3t7d0s2
Vendor id       : SEAGATE
Product id      : ST330055FSUN300G
Revision        : 3093
Serial Number   : 2000001D38AC4B15

node1 * / # vxfenadm -i /dev/rdsk/c3t0d0s2
Vendor id       : SEAGATE
Product id      : ST330057FSUN300G
Revision        : 0605
Serial Number   : 2000001D38FCD64A

node2 * / # vxfenadm -i /dev/rdsk/c3t0d0s2
Vendor id       : SEAGATE
Product id      : ST330057FSUN300G
Revision        : 0605
Serial Number   : 2000001D38FCD64A


node1 * / # vxdctl -c mode
mode: enabled: cluster active - MASTER
master: node1

node1 * / # cfscluster status

  Node             :  node1
  Cluster Manager  :  running
  CVM state        :  running
  No mount point registered with cluster configuration


  Node             :  node2
  Cluster Manager  :  running
  CVM state        :  running
  No mount point registered with cluster configuration

node1 * / # hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  node1                RUNNING              0
A  node2                RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  cvm             node1                Y          N               ONLINE
B  cvm             node2                Y          N               ONLINE

node1 * / # vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
disk_0       auto:none      -            -           online invalid       c1t3d0s2         -
disk_1       auto:none      -            -           online invalid       c1t2d0s2         -
disk_2       auto:none      -            -           online invalid       c3t6d0s2         -
disk_3       auto:cdsdisk   -            -           online               c3t7d0s2         -
disk_4       auto:cdsdisk   -            -           online               c3t0d0s2         -
disk_5       auto:cdsdisk   -            -           online               c3t5d0s2         -
disk_6       auto:cdsdisk   -            -           online               c3t3d0s2         -
disk_7       auto:none      -            -           online invalid       c3t1d0s2         -
disk_8       auto:none      -            -           online invalid       c3t2d0s2         -
disk_9       auto:none      -            -           online invalid       c3t4d0s2         -
disk_10      auto:none      -            -           online invalid       c1t1d0s2         -
disk_11      auto:ZFS       -            -           ZFS                  c1t0d0s2         -

node2 * / # vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
disk_0       auto:none      -            -           online invalid       c1t3d0s2         -
disk_1       auto:none      -            -           online invalid       c1t1d0s2         -
disk_2       auto:ZFS       -            -           ZFS                  c1t0d0s2         -
disk_3       auto:none      -            -           online invalid       c3t6d0s2         -
disk_4       auto:cdsdisk   -            -           online               c3t7d0s2         -
disk_5       auto:cdsdisk   -            -           online               c3t0d0s2         -
disk_6       auto:cdsdisk   -            -           online               c3t5d0s2         -
disk_7       auto:cdsdisk   -            -           online               c3t3d0s2         -
disk_8       auto:none      -            -           online invalid       c3t1d0s2         -
disk_9       auto:none      -            -           online invalid       c3t2d0s2         -
disk_10      auto:none      -            -           online invalid       c3t4d0s2         -
disk_11      auto:none      -            -           online invalid       c1t2d0s2         -

node1 * / # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   30ae02 membership 01
Port b gen   30ae15 membership 01
Port d gen   30ae06 membership 01
Port f gen   30ae22 membership 01
Port h gen   30ae18 membership 01
Port o gen   30ae01 membership 01
Port u gen   30ae1f membership 01
Port v gen   30ae1b membership 01
Port w gen   30ae1d membership 01
Port y gen   30ae1a membership 01

node2 * / # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   30ae02 membership 01
Port b gen   30ae15 membership 01
Port d gen   30ae06 membership 01
Port f gen   30ae22 membership 01
Port h gen   30ae18 membership 01
Port o gen   30ae01 membership 01
Port u gen   30ae1f membership 01
Port v gen   30ae1b membership 01
Port w gen   30ae1d membership 01
Port y gen   30ae1a membership 01
 

Following is the error:

node1 * / # vxdg -s init oradg oradg01=disk_6

VxVM vxdg ERROR V-5-1-585 Disk group oradg: cannot create: Record not in disk group

Regards,

Bruce

Hi,

In continuation to my previous mail, the external storage is a StorageTek 6140 JBOD and the fibre channel connectivity exists from the Sun Fire v440 servers to the array.

I also see the following messages:

May 23 11:38:52 node1 gab: [ID 272960 kern.notice] GAB INFO V-15-1-20036 Port a[GAB_Control (refcount 1)] gen   30ae02 memb                                  ership 01
May 23 11:38:52 node1 gab: [ID 828938 kern.notice] GAB INFO V-15-1-20037 Port a[GAB_Control (refcount 1)] gen   30ae02   je                                  opardy ;1
May 23 11:39:09 node1 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-5759 oradg: dg create with I/O fence enabled
May 23 11:39:09 node1 vxvm:vxconfigd: [ID 702911 daemon.notice]
May 23 11:39:09 node1 llt: [ID 602713 kern.notice] LLT INFO V-14-1-10023 lost 66 hb seq 39694 from 1 link 0 (ce1)
May 23 11:39:10 node1 scsi: [ID 107833 kern.warning] WARNING: /pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w2200001d38fcdbcf,0 (ssd                                  10):
May 23 11:39:10 node1   reservation conflict
  jeopardy ;1
May 23 11:39:33 node1 Had[5772]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10111 System node2 (Node '1') is in Regular and                                   Jeopardy Memberships - Membership: 0x3, Jeopardy: 0x2
May 23 11:39:33 node1 gab: [ID 272960 kern.notice] GAB INFO V-15-1-20036 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen   30ae1                                  a membership 01
May 23 11:39:33 node1 gab: [ID 828938 kern.notice] GAB INFO V-15-1-20037 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen   30ae1                                  a   jeopardy ;1
May 23 11:39:33 node1 gab: [ID 272960 kern.notice] GAB INFO V-15-1-20036 Port o[VCSMM (refcount 2)] gen   30ae01 membership                                   01
May 23 11:39:33 node1 gab: [ID 828938 kern.notice] GAB INFO V-15-1-20037 Port o[VCSMM (refcount 2)] gen   30ae01   jeopardy                                   ;1
May 23 11:39:33 node1 gab: [ID 272960 kern.notice] GAB INFO V-15-1-20036 Port u[GAB_USER_CLIENT (refcount 0)] gen   30ae1f                                   membership 01
May 23 11:39:33 node1 gab: [ID 828938 kern.notice] GAB INFO V-15-1-20037 Port u[GAB_USER_CLIENT (refcount 0)] gen   30ae1f                                     jeopardy ;1
May 23 11:39:56 node1 llt: [ID 602713 kern.notice] LLT INFO V-14-1-10023 lost 95 hb seq 40582 from 1 link 0 (ce1)
May 23 11:40:31 node1 llt: [ID 602713 kern.notice] LLT INFO V-14-1-10023 lost 700 hb seq 41283 from 1 link 0 (ce1)
May 23 11:40:36 node1 llt: [ID 602713 kern.notice] LLT INFO V-14-1-10023 lost 98 hb seq 41382 from 1 link 0 (ce1)
May 23 11:40:36 node1 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 0 (ce1) node 1 active
May 23 11:40:38 node1 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (ce1) node 1 in trouble

Regards,

Bruce

Hi,

main.cf file:

node1 * / # cat /etc/VRTSvcs/conf/config/main.cf
include "OracleASMTypes.cf"
include "types.cf"
include "CFSTypes.cf"
include "CRSResource.cf"
include "CVMTypes.cf"
include "Db2udbTypes.cf"
include "MultiPrivNIC.cf"
include "OracleTypes.cf"
include "PrivNIC.cf"
include "SybaseTypes.cf"

cluster vcs (
        UserNames = { admin = dlmElgLimHmmKumGlj }
        Administrators = { admin }
        UseFence = SCSI3
        HacliUserLevel = COMMANDROOT
        )

system node1 (
        )

system node2 (
        )

group cvm (
        SystemList = { node1 = 0, node2 = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { node1, node2 }
        )

        CFSfsckd vxfsckd (
                )

        CVMCluster cvm_clus (
                CVMClustName = vcs
                CVMNodeId = { node1 = 0, node2 = 1 }
                CVMTransport = gab
                CVMTimeout = 200
                )

        CVMVxconfigd cvm_vxconfigd (
                Critical = 0
                CVMVxconfigdArgs = { syslog }
                )

        cvm_clus requires cvm_vxconfigd
        vxfsckd requires cvm_clus


        // resource dependency tree
        //
        //      group cvm
        //      {
        //      CFSfsckd vxfsckd
        //          {
        //          CVMCluster cvm_clus
        //              {
        //              CVMVxconfigd cvm_vxconfigd
        //              }
        //          }
        //      }
 

Regards,

Bruce

This is bit strange ... you are on the master node & still are unable to create the diskgroup ... since CVM is online, I am assuming that all the licenses are intact ...

By the way what is product version & OS version ?

I may think of following doubts ..

1. Coordinator disks have any pre-existing keys !  Try using a "vxfenclearpre" to clear pre-existing keys.

2 . Did u check "vxfentsthdw" utility & confirmed that all the tests are passed ?

3. coordinator dg should be deported on all the nodes ... not sure at this time whether you have any keys on coordinator disks or no..

4. were the disks presented here were having preexisting vxvm config ? is this setup an upgraded setup ?

can u paste following:

# cat /etc/vxfentab

# cat /etc/vxfenmode

# modinfo |egrep 'llt|gab|vxfen'

# /sbin/vxfenadm -g all -f /etc/vxfentab

also paste output of vxfentsthdw utility ..

G

Hi,

This is a fresh installation. The OS is Solaris 10 Update 10 SPARC 64-bit and the cluster software is 6.0. The vxfentsthdw utility was run and all the disks succeeded the test. Regarding the co-ordinator diskgroup, it is deported from both the nodes and was not imported during shared diskgroup creation. The disks were having old vxvm configuration. After a fresh installation of OS, the vxdiskunetup -C command was run and the disks were re-initialzed.

node1 * / # cat /etc/vxfentab
#
# /etc/vxfentab:
# DO NOT MODIFY this file as it is generated by the
# VXFEN rc script from the file /etc/vxfendg.
#
/dev/rdsk/c3t7d0s2
/dev/rdsk/c2t7d0s2
/dev/rdsk/c3t0d0s2
/dev/rdsk/c2t0d0s2
/dev/rdsk/c3t5d0s2
/dev/rdsk/c2t5d0s2

node1 * / # cat /etc/vxfenmode
#
# vxfen_mode determines in what mode VCS I/O Fencing should work.
#
# available options:
# scsi3      - use scsi3 persistent reservation disks
# customized - use script based customized fencing
# disabled   - run the driver but don't do any actual fencing
#
vxfen_mode=scsi3

#
# scsi3_disk_policy determines the way in which I/O Fencing communicates with
# the coordination disks.
#
# available options:
#       dmp - use dynamic multipathing
#       raw - connect to disks using the native interface
#
# Although  vxfen  supports  raw  for scsi3_disk_policy, it is now deprecated.
# It is recommended to use dmp as the scsi3_disk_policy for coordinator disks.

scsi3_disk_policy=raw
node1 * / # modinfo |egrep 'llt|gab|vxfen'
234 7af02000  31368 344   1  llt (LLT 6.0)
235 7af2a000  65e18 345   1  gab (GAB device 6.0)
236 7ae70000  79758 346   1  vxfen (VRTS Fence 6.0)

node1 * / # /sbin/vxfenadm -g all -f /etc/vxfentab
VXFEN vxfenadm WARNING V-11-2-2414 This option is deprecated and would be removed with the next release.
Please use the -s option.

Device Name: /dev/rdsk/c3t0d0s2
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[1]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[2]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001
key[3]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001

Device Name: /dev/rdsk/c2t7d0s2
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[1]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[2]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001
key[3]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001

Device Name: /dev/rdsk/c2t5d0s2
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[1]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[2]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001
key[3]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001

Device Name: /dev/rdsk/c2t0d0s2
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[1]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[2]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001
key[3]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001

Device Name: /dev/rdsk/c3t7d0s2
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[1]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[2]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001
key[3]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001

Device Name: /dev/rdsk/c3t5d0s2
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[1]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,48
        Key Value [Character Format]: VF000000
key[2]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001
key[3]:
        Key Value [Numeric Format]:  86,70,48,48,48,48,48,49
        Key Value [Character Format]: VF000001

Regards,

Bruce

ok keys are looking fine ....

can u paste

# vxdisk -o alldgs list

# ps -ef  |grep -i vxconfigd

# vxdg list

# vxdg list oradg

# vxlicrep -e

try initiaing a diskgroup with other name .... lets say testdg (error is appearing for oradg) ... also try creating a local diskgroup without -s option & share the results ..

G

I would check you can create a private non-shared diskgroup first:

vxdg init oradg oradg01=disk_6

If this works, then deport diskgroup and try to import on second node to confirm both systems can access disk.

If above does not work, then issue has nothing to do with CVM, but if it does work, then deport diskgroup and try to import (on master node) as shared (effectively this converts a private diskgroup to a shared diskgroup):

vxdg -s import oradg

Mike