Hi Mike,
I did some testing and I read now your latest post. When usign VMware, something to do is to set the enableUUID flag to true. It is clear in your case it is not set because disk are listed as sdb instead of disk_0. Please set disk.EnableUUID to "TRUE" and give it another try.
I have been using that kind of environment very frequently with no issues. In fact, it is similar to what you used last week in the FSS lab in VISION. As you stated, there is no need for SCSI3 here and also no need for multi-write flag at all. But there is a need for the enableUUID.
I tried to reproduce the issue in my lab just in case, but I had no success. This is what I did:
Down is my CVM master node (this should be irrelevant, but just in case):
down~> vxdctl -c mode
mode: enabled: cluster active - MASTER
master: down
And I have a three node cluster:
down~> vxclustadm nidmap
Name CVM Nid CM Nid State
down 1 0 Joined: Master
strange 2 2 Joined: Slave
up 0 1 Joined: Slave
I am going to export one of the local disks for down server:
down~> vxdisk export disk_18
It is exported. Also I have visibility from local disks from other nodes:
down~> vxdisk list | grep disk_18
disk_18 auto:cdsdisk - - online exported
up_disk_18 auto:cdsdisk - - online remote
down~>
I am going to create a DG using only that disk as you did:
down~> vxdg -s -o fss init mikedg fd1_La=disk_18
down~>
Here the DG:
down~> vxdisk -g mikedg list
DEVICE TYPE DISK GROUP STATUS
disk_18 auto:cdsdisk fd1_La mikedg online exported shared
down~>
I also have the visibility from the up node:
up~> vxdisk -g mikedg list
DEVICE TYPE DISK GROUP STATUS
down_disk_18 auto:cdsdisk fd1_La mikedg online shared remote
up~>
Now I stop VCS on the up node:
up~> hastop –local
This is the current situation:
down~> vxclustadm nidmap
Name CVM Nid CM Nid State
down 1 0 Joined: Master
strange 2 2 Joined: Slave
up 0 1 Out of Cluster
down~>
Because up is now out of the cluster, it only has local storage visibility:
up~> vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk - - online exported shared
disk_1 auto:cdsdisk - - online exported shared
disk_2 auto:cdsdisk - - online exported shared
disk_3 auto:cdsdisk - - online exported shared
disk_4 auto:cdsdisk - - online exported shared
disk_5 auto:cdsdisk - - online exported shared
disk_6 auto:cdsdisk - - online exported shared
disk_7 auto:cdsdisk - - online exported shared
disk_8 auto:cdsdisk - - online exported shared
disk_9 auto:cdsdisk - - online exported
disk_10 auto:cdsdisk - - online exported
disk_11 auto:cdsdisk - - online exported
disk_12 auto:cdsdisk - - online exported
disk_13 auto:cdsdisk - - online exported
disk_14 auto:cdsdisk - - online exported
disk_15 auto:cdsdisk - - online exported
disk_16 auto:cdsdisk - - online exported
disk_17 auto:cdsdisk - - online exported
disk_18 auto:cdsdisk - - online exported
disk_19 auto:cdsdisk - - online exported
disk_20 auto:cdsdisk - - online exported
disk_21 auto:cdsdisk disk_21 gold02 online
disk_22 auto:cdsdisk disk_22 gold02 online
disk_24 auto:cdsdisk disk_24 gold02 online
fusionio0_0 auto:cdsdisk - - online ssdtrim exported
sda auto:none - - online invalid
up~>
Now we start the cluster again and the up node got the visibility for the mikedg again with no issues:
up~> vxdisk -g mikedg list
DEVICE TYPE DISK GROUP STATUS
down_disk_18 auto:cdsdisk fd1_La mikedg online shared remote
up~>
And I can create a volume with no issues:
up~> vxassist -g mikedg make vol1 100m
up~>
If you still can reproduce the issue after setting the UUID, please send me an email so we can collect some debug logs from vxconfigd