vxdisk list showing errors on multiple disks, and I am unable to start cluster on slave node.

Just to explain, why `vxdg flush` worked and why the joiner node initially failed to join the node? i.e. solution provided by rsharma1.

Details:

As part of join protocols:

- CVM master send the list of online disks (from imported shared disk groups) that it can see to the joiner.

- Now joiner check whether it can see those disks or not. If not, it creates a dummy entry by fetching some basic properties of disks from other nodes of cluster.

Why node join failed in this case:

- There were some disks on master that are in online state but actually missing connectivity globally (all nodes of cluster). And hence were potential candidate to be detached (error state). But we don't proactively detach disks (for some valid reasons) and it will only get detached as part of I/Os on that disk.

- So now any joiner is expected to have connectivity to these disks as well (online on master) or the join will fail.

How `vxdg flush` solved this problem:

- vxdg flush triggers some private region I/Os on the disks, as part of refershing the contents.

- And as mentioned in previous these I/Os will detach the globally(all nodes of cluster) disconnected disks on master.

NOTE: We have handled some of these (likely) scenario proactively but not all.

Thanks & Regards,

Pankaj Tandon (CVM Team)

hello all

vxdg -g <disk_group> flush --> from CVM master node fixed my problem

all disks are visible after rescan and node was successfully joined to cluster.

Again, thanks for your time guys!

ok my bad ... -s came post 5.1

try

# vxfenadm -g all -f /tmp/diskfile    ( -g for registration keys)

# vxfenadm -r all -f /tmp/diskfile     (-r for reservation keys)

G

there is no such option in vxfenadm (-s) 

Well we can't say that IOFencing is cause, if reservation was an issue, even format would have problems reading the disks. can you get below outputs to confirm the IOFencing bit (below commands won't cause any harm, its just reading keys from disks)

Create a file with some error disks in it

# vxfenadm -s all -f /tmp/diskfile

# vxfenadm -r all -f /tmp/diskfile

G

I think that vxfen cause all my problems, because I can access to error disks via format command. I can see size od disk, layout, partitioning everything. But I have no idea how to confirm that vxfen is really causing all this problems.

To summarize, the issue here not CVM at this stage. As mentione above, for CVM cluster to work correctly, you would first need to have 70 shared disks on node filesvr1 as well, that too, disks should be accessible & not in error state. A disk in error state is indicating that somehow vxvm is unable to read out private region of the disk.

Problem here is 1 layer below, you need to find why the disks are in error state. First output which you gave in original post suggests that emc paths are enabled from DMP view. So here is what I would recommend

1. Confirm if Storage is alright, all 70 luns are intact & assigned to filesvr1.

2. verify SAN switches that the zoning is correct & filesvr1 should see 70 luns

3. OS is able to see all the disks correctly in format output, there is no disk in "drive not available" or any other inconsistent state. Any of disk which is in error state above, try to access that disk from "format" command & see if you can read the labels of the disk. "vxdisk -e list" command will help you map the emcpowerxx to c#t#d# names.

4. Verify powerpath can see the disks (powermt -display all) & paths are not dead

5. One more check you can do is on IOFencing. Do you have IOFencing on these disks ? are there any stale keys lying on these disks (vxfenadm -g <diskgroup> -s /dev/rdsk/cxtxdx), quite possible that fencing is restricting the IO access to these disks.

once above steps come clean, you might need to do a "vxdctl enable" for vxconfigd to establish connections with disk again. once vxdctl enable is executed, there is quite a possibility of you need to do a "vxreattach" of the disks to diskgroup however disks should be out of error state for that.

G

yes, vxdisk list see 36 disks in error and 34 are fine (online shared) , powermt display dev=all see all 70 disks without any problems (all paths are up)

solaris see all disks in format command.

I've attached output of commands

Are you able to see disks from OS. what is the o/p of # echo|format ? 

Are you able to see all the paths from EMC powerpath ?

19:28:36 [root@filesvr2:~]# vxdisk -o alldgs list |grep -i shared | wc -l
      70

19:28:52 [root@filesvr1:~]# vxdisk -o alldgs list |grep -i shared | wc -l
      36