03-25-2015 06:05 AM
Hi All,
Recently we have upgraded cluster nodes to 5.1SP1RP4. After upgrade i am not able to add new disk in DG and getting below error. I noticed all the disks now have udid_mismatch flag which was not the case before upgrade.
root # vxdg -g IPREDODG adddisk emc8_0d71=emc8_0d71
VxVM vxdg ERROR V-5-1-0 Disk Group IPREDODG has only cloned disks and trying to add standard disk to diskgroup. Mix of standard and cloned disks in a diskgroup is not allowed. Please follow the vxdg (1M) man page.
I found a doc which talks about similar issue in 5.1SP1RP3 .http://www.symantec.com/business/support/index?page=content&id=TECH204069
My question is that in my setup DGs are in cluster and we don't want to deport them. Can we update the udid online without outage?
Or can anyone please suggest any other solution which doesn't require downtime?
03-25-2015 09:02 AM
Hi,
No, you have to deport, clear UUID mismatch, and set CLONE OFF, then import
03-25-2015 10:34 AM
I don't think you can update udid online, but what you can do is add a clone disk and then you can sort clone flag out on your next scheduled outage - example shown below:
# vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sdc testdg online clone_disk sdf auto:cdsdisk - - online # vxdg -g testdg adddisk sdf VxVM vxdg ERROR V-5-1-18510 Device sdf is a non-clone disk. Adding a non-clone disk to a clone diskgroup is not allowed # vxdisk set sdf clone=on # vxdg -g testdg adddisk sdf # vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sdc testdg online clone_disk sdf auto:cdsdisk sdf testdg online clone_disk
MIke
03-25-2015 10:53 AM
Thinking outside the box, nice!
03-25-2015 11:01 PM
03-25-2015 11:19 PM
Below are the outputs for existing disk in DG and the new disk. My mistake in last comment new disk don't have udid_mismatch flag on them.
root # vxdisk -v list emc7_1999 | grep -i udid ----- existing disk in dg
flags: online ready private autoconfig udid_mismatch noautoimport
udid: EMC%5FSYMMETRIX%5F000290101186%5F8601999008
tag udid_asl=EMC%5FSYMMETRIX%5F101186%5F60060480000290101186533031393939
root# vxdisk -v list emc9_096c | grep -i udid --- new disk
udid: EMC%5FSYMMETRIX%5F000298700599%5F990096C000
tag udid_asl=EMC%5FSYMMETRIX%5F000298700599%5F990096C000
root # vxdisk -v list emc7_1999 | grep -i flag ----- existing disk in dg
flags: online ready private autoconfig udid_mismatch noautoimport
root # vxdisk -v list emc9_096c | grep -i flag --- new disk
flags: online ready private autoconfig autoimport thinrclm
root #
03-25-2015 11:33 PM
emc7_1999 has a mismatched UUID
03-25-2015 11:51 PM
Yes the existing disk in DG does have udid_mismatch.
03-26-2015 12:02 AM
So I would clear it using the process in the note.
03-26-2015 05:07 AM
On my vbox setup I am able to add disk with clone or udid_mismatch, but you can clear udid first also - see below:
# vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sdc testdg online clone_disk sdf auto:cdsdisk - - online udid_mismatch # vxdg -g testdg adddisk sdf # vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sdc testdg online clone_disk sdf auto:cdsdisk sdf testdg online udid_mismatch # vxdg -g testdg rmdisk sdf # vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sdc testdg online clone_disk sdf auto:cdsdisk - - online udid_mismatch # vxdisk updateudid sdf # vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sdc testdg online clone_disk sdf auto:cdsdisk - - online clone_disk # vxdg -g testdg adddisk sdf # vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sdc testdg online clone_disk sdf auto:cdsdisk sdf testdg online clone_disk
Mike
03-26-2015 05:39 AM
Mike,
Are you testing on Linux box? Will it behave the same for solaris? We did similar upgrade on linux box but don't see the udid_mismatch flag on disks in linux.
Can you try adding a disk which doesn't have udid_mismatch flag to a DG conatining udid_mismatch disks?
03-26-2015 10:01 AM
Yes I am doing this on Linux and it should in theory behave the same on Solaris, but udid doesn't work properly on vbox so can't be sure.
For adding a disk which doesn't have udid_mismatch flag to a DG conatining udid_mismatch:
# vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk - - online sdf auto:cdsdisk sdf testdg online udid_mismatch # vxdg -g testdg adddisk sde VxVM vxdg ERROR V-5-1-18510 Device sde is a non-clone disk. Adding a non-clone disk to a clone diskgroup is not allowed # # vxdisk set sde clone=on # vxdg -g testdg adddisk sde # vxdisk list DEVICE TYPE DISK GROUP STATUS sde auto:cdsdisk sde testdg online clone_disk sdf auto:cdsdisk sdf testdg online udid_mismatch #
As far as I can see, you have nothing to lose, by setting clone=on on new disk to see if this means you can add to diskgroup -i.e worse that can happen is it still won't let you add it to diskgroup
Mike
03-26-2015 11:10 PM
Mike,
We followed the steps of updating the udid and clone flags. One one node the the udid_mismatch flag is gone from all disks but on other node when we check vxdisk o/p the flag is still thr :( . I was searching for the error and found that you faced similar issue earlier on 6.1. Do let me know what you have done.
In the meantime i have created a support case also and the engineer also suggested the same steps. I've also updated him about the current situation.
03-27-2015 03:15 AM
I tested on 6.1 on RHEL 5 running in vbox. There is an issue with udid on vbox as the udid should be created from the serial number of the LUN obtained from the device layer, but on vbox, the LUN serial number obtained from the device layer is made up of hostname and O/S disk device name and hence is different between nodes so you get udid mismatch on one node and not the other which is what makes it easy for me to reproduce issue as if I run vxdisksetup on a disk on one system the other system sees udid mismatch.
You shouldn't have the "vbox" issue as your udid looks to be based on the WWN, but you can check udid yourself from the output of vxdisk -v list:
The value of udid is the value of the udid of the disk
The value of udid_asl (after word tag) is the udid written in the private region of the disk
Note it is more clear if you get udid from "vxdisk -p list" as then the fields are "UDID" and "PRIV_UDID"
So if the 2 match, then everything is fine, but if they do not match then you get udid_mismatch. The value of both of these values should be identical when viewed from other nodes for the same disk (this is where issue with vbox lies).
When you said you "We followed the steps of updating the udid and clone flags." is this my steps of adding clone disk to diskgroup while online, or the technote steps of deporting diskgroup and unsetting clone?
If you are seeing issues on one node and not the other then post output of:
vxdisk -p list problem_disk | egrep "UDID|SERIAL"
Mike
03-27-2015 05:11 AM
something weird is in the o/p . If you see below the udid is same in all case but on node02 you see the udid_mismatch flag while on node01 it's not.
root@node01 # vxdisk -p list emc7_1699 | grep -i "UDID"
UDID : EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node01 # vxdisk -v list emc7_1699 | grep -i udid
udid: EMC%5FSYMMETRIX%5F000290101186%5F8601699008
tag udid_asl=EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node01 #
root@node02 # vxdisk -p list emc7_1699 | grep -i "UDID"
UDID : EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node02 # vxdisk -v list emc7_1699 | grep -i udid
flags: online ready private autoconfig udid_mismatch noautoimport imported
udid: EMC%5FSYMMETRIX%5F000290101186%5F8601699008
tag udid_asl=EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node02 #
03-27-2015 06:19 AM
Try running "vxdisk -o alldgs list" on node that has wrong info as this reads the private region of the disks and updates the "in-memory" copy you see when you run "vxdisk list".
Mike
05-08-2015 04:27 AM
Hi Guys,
We finally were able to resolve the issue.
1) First we deported the DG and cleared the udid_mismatch as mentioned in the Technote.
2) Next I noticed that ssh was not working from one node to other(someone changed the ssh keys :( ) so sorted that. Not sure if that can be the reason for having the udid_mismatch flag on one node and not on other.
A clean reboot after that fixed the things.