cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to add a new disk to an existing disk group after upgrade to 5.1SP1RP4

SaGu
Level 3
Certified

Hi All,

 

Recently we have upgraded cluster nodes to 5.1SP1RP4. After upgrade i am not able to add new disk in DG and getting below error. I noticed all the disks now have udid_mismatch flag which was not the case before upgrade.

root #  vxdg -g IPREDODG adddisk emc8_0d71=emc8_0d71
VxVM vxdg ERROR V-5-1-0 Disk Group IPREDODG has only cloned disks and trying to add standard disk to diskgroup. Mix of standard and cloned disks in a diskgroup is not allowed. Please follow the vxdg (1M) man page.

I found a doc which talks about similar issue in 5.1SP1RP3 .http://www.symantec.com/business/support/index?page=content&id=TECH204069

My question is that in my setup DGs are in cluster and we don't want to deport them. Can we update the udid online without outage?

Or can anyone please suggest any other solution which doesn't require downtime?


 

16 REPLIES 16

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Hi,

 

No, you have to deport, clear UUID mismatch, and set CLONE OFF, then import

mikebounds
Level 6
Partner Accredited

I don't think you can update udid online, but what you can do is add a clone disk and then you can sort clone flag out on your next scheduled outage - example shown below:

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sdc          testdg       online clone_disk
sdf          auto:cdsdisk    -            -            online
# vxdg -g testdg adddisk sdf
VxVM vxdg ERROR V-5-1-18510 Device sdf is a non-clone disk.
Adding a non-clone disk to a clone diskgroup is not allowed
# vxdisk set sdf clone=on
# vxdg -g testdg adddisk sdf
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sdc          testdg       online clone_disk
sdf          auto:cdsdisk    sdf          testdg       online clone_disk

 

MIke

 

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Thinking outside the box, nice!

SaGu
Level 3
Certified
Mike & Riaan thanks for your replies! Mike , We are adding disks for storage migration. Even the disks from new storage has udid_mismatch flag but no "clone" flag. What can be the impact if we leave all disks with clone flag enabled? Also, Do we have to follow the same steps when adding the disks in coordinator DG ?

SaGu
Level 3
Certified

Below are the outputs for existing disk in DG and the new disk. My mistake in last comment new disk don't have udid_mismatch flag on them.

 

root # vxdisk -v list emc7_1999 | grep -i udid     ----- existing disk in dg
flags:     online ready private autoconfig udid_mismatch noautoimport
udid:      EMC%5FSYMMETRIX%5F000290101186%5F8601999008
 tag      udid_asl=EMC%5FSYMMETRIX%5F101186%5F60060480000290101186533031393939


root#  vxdisk -v list emc9_096c | grep -i udid    --- new disk
udid:      EMC%5FSYMMETRIX%5F000298700599%5F990096C000
 tag      udid_asl=EMC%5FSYMMETRIX%5F000298700599%5F990096C000


root # vxdisk -v list emc7_1999 | grep -i flag  ----- existing disk in dg     
flags:     online ready private autoconfig udid_mismatch noautoimport


root # vxdisk -v list emc9_096c | grep -i  flag    --- new disk
flags:     online ready private autoconfig autoimport thinrclm
root #

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

emc7_1999 has a mismatched UUID

SaGu
Level 3
Certified

Yes the existing disk in DG does have udid_mismatch.

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

So I would clear it using the process in the note.
 

mikebounds
Level 6
Partner Accredited

On my vbox setup I am able to add disk with clone or udid_mismatch, but you can clear udid first also - see below:

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sdc          testdg       online clone_disk
sdf          auto:cdsdisk    -            -            online udid_mismatch
# vxdg -g testdg adddisk sdf
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sdc          testdg       online clone_disk
sdf          auto:cdsdisk    sdf          testdg       online udid_mismatch
# vxdg -g testdg rmdisk sdf
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sdc          testdg       online clone_disk
sdf          auto:cdsdisk    -            -            online udid_mismatch
# vxdisk updateudid sdf
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sdc          testdg       online clone_disk
sdf          auto:cdsdisk    -            -            online clone_disk
# vxdg -g testdg adddisk sdf
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sdc          testdg       online clone_disk
sdf          auto:cdsdisk    sdf          testdg       online clone_disk

 

Mike

 

 

SaGu
Level 3
Certified

Mike,

Are you testing on Linux box? Will it behave the same for solaris? We did similar upgrade on linux box but don't see the udid_mismatch flag on disks in linux.

Can you try adding a disk which doesn't have udid_mismatch flag to a DG conatining udid_mismatch disks?
 

mikebounds
Level 6
Partner Accredited

Yes I am doing this on Linux and it should in theory behave the same on Solaris, but udid doesn't work properly on vbox so can't be sure.

For adding a disk which doesn't have udid_mismatch flag to a DG conatining udid_mismatch:

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    -            -            online
sdf          auto:cdsdisk    sdf          testdg       online udid_mismatch
# vxdg -g testdg adddisk sde
VxVM vxdg ERROR V-5-1-18510 Device sde is a non-clone disk.
Adding a non-clone disk to a clone diskgroup is not allowed
#
# vxdisk set sde clone=on
# vxdg -g testdg adddisk sde
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
sde          auto:cdsdisk    sde          testdg       online clone_disk
sdf          auto:cdsdisk    sdf          testdg       online udid_mismatch
#

 

 

As far as I can see, you have nothing to lose, by setting clone=on on new disk to see if this means you can add to diskgroup -i.e worse that can happen is it still won't let you add it to diskgroup

 

Mike

SaGu
Level 3
Certified

Mike,

We followed the steps of updating the udid and clone flags. One one node the the udid_mismatch flag is gone from all disks but on other node when we check vxdisk o/p the flag is still thr :(  . I was searching for the error and found that you faced similar issue earlier on 6.1. Do let me know what you have done. 

In the meantime i have created a support case also and the engineer also suggested the same steps. I've also updated him about the current situation.

mikebounds
Level 6
Partner Accredited

I tested on 6.1 on RHEL 5 running in vbox.  There is an issue with udid on vbox as the udid should be created from the serial number of the LUN obtained from the device layer, but on vbox, the LUN serial number obtained from the device layer is made up of hostname and O/S disk device name and hence is different between nodes so you get udid mismatch on one node and not the other which is what makes it easy for me to reproduce issue as if I run vxdisksetup on a disk on one system the other system sees udid mismatch.

You shouldn't have the "vbox" issue as your udid looks to be based on the WWN, but you can check udid yourself from the output of vxdisk -v list:

The value of udid is the value of the udid of the disk

The value of udid_asl (after word tag) is the udid written in the private region of the disk

Note it is more clear if you get udid from "vxdisk -p list" as then the fields are "UDID" and "PRIV_UDID"

So if the 2 match, then everything is fine, but if they do not match then you get udid_mismatch.  The value of both of these values should be identical when viewed from other nodes for the same disk (this is where issue with vbox lies).

When you said you "We followed the steps of updating the udid and clone flags." is this my steps of adding clone disk to diskgroup while online, or the technote steps of deporting diskgroup and unsetting clone?

If you are seeing issues on one node and not the other then post output of:

vxdisk -p list problem_disk | egrep "UDID|SERIAL"

Mike

 

 

SaGu
Level 3
Certified

something weird is in the o/p . If you see below the udid is same in all case but on node02 you see the udid_mismatch flag while on node01 it's not.

 

root@node01 # vxdisk -p list emc7_1699 | grep -i "UDID"
UDID           : EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node01 # vxdisk -v list emc7_1699 | grep -i udid
udid:      EMC%5FSYMMETRIX%5F000290101186%5F8601699008
 tag      udid_asl=EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node01 #


root@node02 #  vxdisk -p list emc7_1699 | grep -i "UDID"
UDID           : EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node02 # vxdisk -v list emc7_1699 | grep -i udid
flags:     online ready private autoconfig udid_mismatch noautoimport imported
udid:      EMC%5FSYMMETRIX%5F000290101186%5F8601699008
 tag      udid_asl=EMC%5FSYMMETRIX%5F000290101186%5F8601699008
root@node02 #

mikebounds
Level 6
Partner Accredited

Try running "vxdisk -o alldgs list" on node that has wrong info as this reads the private region of the disks and updates the "in-memory" copy you see when you run "vxdisk list".

Mike

 

SaGu
Level 3
Certified

Hi Guys,

 

We finally were able to resolve the issue.

1) First we deported the DG and cleared the udid_mismatch as mentioned in the Technote.

2) Next I noticed that ssh was not working from one node to other(someone changed the ssh keys :(  ) so sorted that. Not sure if that can be the reason for having the udid_mismatch flag on one node and not on other.

A clean reboot after that fixed the things.