11-15-2010 12:05 PM
Hi,
I am having problem initializing the disk with "vxdisksetup". These are the LUNS which were removed from veritas & storage , and have been re-allocated back to the same system. I can see the disks from OS and also in veritas but with "error" status.
vxdisksetup exits with the following error, but I can see the device in both /dev/vx/dmp and also in /dev/vx/rdmp. I tried removing the disk from veritas, remove the files from both /dev/vx/dmp and /dev/vx/rdmp, label the disk and then re-run "vxdctl enable" but seeing all the 3 disks in "error" status. I didn't move the file disk.info file & run "vxconfigd -K". Its a cluster node (Sun cluster) so wondering if there is anything I can do without moving disk.info file.
# vxdisk list |grep error
c2t5006048AD52F4106d14s2 auto - - error
c2t5006048AD52F4106d35s2 auto - - error
c2t5006048AD52F4106d36s2 auto - - error
# vxdisksetup -i c2t5006048AD52F4106d14 format=sliced
prtvtoc: /dev/vx/rdmp/c2t5006048AD52F4106d14s2: No such device or address
# prtvtoc /dev/vx/rdmp/c2t5006048AD52F4106d14s2
prtvtoc: /dev/vx/rdmp/c2t5006048AD52F4106d14s2: No such device or address
# prtvtoc /dev/rdsk/c2t5006048AD52F4106d14s2
* /dev/rdsk/c2t5006048AD52F4106d14s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 533 sectors/track
* 30 tracks/cylinder
* 15990 sectors/cylinder
* 65535 cylinders
* 65533 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 0 1047872670 1047872669
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
2 5 01 0 1047872670 1047872669
#
Question on vxdiskunsetup:
I have also noticed the following error on some disks, when running "vxdiskunsetup".
#vxdiskunsetup c2t5006048AD52F4126d39
VxVM vxdiskunsetup ERROR V-5-2-3524 c2t5006048AD52F4126d39: No dmpnode is associated
Thanks,
11-15-2010 07:19 PM
what about paths to those disks ? can you paste following:
vxdisk list c2t5006048AD52F4106d14s2
& similar output for other disks ...
you are getting prtvtoc output from raw disk or DMP node ?
Can you also paste:
# modinfo | grep -i vx
# vxdmpadm listctlr all
# vxdmpadm listenclosure all
# vxdisk -e list | egrep 'cxtxdcx | cxtxdx' (put your errored disk devices here)
Gaurav
11-16-2010 07:25 AM
Gaurav,
I am seeing error while running prtvtoc for DMP node and don't have any issue for getting disk geometry for rdsk. I have pasted the outputs also.
# prtvtoc /dev/vx/rdmp/c2t5006048AD52F4106d14s2
prtvtoc: /dev/vx/rdmp/c2t5006048AD52F4106d14s2: No such device or address
# vxdisk list |grep error
NONAMEs2 auto - - error
c2t5006048AD52F4106d14s2 auto - - error
c2t5006048AD52F4106d35s2 auto - - error
c2t5006048AD52F4106d36s2 auto - - error
#
# vxdisk list c2t5006048AD52F4106d14s2
Device: c2t5006048AD52F4106d14s2
devicetag: c2t5006048AD52F4106d14
type: auto
flags: online error private autoconfig
pubpaths: block=/dev/vx/dmp/c2t5006048AD52F4106d14s2 char=/dev/vx/rdmp/c2t5006048AD52F4106d14s2
guid: -
udid: INVALID
site: -
Multipathing information:
numpaths: 1
emcpower18c state=disabled
#
# vxdisk list c2t5006048AD52F4106d35s2
Device: c2t5006048AD52F4106d35s2
devicetag: c2t5006048AD52F4106d35
type: auto
flags: online error private autoconfig
pubpaths: block=/dev/vx/dmp/c2t5006048AD52F4106d35s2
char=/dev/vx/rdmp/c2t5006048AD52F4106d35s2
guid: -
udid: INVALID
site: -
Multipathing information:
numpaths: 1
emcpower16c state=disabled
#
# vxdisk list c2t5006048AD52F4106d36s2
Device: c2t5006048AD52F4106d36s2
devicetag: c2t5006048AD52F4106d36
type: auto
flags: online error private autoconfig
pubpaths: block=/dev/vx/dmp/c2t5006048AD52F4106d36s2 char=/dev/vx/rdmp/c2t5006048AD52F4106d36s2
guid: -
udid: INVALID
site: -
Multipathing information:
numpaths: 1
emcpower15c state=disabled
#
# modinfo|grep -i vx
50 7bec6000 3e4e0 308 1 vxdmp (VxVM 5.0MP3: DMP Driver)
52 7ba00000 209248 309 1 vxio (VxVM 5.0MP3 I/O driver)
54 7bbe90f0 c78 310 1 vxspec (VxVM 5.0MP3 control/status driv)
244 7aa213c0 cb0 311 1 vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
245 79e00000 1d89e0 21 1 vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
263 79fe4000 a9e0 312 1 fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)
# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
emcp PP_EMC ENABLED pp_emc0
c0 Disk ENABLED disk
c1 Disk ENABLED disk
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_
COUNT
================================================================================
===
disk Disk DISKS CONNECTED Disk 4
#
# vxdisk -e list | egrep 'c2t5006048AD52F4106d14s2|c2t5006048AD52F4106d35s2|c2t5006048AD52F4106d36s2'
c2t5006048AD52F4106d14s2 auto - - error emcpower18c -
c2t5006048AD52F4106d35s2 auto - - error emcpower16c -
c2t5006048AD52F4106d36s2 auto - - error emcpower15c -
# vxdmpadm getsubpaths ctlr=emcp|grep -i disabled
emcpower17c DISABLED - PP_EMC pp_emc0 -
emcpower18c DISABLED - c2t5006048AD52F4106d14s2 PP_EMC pp_emc0 -
emcpower16c DISABLED - c2t5006048AD52F4106d35s2 PP_EMC pp_emc0 -
emcpower15c DISABLED - c2t5006048AD52F4106d36s2 PP_EMC pp_emc0 -
11-16-2010 07:50 AM
nmcherla,
You mentioned prtvtoc of the disk works - are you using the cXtXdX device or the emcpower devices?
Could you also provide the following output for the powerpath devices, since VxVM is using the emcpower devices as the underlying path for the disks that you've indicated are in error
# /etc/powermt display dev=emcpower15c
# /etc/powermt display dev=emcpower16c
# /etc/powermt display dev=emcpower17c
# /etc/powermt display dev=emcpower18c
11-16-2010 09:19 AM
I am using cxtxdx device not the emcpower device for prtvtoc. Powerpath doesn't list the emcpower devices that VxVM uses (emcpowerxxc).
The Pseduo name that EMC powerpath uses for the device C2t5006048AD52F4106d14s2 is "emcpower54a",( VXVM uses emcpower18c. ) And both the paths are active and alive. As I said these are the LUNS removed and re-assigned to the same system and these devices were under veritas control before. I think some how VxVM is using the previous information of the LUN as we can see the emcpower device used by Veritas is different from powerpath.
# vxdisk list c2t5006048AD52F4106d14s2
Device: c2t5006048AD52F4106d14s2
devicetag: c2t5006048AD52F4106d14
type: auto
flags: online error private autoconfig
pubpaths: block=/dev/vx/dmp/c2t5006048AD52F4106d14s2 char=/dev/vx/rdmp/c2t5006048AD52F4106d14s2
guid: -
udid: INVALID
site: -
Multipathing information:
numpaths: 1
emcpower18c state=disabled
# powermt display dev=emcpower18c
Bad dev value emcpower18c, or not under Powerpath control.
#
# powermt display dev=c2t5006048AD52F4106d14s2
Pseudo name=emcpower54a
Symmetrix ID=000190102788
Logical device ID=0AFB
state=alive; policy=SymmOpt; priority=0; queued-IOs=0
==============================================================================
---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3072 pci@3,700000/SUNW,emlxs@0/fp@0,0 c2t5006048AD52F4106d14s0 FA 7cA active alive 0 0
3073 pci@13,700000/SUNW,emlxs@0/fp@0,0 c4t5006048AD52F4109d14s0 FA 10cA active alive 0 0
# vxdisk list |grep error
DEVICE TYPE DISK GROUP STATUS
NONAMEs2 auto - - error
c2t5006048AD52F4106d14s2 auto - - error
c2t5006048AD52F4106d35s2 auto - - error
c2t5006048AD52F4106d36s2 auto - - error
# prtvtoc /dev/rdsk/c2t5006048AD52F4106d14s2
* /dev/rdsk/c2t5006048AD52F4106d14s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 533 sectors/track
* 30 tracks/cylinder
* 15990 sectors/cylinder
* 65535 cylinders
* 65533 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 0 1047872670 1047872669
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
2 5 01 0 1047872670 1047872669
#
11-16-2010 10:18 AM
If the devices were used by Veritas before, then removed and represented without updating/removing the disks cleanly from Veritas first, then this would explain why the entries are showing in error state, as the configuration hasn't been updated cleanly.
At this point, the best bet is to:
1. remove the devices in VxVM:
# vxdisk rm <disk> ### ie: the disks shown in error
2. Remove the entries for those disks under /dev/vx/dmp and /dev/vx/rdmp
3. Rebuild the dmp device tree
# vxdctl initdmp
4. Now rescan
# vxdisk scandisks
NOTE: this still may not work as the devices were not removed cleanly, so the incorrect config may be stuck in the kernel. If the above does not work, I'd suggest scheduling a reboot rather than trying to remove the disk.info file, as this probably won't work either/may confuse the issue further.
To avoid this situation in future, ensure disks are removed from VxVM cleanly (ie: vxdisk rm, removing devices under /dev/vx/*dmp/, ensuring disks are not rescanned in after this point by stopping vxesd) PRIOR to cleaning up PowerPath and removing the disks from the OS.
11-16-2010 11:30 AM
I have tried removing the disks (vxdisk rm), removing entries in /dev/vx/*dmp, and ran vxdctl enable. But didn't run initdmp. Its a prmiary cluster node (Sun cluster), will the initdmp by any chance failover the groups?
To remove the LUNS, do we need to remove the devices under /dev/vx/*dmp after removing the disk from veritas?
Thanks,
11-16-2010 07:44 PM
initdmp should not ideally cause a failover of cluster.... but I doubt it to resolve the issue in your case.
I would reccomend to plan for outage & do a reconfig reboot ... additionally it will be good if power devices are reconfigured too (powercf) which would be done during the boot time..
since devices were represented, I believe that power devices names were changed which veritas has not picked up.
I also noted above that EMC array is not listing in vxdmpadm listenclosure output... can you confirm if anything you have excluded ?
# vxdmpadm listexclude all
Gaurav
11-17-2010 06:19 AM
Gaurav,
I did see that and nothing has been excluded. What could be the reason?
# vxdmpadm listexclude all
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT
===================================================================================
disk Disk DISKS CONNECTED Disk 4
#
11-17-2010 07:57 PM
Again my recommendation would be to do a reconfig reboot ... not sure how it landed to this state...
btw, did you tried a devfsadm before running a vxdctl enable ?
Gaurav
11-18-2010 09:10 AM
Yes. I have used devfsadm to identify the new luns and then vxdctl enable. array.info file have all the information but from where did the "vxdmpadm listctlr and enclosure" getting the the information? I was trying to avoid reboot but looks like there is no other option for me at this point. I will schedule a reboot for this server. Thanks Gaurav and to every one who replied for my post.