Forum Discussion

nmcherla's avatar
nmcherla
Level 3
14 years ago

vxdisksetup error

Hi,

I am having problem initializing the disk with "vxdisksetup".  These are the LUNS which were removed from veritas & storage , and have been re-allocated back to the same system.  I can see the disks from OS and also in veritas but with "error" status. 

vxdisksetup exits with the following error, but I can see the device in both /dev/vx/dmp and also in /dev/vx/rdmp.  I tried removing the disk from veritas, remove the files from both /dev/vx/dmp and /dev/vx/rdmp, label the disk and then re-run "vxdctl enable" but seeing all the 3 disks in "error" status.  I didn't move the file disk.info file & run "vxconfigd -K".  Its a cluster node (Sun cluster) so wondering if there is anything I can do without moving disk.info file.

# vxdisk list |grep error
c2t5006048AD52F4106d14s2 auto            -            -            error
c2t5006048AD52F4106d35s2 auto            -            -            error
c2t5006048AD52F4106d36s2 auto            -            -            error

# vxdisksetup -i c2t5006048AD52F4106d14 format=sliced
prtvtoc: /dev/vx/rdmp/c2t5006048AD52F4106d14s2: No such device or address

# prtvtoc /dev/vx/rdmp/c2t5006048AD52F4106d14s2
prtvtoc: /dev/vx/rdmp/c2t5006048AD52F4106d14s2: No such device or address

# prtvtoc /dev/rdsk/c2t5006048AD52F4106d14s2
* /dev/rdsk/c2t5006048AD52F4106d14s2 partition map
*
* Dimensions:
*     512 bytes/sector
*     533 sectors/track
*      30 tracks/cylinder
*   15990 sectors/cylinder
*   65535 cylinders
*   65533 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*           0 1047872670 1047872669
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       2      5    01          0 1047872670 1047872669
#
 

Question on vxdiskunsetup:

I have also noticed the following error on some disks, when running "vxdiskunsetup". 

#vxdiskunsetup c2t5006048AD52F4126d39

VxVM vxdiskunsetup ERROR V-5-2-3524 c2t5006048AD52F4126d39: No dmpnode is associated



Thanks,

10 Replies

  • what about paths to those disks ? can you paste following:

    vxdisk list c2t5006048AD52F4106d14s2

    & similar output for other disks ...

    you are getting prtvtoc output from raw disk or DMP node ?

    Can you also paste:

    # modinfo | grep -i vx

    # vxdmpadm listctlr all

    # vxdmpadm listenclosure all

    # vxdisk -e list | egrep 'cxtxdcx | cxtxdx'   (put your errored disk devices here)

     

    Gaurav

  • Gaurav,

    I am seeing error while  running prtvtoc for DMP node and don't have any issue for getting disk geometry for rdsk.  I have pasted the outputs also.

    # prtvtoc /dev/vx/rdmp/c2t5006048AD52F4106d14s2
    prtvtoc: /dev/vx/rdmp/c2t5006048AD52F4106d14s2: No such device or address

    # vxdisk list |grep error
    NONAMEs2     auto            -            -            error
    c2t5006048AD52F4106d14s2 auto            -            -            error
    c2t5006048AD52F4106d35s2 auto            -            -            error
    c2t5006048AD52F4106d36s2 auto            -            -            error
    #

    # vxdisk list c2t5006048AD52F4106d14s2
    Device:    c2t5006048AD52F4106d14s2
    devicetag: c2t5006048AD52F4106d14
    type:      auto
    flags:     online error private autoconfig
    pubpaths:  block=/dev/vx/dmp/c2t5006048AD52F4106d14s2 char=/dev/vx/rdmp/c2t5006048AD52F4106d14s2
    guid:      -
    udid:      INVALID
    site:      -
    Multipathing information:
    numpaths:   1
    emcpower18c     state=disabled
    #
    # vxdisk list c2t5006048AD52F4106d35s2
    Device:    c2t5006048AD52F4106d35s2
    devicetag: c2t5006048AD52F4106d35
    type:      auto
    flags:     online error private autoconfig
    pubpaths:  block=/dev/vx/dmp/c2t5006048AD52F4106d35s2

    char=/dev/vx/rdmp/c2t5006048AD52F4106d35s2
    guid:      -
    udid:      INVALID
    site:      -
    Multipathing information:
    numpaths:   1
    emcpower16c     state=disabled
    #
    # vxdisk list c2t5006048AD52F4106d36s2
    Device:    c2t5006048AD52F4106d36s2
    devicetag: c2t5006048AD52F4106d36
    type:      auto
    flags:     online error private autoconfig
    pubpaths:  block=/dev/vx/dmp/c2t5006048AD52F4106d36s2 char=/dev/vx/rdmp/c2t5006048AD52F4106d36s2
    guid:      -
    udid:      INVALID
    site:      -
    Multipathing information:
    numpaths:   1
    emcpower15c     state=disabled
    #

    # modinfo|grep -i vx
     50 7bec6000  3e4e0 308   1  vxdmp (VxVM 5.0MP3: DMP Driver)
     52 7ba00000 209248 309   1  vxio (VxVM 5.0MP3 I/O driver)
     54 7bbe90f0    c78 310   1  vxspec (VxVM 5.0MP3 control/status driv)
    244 7aa213c0    cb0 311   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
    245 79e00000 1d89e0  21   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
    263 79fe4000   a9e0 312   1  fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)

    # vxdmpadm listctlr all
    CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
    =====================================================
    emcp            PP_EMC          ENABLED      pp_emc0
    c0              Disk            ENABLED      disk
    c1              Disk            ENABLED      disk
     

    # vxdmpadm listenclosure all

    ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_
    COUNT
    ================================================================================
    ===
    disk              Disk           DISKS                CONNECTED    Disk        4
    #

     # vxdisk -e list | egrep 'c2t5006048AD52F4106d14s2|c2t5006048AD52F4106d35s2|c2t5006048AD52F4106d36s2'
    c2t5006048AD52F4106d14s2 auto      -             -            error        emcpower18c  -
    c2t5006048AD52F4106d35s2 auto      -             -            error        emcpower16c  -
    c2t5006048AD52F4106d36s2 auto      -             -            error        emcpower15c  -
     

    # vxdmpadm getsubpaths ctlr=emcp|grep -i disabled
    emcpower17c  DISABLED     -                       PP_EMC       pp_emc0        -
    emcpower18c  DISABLED     -          c2t5006048AD52F4106d14s2 PP_EMC       pp_emc0        -
    emcpower16c  DISABLED     -          c2t5006048AD52F4106d35s2 PP_EMC       pp_emc0        -
    emcpower15c  DISABLED     -          c2t5006048AD52F4106d36s2 PP_EMC       pp_emc0        -
     

  • nmcherla,

    You mentioned prtvtoc of the disk works - are you using the cXtXdX device or the emcpower devices?

    Could you also provide the following output for the powerpath devices, since VxVM is using the emcpower devices as the underlying path for the disks that you've indicated are in error

    # /etc/powermt display dev=emcpower15c
    # /etc/powermt display dev=emcpower16c
    # /etc/powermt display dev=emcpower17c
    # /etc/powermt display dev=emcpower18c

  • I am using cxtxdx device not the emcpower device for prtvtoc.  Powerpath doesn't list the emcpower devices that VxVM uses (emcpowerxxc). 

    The Pseduo name that EMC powerpath uses for the device C2t5006048AD52F4106d14s2 is "emcpower54a",( VXVM uses emcpower18c. )  And both the paths are active and alive. As I said these are the LUNS removed and re-assigned to the same system and these devices were under veritas control before.  I think some how VxVM is using the previous information of the LUN as we can see the emcpower device used by Veritas is different from powerpath.
     

    # vxdisk list c2t5006048AD52F4106d14s2
    Device:    c2t5006048AD52F4106d14s2
    devicetag: c2t5006048AD52F4106d14
    type:      auto
    flags:     online error private autoconfig
    pubpaths:  block=/dev/vx/dmp/c2t5006048AD52F4106d14s2 char=/dev/vx/rdmp/c2t5006048AD52F4106d14s2
    guid:      -
    udid:      INVALID
    site:      -
    Multipathing information:
    numpaths:   1
    emcpower18c     state=disabled
     

    #  powermt display dev=emcpower18c
    Bad dev value emcpower18c, or not under Powerpath control.
    #
     

    # powermt display dev=c2t5006048AD52F4106d14s2
    Pseudo name=emcpower54a
    Symmetrix ID=000190102788
    Logical device ID=0AFB
    state=alive; policy=SymmOpt; priority=0; queued-IOs=0
    ==============================================================================
    ---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
    ###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
    ==============================================================================
    3072 pci@3,700000/SUNW,emlxs@0/fp@0,0 c2t5006048AD52F4106d14s0 FA  7cA   active  alive      0      0
    3073 pci@13,700000/SUNW,emlxs@0/fp@0,0 c4t5006048AD52F4109d14s0 FA 10cA   active  alive      0      0

     

    # vxdisk list |grep error

    DEVICE       TYPE            DISK         GROUP        STATUS
    NONAMEs2     auto            -            -            error
    c2t5006048AD52F4106d14s2 auto            -            -            error
    c2t5006048AD52F4106d35s2 auto            -            -            error
    c2t5006048AD52F4106d36s2 auto            -            -            error
     

    # prtvtoc /dev/rdsk/c2t5006048AD52F4106d14s2
    * /dev/rdsk/c2t5006048AD52F4106d14s2 partition map
    *
    * Dimensions:
    *     512 bytes/sector
    *     533 sectors/track
    *      30 tracks/cylinder
    *   15990 sectors/cylinder
    *   65535 cylinders
    *   65533 accessible cylinders
    *
    * Flags:
    *   1: unmountable
    *  10: read-only
    *
    * Unallocated space:
    *       First     Sector    Last
    *       Sector     Count    Sector
    *           0 1047872670 1047872669
    *
    *                          First     Sector    Last
    * Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
           2      5    01          0 1047872670 1047872669
    #
     

  • If the devices were used by Veritas before, then removed and represented without updating/removing the disks cleanly from Veritas first, then this would explain why the entries are showing in error state, as the configuration hasn't been updated cleanly.

    At this point, the best bet is to:

    1. remove the devices in VxVM:

    # vxdisk rm <disk> ### ie: the disks shown in error

    2. Remove the entries for those disks under /dev/vx/dmp and /dev/vx/rdmp

    3. Rebuild the dmp device tree

    # vxdctl initdmp

    4. Now rescan

    # vxdisk scandisks

     

    NOTE: this still may not work as the devices were not removed cleanly, so the incorrect config may be stuck in the kernel. If the above does not work, I'd suggest scheduling a reboot rather than trying to remove the disk.info file, as this probably won't work either/may confuse the issue further.

    To avoid this situation in future, ensure disks are removed from VxVM cleanly (ie: vxdisk rm, removing devices under /dev/vx/*dmp/, ensuring disks are not rescanned in after this point by stopping vxesd) PRIOR to cleaning up PowerPath and removing the disks from the OS.

  • I have tried removing the disks (vxdisk rm), removing entries in /dev/vx/*dmp, and ran vxdctl enable.  But didn't run initdmp.  Its a prmiary cluster node (Sun cluster), will the initdmp by any chance failover the groups?

    To remove the LUNS, do we need to remove the devices under /dev/vx/*dmp after removing the disk from veritas?

    Thanks,

  • initdmp should not ideally cause a failover of cluster....  but I doubt it to resolve the issue in your case.

    I would reccomend to plan for outage  & do a reconfig reboot ... additionally it will be good if power devices are reconfigured too (powercf) which would be done during the boot time..

    since devices were represented, I believe that power devices names were changed which veritas has not picked up.

    I also noted above that EMC array is not listing in vxdmpadm listenclosure output... can you confirm if anything you have excluded ?

    # vxdmpadm listexclude all

     

    Gaurav

  • Gaurav,

    I did see that and nothing has been excluded.  What could be the reason?

    # vxdmpadm listexclude all
    # vxdmpadm listenclosure all
    ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT
    ===================================================================================
    disk              Disk           DISKS                CONNECTED    Disk        4
    #
     

  • Again my recommendation would be to do a reconfig reboot ... not sure how it landed to this state...

    btw, did you tried a devfsadm before running a vxdctl enable ?

     

    Gaurav

  • Yes.  I have used devfsadm to identify the new luns and then vxdctl enable.  array.info file have all the information but from where did the "vxdmpadm listctlr and  enclosure" getting the the information? I was trying to avoid reboot but looks like there is no other option for me at this point.  I will schedule a reboot for this server.  Thanks Gaurav and to every one who replied for my post.