Forum Discussion

gregsih's avatar
gregsih
Level 4
12 years ago

Veritas Volume Manager problem

We have 2 Sun 3510 arrays each containing three disks. VxVM, on Solaris 10 for SPARC, has been set up to mirror each disk, ie Array 1 Disk 1 is mirrored to Array 2 Disk 1 etc etc. We had a problem with the power to the controller of array 2 and the disks in that array ended up in a disabled and removed state.

The power has now been restored and the disks are now visible to the operating system again. However whatever we try we can not get VxVM to accept the disks.

It can see them but will not re-add them. We have followed the replace disk procedure which fails with the message 'no device available to replace'.

Any suggestions would be most gratefuly received.

business continuity
Solaris
Storage Foundation
Tip-How to
Troubleshooting
  • g_lee's avatar
    g_lee
    12 years ago

    Greg,

    Try on Disk_9 first

    # vxdiskunsetup -C Disk_9

    # vxdisksetup -i Disk_9 format=cdsdisk privlen=2048

    # vxdisk list Disk_9

    The fields in bold (from your previous output) should be the same. If they're not, stop here and post the vxdisk list Disk_9 output!

    root@server01 # vxdisk list Disk_9
    [...]
    info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
    flags:     online ready private autoconfig autoimport
    pubpaths:  block=/dev/vx/dmp/Disk_9s2 char=/dev/vx/rdmp/Disk_9s2
    version:   3.1
    iosize:    min=512 (bytes) max=2048 (blocks)
    public:    slice=2 offset=2304 len=1140796160 disk_offset=0
    private:   slice=2 offset=256 len=2048 disk_offset=0
    [...]
    Multipathing information:
    numpaths:   1
    c5t40d1s2       state=enabled

    If they are the same, try to add this to the dg now:

    # vxdg -g datadg -k adddisk datadg04=Disk_9

    Repeat steps for Disk_10, Disk_11 (ensure privlen=2048 is specified in the vxdisksetup command)

    If this works, then post the vxprint -qhtrg datadg output after adding the disks so the plexes can be recovered

27 Replies

Sort By
  • Hari_Krishna_V's avatar
    Hari_Krishna_V
    Level 3
    12 years ago

    If the disks are visible in 'vxdisk list' output then you might need to run '/etc/vx/bin/vxreattach <device>' so that the devices get attached back to the disk group.

    A CLI snapshot might help better understand the issue

  • gregsih's avatar
    gregsih
    Level 4
    12 years ago
    Sorry but I have tried that with the result shown below:


root@server01 # vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
Disk_0       auto:sliced     rootdg01     rootdg       online
Disk_4       auto:cdsdisk    datadg01     datadg       online
Disk_5       auto:cdsdisk    datadg02     datadg       online
Disk_6       auto:cdsdisk    datadg03     datadg       online
Disk_7       auto:sliced     rootdg02     rootdg       online nohotuse
-            -         datadg04     datadg       removed nohotuse was:Disk_9
-            -         datadg06     datadg       removed nohotuse was:Disk_11
-            -         datadg05     datadg       removed was:Disk_10
root@server01 # vxreattach Disk_9
VxVM vxdisk ERROR V-5-1-537 Device Disk_9: Not in the configuration
VxVM vxdisk ERROR V-5-1-558 Disk Disk_9: Disk not in the configuration
root@server01 #  vxprint
Disk group: rootdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg rootdg       rootdg       -        -        -        -        -       -

dm rootdg01     Disk_0       -        143349311 -       -        -       -
dm rootdg02     Disk_7       -        143328960 -       NOHOTUSE -       -

sd rootdg01Priv -            ENABLED  20351    -        -        -       PRIVATE

v  rootvol      root         ENABLED  135177984 -       ACTIVE   -       -
pl rootvol-01   rootvol      ENABLED  135177984 -       ACTIVE   -       -
sd rootdg01-B0  rootvol-01   ENABLED  1        0        -        -       Block0
sd rootdg01-02  rootvol-01   ENABLED  135177983 1       -        -       -
pl rootvol-02   rootvol      ENABLED  135177984 -       ACTIVE   -       -
sd rootdg02-01  rootvol-02   ENABLED  135177984 0       -        -       -

v  swapvol      swap         ENABLED  8150976  -        ACTIVE   -       -
pl swapvol-01   swapvol      ENABLED  8150976  -        ACTIVE   -       -
sd rootdg01-01  swapvol-01   ENABLED  8150976  0        -        -       -
pl swapvol-02   swapvol      ENABLED  8150976  -        ACTIVE   -       -
sd rootdg02-02  swapvol-02   ENABLED  8150976  0        -        -       -

Disk group: datadg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg datadg       datadg       -        -        -        -        -       -

dm datadg01     Disk_4       -        1140796160 -      -        -       -
dm datadg02     Disk_5       -        1140796160 -      -        -       -
dm datadg03     Disk_6       -        1140796160 -      -        -       -
dm datadg04     -            -        -        -        REMOVED  -       -
dm datadg05     -            -        -        -        REMOVED  -       -
dm datadg06     -            -        -        -        REMOVED  -       -

v  vol01        fsgen        ENABLED  1140795392 -      ACTIVE   -       -
pl vol01-01     vol01        ENABLED  1140795392 -      ACTIVE   -       -
sd datadg01-01  vol01-01     ENABLED  1140795392 0      -        -       -
pl vol01-02     vol01        DISABLED 1140795392 -      REMOVED  -       -
sd datadg04-01  vol01-02     DISABLED 1140795392 0      REMOVED  -       -

v  vol02        fsgen        ENABLED  1140795392 -      ACTIVE   -       -
pl vol02-01     vol02        ENABLED  1140795392 -      ACTIVE   -       -
sd datadg02-01  vol02-01     ENABLED  1140795392 0      -        -       -
pl vol02-02     vol02        DISABLED 1140795392 -      REMOVED  -       -
sd datadg05-01  vol02-02     DISABLED 1140795392 0      REMOVED  -       -

v  vol03        fsgen        ENABLED  1140795392 -      ACTIVE   -       -
pl vol03-01     vol03        ENABLED  1140795392 -      ACTIVE   -       -
sd datadg03-01  vol03-01     ENABLED  1140795392 0      -        -       -
pl vol03-02     vol03        DISABLED 1140795392 -      REMOVED  -       -
sd datadg06-01  vol03-02     DISABLED 1140795392 0      REMOVED  -       -

    Hope this helps.

     

    Greg

  • TonyGriffiths's avatar
    TonyGriffiths
    Level 6
    12 years ago

    You mentioned that the failed disks are now visble by the OS, but I could not see them in the vxdisk list output above, I can only see the removed references.

    cheers

    tony

  • gregsih's avatar
    gregsih
    Level 4
    12 years ago

    The disks in question are :

     

    datadg04

    datadg05

    datadg06

    All of which both Veritas and Solaris can see but nothing I do will pursuade Veritas to re-add them to the disk group datadg.

  • TonyGriffiths's avatar
    TonyGriffiths
    Level 6
    12 years ago

    Can you post the latest "vxdisk list" output, if they are visible to VxVM, they should be listed there with the correct state. All I could see in the above are the "removed" references, not the actual disks

    cheers

    tony

  • gregsih's avatar
    gregsih
    Level 4
    12 years ago

    The latest vxdisk list output is posted above.

    As the disks are showing as removed I have tried the re-add and add procedures with no result.

    Posted above is the current situation. Please bear in mind that the disks showing as removed are the disks we are trying to re-add.

  • TonyGriffiths's avatar
    TonyGriffiths
    Level 6
    12 years ago

    Hi

    When a disk gets "detached" or "removed" the disk will be listed as that state as its still refrenced by the imported DG. If the problem disk is still accessible, then it will also show up in an entry but not as part of a DG

     

    For example I do not see, Disk_9, Disk_10 & Disk11 in that list, only the detached references

    Possibly you need to run a rescan by VxVM

    cheers

    tony 

  • gregsih's avatar
    gregsih
    Level 4
    12 years ago

    Hi Tony

     

    you were right. a re-scan of the disks causes the following output :-

     

    root@server01 # vxdisk list
    DEVICE       TYPE            DISK         GROUP        STATUS
    Disk_0       auto:sliced     rootdg01     rootdg       online
    Disk_4       auto:cdsdisk    datadg01     datadg       online
    Disk_5       auto:cdsdisk    datadg02     datadg       online
    Disk_6       auto:cdsdisk    datadg03     datadg       online
    Disk_7       auto:sliced     rootdg02     rootdg       online nohotuse
    Disk_9       auto:cdsdisk    -            -            online
    Disk_10      auto:cdsdisk    -            -            online
    Disk_11      auto:cdsdisk    -            -            online
    -            -         datadg04     datadg       removed nohotuse was:Disk_9
    -            -         datadg06     datadg       removed nohotuse was:Disk_11
    -            -         datadg05     datadg       removed was:Disk_10

     

    But we still can not re-add them to the datadg.

     

    Regards

     

    Greg

  • g_lee's avatar
    g_lee
    Level 6
    12 years ago

    Please provide the following output:

    # vxdisk list Disk_9
    # vxdisk list Disk_10
    # vxdisk list Disk_11
    # vxprint -mg datadg | egrep -i "^dm|disk|device"

  • gregsih's avatar
    gregsih
    Level 4
    12 years ago

    Hi Tony

    Please see below:

     

    Regards

     

    Greg

    root@server01 # vxdisk list Disk_9
    Device:    Disk_9
    devicetag: Disk_9
    type:      auto
    hostid:    server01
    disk:      name= id=1196418828.29.server01
    group:     name=datadg id=1129915301.61.server01
    info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
    flags:     online ready private autoconfig autoimport
    pubpaths:  block=/dev/vx/dmp/Disk_9s2 char=/dev/vx/rdmp/Disk_9s2
    version:   3.1
    iosize:    min=512 (bytes) max=2048 (blocks)
    public:    slice=2 offset=2304 len=1140796160 disk_offset=0
    private:   slice=2 offset=256 len=2048 disk_offset=0
    update:    time=1360778965 seqno=0.52
    ssb:       actual_seqno=0.3
    headers:   0 240
    configs:   count=1 len=1280
    logs:      count=1 len=192
    Defined regions:
     config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
     config   priv 000256-001343[001088]: copy=01 offset=000192 enabled
     log      priv 001344-001535[000192]: copy=01 offset=000000 enabled
     lockrgn  priv 001536-001679[000144]: part=00 offset=000000
    Multipathing information:
    numpaths:   1
    c5t40d1s2       state=enabled
    root@server01 # vxdisk list Disk_10
    Device:    Disk_10
    devicetag: Disk_10
    type:      auto
    hostid:    server01
    disk:      name= id=1288191337.26.server01
    group:     name=datadg id=1129915301.61.server01
    info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
    flags:     online ready private autoconfig autoimport
    pubpaths:  block=/dev/vx/dmp/Disk_10s2 char=/dev/vx/rdmp/Disk_10s2
    version:   3.1
    iosize:    min=512 (bytes) max=2048 (blocks)
    public:    slice=2 offset=2304 len=1140796160 disk_offset=0
    private:   slice=2 offset=256 len=2048 disk_offset=0
    update:    time=1360778965 seqno=0.40
    ssb:       actual_seqno=0.1
    headers:   0 240
    configs:   count=1 len=1280
    logs:      count=1 len=192
    Defined regions:
     config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
     config   priv 000256-001343[001088]: copy=01 offset=000192 enabled
     log      priv 001344-001535[000192]: copy=01 offset=000000 enabled
     lockrgn  priv 001536-001679[000144]: part=00 offset=000000
    Multipathing information:
    numpaths:   1
    c5t40d0s2       state=enabled
    root@server01 # vxdisk list Disk_11
    Device:    Disk_11
    devicetag: Disk_11
    type:      auto
    hostid:    server01
    disk:      name= id=1196418973.33.server01
    group:     name=datadg id=1129915301.61.server01
    info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
    flags:     online ready private autoconfig autoimport
    pubpaths:  block=/dev/vx/dmp/Disk_11s2 char=/dev/vx/rdmp/Disk_11s2
    version:   3.1
    iosize:    min=512 (bytes) max=2048 (blocks)
    public:    slice=2 offset=2304 len=1140796160 disk_offset=0
    private:   slice=2 offset=256 len=2048 disk_offset=0
    update:    time=1360778965 seqno=0.54
    ssb:       actual_seqno=0.3
    headers:   0 240
    configs:   count=1 len=1280
    logs:      count=1 len=192
    Defined regions:
     config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
     config   priv 000256-001343[001088]: copy=01 offset=000192 enabled
     log      priv 001344-001535[000192]: copy=01 offset=000000 enabled
     lockrgn  priv 001536-001679[000144]: part=00 offset=000000
    Multipathing information:
    numpaths:   1
    c5t40d2s2       state=enabled

    root@server01 # vxprint -mg datadg|egrep -i "^dm|disk|device"
            diskdetpolicy=global
    dm   datadg01
            da_name=Disk_4
            device_tag=Disk_4
            pub_bpath="/dev/vx/dmp/Disk_4s2
            priv_bpath="/dev/vx/dmp/Disk_4s2
            pub_cpath="/dev/vx/rdmp/Disk_4s2
            priv_cpath="/dev/vx/rdmp/Disk_4s2
            diskid=1129914971.25.server01
            last_diskid=1129914971.25.server01
            last_da_name=Disk_4
            last_disk_offset=2304
    dm   datadg02
            da_name=Disk_5
            device_tag=Disk_5
            pub_bpath="/dev/vx/dmp/Disk_5s2
            priv_bpath="/dev/vx/dmp/Disk_5s2
            pub_cpath="/dev/vx/rdmp/Disk_5s2
            priv_cpath="/dev/vx/rdmp/Disk_5s2
            diskid=1129914973.27.server01
            last_diskid=1129914973.27.server01
            last_da_name=Disk_5
            last_disk_offset=2304
    dm   datadg03
            da_name=Disk_6
            device_tag=Disk_6
            pub_bpath="/dev/vx/dmp/Disk_6s2
            priv_bpath="/dev/vx/dmp/Disk_6s2
            pub_cpath="/dev/vx/rdmp/Disk_6s2
            priv_cpath="/dev/vx/rdmp/Disk_6s2
            diskid=1129914975.29.server01
            last_diskid=1129914975.29.server01
            last_da_name=Disk_6
            last_disk_offset=2304
    dm   datadg04
            last_diskid=1196418828.29.server01
            last_da_name=Disk_9
            last_disk_offset=2304
    dm   datadg05
            last_diskid=1288191337.26.server01
            last_da_name=Disk_10
            last_disk_offset=2304
    dm   datadg06
            last_diskid=1196418973.33.server01
            last_da_name=Disk_11
            last_disk_offset=2304
            nodevice=off
            da_name=Disk_4
            device_tag=Disk_4
            path="/dev/vx/dmp/Disk_4s2
            mkdevice=off
            nodevice=off
            da_name=Disk_5
            device_tag=Disk_5
            path="/dev/vx/dmp/Disk_5s2
            mkdevice=off
            nodevice=off
            da_name=Disk_6
            device_tag=Disk_6
            path="/dev/vx/dmp/Disk_6s2
            mkdevice=off
            nodevice=off
            device_tag=
            mkdevice=off
            nodevice=off
            device_tag=
            mkdevice=off
            nodevice=off
            device_tag=
            mkdevice=off