Forum Discussion

SaGu's avatar
SaGu
Level 3
14 years ago

Plex disabled stale while volume is enabled active

Hi All,

 

After a power outage i have a situation that the plexes went in disabled stale state while the volume is enabled active.

 

I've tried to fix clean the plex but then it says volume is enabled, anyways it's right because the volume is fine.

Any ideas how to get the plex enabled active again?

 

v  v_archlog    -            ENABLED  ACTIVE   184311808 SELECT   -        fsgen
pl v_archlog-01 v_archlog    DISABLED STALE    184312800 CONCAT   -        WO
sd mams06-dh-02 v_archlog-01 mams06-dh 20972160 41944320 0        HDS9980V0_10 ENA
sd mams11-dh-01 v_archlog-01 mams11-dh 0       71184960 41944320  HDS9980V0_0 ENA
sd mams11-lm-01 v_archlog-01 mams11-lm 0       71183520 113129280 HDS9980V1_0 ENA
pl v_archlog-02 v_archlog    ENABLED  ACTIVE   184312800 CONCAT   -        RW
sd mams06-lm-01 v_archlog-02 mams06-lm 0       41944320 0         HDS9980V1_10 ENA
sd mams12-dh-01 v_archlog-02 mams12-dh 0       71184960 41944320  HDS9980V0_1 ENA
sd mams12-lm-01 v_archlog-02 mams12-lm 0       71183520 113129280 HDS9980V1_1 ENA

  • Hello,

    one of the plex is disabled & its a mirrored volume, thats the reason volume is still in enabled state...

    I would suggest first:

    # vxrecover -bsE -g <diskgroup>

    THis would try to recover the plex & sync again. In case if this doesn't works, I would suggest to detach the stale plex & attach it again.. you can do so by "vxplex det" & "vxplex att"

     

    Gaurav

  • Hello,

    one of the plex is disabled & its a mirrored volume, thats the reason volume is still in enabled state...

    I would suggest first:

    # vxrecover -bsE -g <diskgroup>

    THis would try to recover the plex & sync again. In case if this doesn't works, I would suggest to detach the stale plex & attach it again.. you can do so by "vxplex det" & "vxplex att"

     

    Gaurav

  • Thanks for the reply!!

     

    Tried vxplex att but it gives i/o error.

    vxvm:vxplex: ERROR: Volume v_archlog, plex v_archlog-02, block 13329120: Plex read:
            Error: Read failure
    vxvm:vxplex: ERROR: I/O error on volume v_archlog, plex v_archlog-01 not attached

     

    Maybe something wrong with the disk or is it ust a block. Any ideas?

  • well from the vxprint output, it looks disks good & enabled

    pl v_archlog-01 v_archlog    DISABLED STALE    184312800 CONCAT   -        WO
    sd mams06-dh-02 v_archlog-01 mams06-dh 20972160 41944320 0        HDS9980V0_10 ENA
    sd mams11-dh-01 v_archlog-01 mams11-dh 0       71184960 41944320  HDS9980V0_0 ENA
    sd mams11-lm-01 v_archlog-01 mams11-lm 0       71183520 113129280 HDS9980V1_0 ENA

     

    Did the det command succeed ? If plex is detached, you tried to fix clean it outside the volume ? what is current vxprint output ?

    Do you see any failed/failing disk in vxdisk list ?

    what is vxvm version you are using ?

     

    Gaurav

  • yes detach command went without error. After detaching i just tried to reattach it.

    Below is the current o/p.

    v  v_archlog    -            ENABLED  ACTIVE   184311808 SELECT   -        fsgen
    pl v_archlog-01 v_archlog    DETACHED STALE    184312800 CONCAT   -        WO
    sd mams06-dh-02 v_archlog-01 mams06-dh 20972160 41944320 0        HDS9980V0_10 ENA
    sd mams11-dh-01 v_archlog-01 mams11-dh 0       71184960 41944320  HDS9980V0_0 ENA
    sd mams11-lm-01 v_archlog-01 mams11-lm 0       71183520 113129280 HDS9980V1_0 ENA
    pl v_archlog-02 v_archlog    ENABLED  ACTIVE   184312800 CONCAT   -        RW
    sd mams06-lm-01 v_archlog-02 mams06-lm 0       41944320 0         HDS9980V1_10 ENA
    sd mams12-dh-01 v_archlog-02 mams12-dh 0       71184960 41944320  HDS9980V0_1 ENA
    sd mams12-lm-01 v_archlog-02 mams12-lm 0       71183520 113129280 HDS9980V1_1 ENA

     

    Earlier all disk were fine but now mams06-lm disk which is in secondary active plex is now in disabled state. So that's also a worry now.

     

    It's veritas 3.5!!

  • OK failing state indicates that there was some temporary iofailure on the public region of the disk (data area) .. It may or may not be a worry.... I would suggest to keep an eye on iostat counters & see if hard/transport errors are increasing or more... if yes, then you may need to think of replacing disk..

    you can also turn off the failing flag on the disk, if it re-appears, you need to investigate...

    # vxedit -g <diskgroup> set failing=off <disk>

    Regarding the stale plex, I believe, you need to disassociate (vxplex dis) the plex (take out of volume), fix clean it & then associate it back.. but again make sure that all background issues with SAN are resolved & the three disks pertaining to bad plex are available & are not in failed/failing state...

    Gaurav

  • SaGu,

    The error you pasted earlier indicates mams06-lm has read errors / is now failing - this is why the detached plex could not be reattached, as VxVM encountered read errors on the "good" plex while trying to reattach.

    ie: the attach command syntax is fine, it's just the "good" plex is now having errors, thus preventing re-attach.

    Is mams06-lm actually seeing errors (on the underlying disk / in iostat)? If it's a transient condition, try the reattach again. If the disk still generates errors/still fails, would recommend ensuring you are backing up the data on this volume, as if the disk in the "good" plex is failing, and you are unable to reattach the plex before it fails, then you will lose the volume and will need to restore from backups once the disk(s) have been replaced.

    Hope that helps,

    Grace

  • It's not possible to clean a dissociated plex, by definition:

    --------------------
    A plex is in a CLEAN state when it is known to contain a consistent copy (mirror) of the volume contents and an operation has disabled the volume. As a result, when all plexes of a volume are clean, no action is required to guarantee that the plexes are identical when that volume is started.
    --------------------

    ie: a dissociated plex is not attached/associated with any volume, so it cannot be clean

    Per previous post, you have detached the plex, so now you need to reattach it (via vxplex att). However, you need to ensure the "good"/active plex is readable/not issuing any read errors, otherwise it won't be able to attach, as it won't be able to read the contents from the active plex to mirror it to the plex being re-attached.

  • Thanks Grace and Gaurav!!

     

    Disk mams06-lm was the culprit. I did an analyze refresh on it in format and then removed the failing flag from it. After this the plex is now synching properly with vxplex att.

     

    Thanks again!!