cancel
Showing results for 
Search instead for 
Did you mean: 

vxevac session got terminated in the middle

sunguru
Level 4
Hi

I was evacuating disk stage_eidr_db_dg03 to stage_eidr_db_dg07 using command line and my putty session got terminated while it was running
pls advise the steps to remove the 6% plex and restart the evac process without any impact to volume and data or any other safe methods.


v  stage_eidr_sdb1_vol -     ENABLED  ACTIVE   1363148800 SELECT  -        fsgen
pl %6           stage_eidr_sdb1_vol ENABLED(SPARSE) TEMPRM 1155530752 CONCAT - WO
sd stage_eidr_db_dg07-01 %6  stage_eidr_db_dg07 0 524205824 631324928 c2t5006048AD53107C3d102 ENA
pl stage_eidr_sdb1_vol-01 stage_eidr_sdb1_vol ENABLED ACTIVE 1363148800 CONCAT - RW
sd stage_eidr_db_dg05-01 stage_eidr_sdb1_vol-01 stage_eidr_db_dg05 0 524205824 0 c2t5006048AD53107C3d100 ENA
sd stage_eidr_db_dg06-03 stage_eidr_sdb1_vol-01 stage_eidr_db_dg06 14680064 107119104 524205824 c2t5006048AD53107C3d101 ENA
sd stage_eidr_db_dg03-01 stage_eidr_sdb1_vol-01 stage_eidr_db_dg03 0 524205824 631324928 c2t5006016039A03E36d4 ENA
sd stage_eidr_db_dg04-01 stage_eidr_sdb1_vol-01 stage_eidr_db_dg04 0 207618048 1155530752 c3t5006016839A03E36d93 ENA




test-lab-1:- modinfo |grep -i vx
 36  1354270  3a728 289   1  vxdmp (VxVM 5.0MP1-2007-01-22a: DMP Dr)
 38 7c002000 33fd48 290   1  vxio (VxVM 5.0MP1-2007-01-22a I/O dri)
 40  138ab38    d48 291   1  vxspec (VxVM 5.0MP1-2007-01-22a control)
214 7b2b1068    c30 292   1  vxportal (VxFS 5.0_REV-5.0MP1u_sol portal)
215 7aa00000 1bc8a0  21   1  vxfs (VxFS 5.0_REV-5.0MP1u_sol SunOS )
218  13b5360  4fe28 296   1  vxfen (VRTS Fence 5.0MP1)
234 7b3d6000   a2c8 293   1  fdd (VxQIO 5.0_REV-5.0MP1u_sol Quick)


1 ACCEPTED SOLUTION

Accepted Solutions

Gaurav_S
Moderator
Moderator
   VIP    Certified

hmm.. thats unlucky,...  as per error message also


V-5-1-809

VxVM vxplex ERROR V-5-1-809 Plex plex in volume volume is locked by another utility.

The vxplex command fails because a previous operation to attach a plex did not complete. The vxprint command should show that one or both of the temporary and persistent utility fields (TUTIL0 and PUTIL0) of the volume and one of its plexes are set.

If the vxtask list command does not show a task running for the volume, use the vxmend command to clear the TUTIL0 and PUTIL0 fields for the volume and all its components for which these fields are set:

# vxmend -g diskgroup clear all volume plex ...

The same thing we did manually with vxedit command....

Is this node part of any cluster or is standalone host ? Also are you using CVM ? just want to know if we have option to restart vxconfigd daemon.... could be a possibility that state of plex & subdisk are holded in kernel & are not releasing...

Gaurav


View solution in original post

12 REPLIES 12

Gaurav_S
Moderator
Moderator
   VIP    Certified
Hello Sunguru...

I hope there is nothing running in vxtask list.....

You have a concat volume, In that case, straight choice would be to remove the %6 plex .. That anyways is a sparse plex so shouldn't have any consistent data..

# vxedit -g <diskgroup> -rf rm %6

Once this is done, you can retrigger the vxevac .... Before triggering vxevac, I would suggest, it would be great if DG has some free space & also /tmp also has some free space...


Gaurav

sunguru
Level 4
pls clarify if we need to disassociate and detach the %6 plex before running below command as we get the below error.

hostname # vxedit -g stage_eidr_db_dg -rf rm %6
VxVM vxedit ERROR V-5-1-818 Plex %6 is associated, cannot remove

Gaurav_S
Moderator
Moderator
   VIP    Certified
Well you can try offlining first (though not sure if it will offline) but worth to try..... then try disassociating it

# vxmend -g <diskgroup> off %6
# vxplex -g <diskgroup> dis %6

Then try running the same command...


Gaurav

Gaurav_S
Moderator
Moderator
   VIP    Certified

Also can u paste following:

# vxprint -g <diskroup> 


I would like to see if any temp flags are set on the plex..

Gaurav

sunguru
Level 4
hostname#vxprint -g stage_eidr_db_dg
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg stage_eidr_db_dg stage_eidr_db_dg - -       -        -        -       -
dm stage_eidr_db_dg03 c2t5006016039A03E36d4s2 - 524205824 - -    -       -
dm stage_eidr_db_dg04 c3t5006016839A03E36d93s2 - 268337152 - -   -       -
dm stage_eidr_db_dg05 c2t5006048AD53107C3d100s2 - 534683488 - -  -       -
dm stage_eidr_db_dg06 c2t5006048AD53107C3d101s2 - 534683488 - -  -       -
dm stage_eidr_db_dg07 c2t5006048AD53107C3d102s2 - 534683488 - -  -       -
dm stage_eidr_db_dg08 c2t5006048AD53107C3d103s2 - 534683488 - -  -       -
v  eidr_sfra3   fsgen        ENABLED  2097152  -        ACTIVE   -       -
pl eidr_sfra3-01 eidr_sfra3  ENABLED  2097152  -        ACTIVE   -       -
sd stage_eidr_db_dg06-01 eidr_sfra3-01 ENABLED 2097152 0 -       -       -
v  stage_eidr_oraagent_vol fsgen ENABLED 12582912 -     ACTIVE   -       -
pl stage_eidr_oraagent_vol-01 stage_eidr_oraagent_vol ENABLED 12582912 - ACTIVE - -
sd stage_eidr_db_dg06-02 stage_eidr_oraagent_vol-01 ENABLED 12582912 0 - - -
v  stage_eidr_sdb1_vol fsgen ENABLED  1363148800 -      ACTIVE   ATT1    -
pl %6           stage_eidr_sdb1_vol ENABLED(SPARSE) 1155530752 - TEMPRM SDMVTMP -
sd stage_eidr_db_dg07-01 %6  ENABLED  524205824 631324928 -      SDMVDST -
pl stage_eidr_sdb1_vol-01 stage_eidr_sdb1_vol ENABLED 1363148800 - ACTIVE SDMV1 -
sd stage_eidr_db_dg05-01 stage_eidr_sdb1_vol-01 ENABLED 524205824 0 - -  -
sd stage_eidr_db_dg06-03 stage_eidr_sdb1_vol-01 ENABLED 107119104 524205824 - - -
sd stage_eidr_db_dg03-01 stage_eidr_sdb1_vol-01 ENABLED 524205824 631324928 - SDMVSRC -
sd stage_eidr_db_dg04-01 stage_eidr_sdb1_vol-01 ENABLED 207618048 1155530752 - - -
v  stage_eidr_sfra1_vol fsgen ENABLED 159383552 -       ACTIVE   -       -
pl stage_eidr_sfra1_vol-01 stage_eidr_sfra1_vol ENABLED 159383552 - ACTIVE - -
sd stage_eidr_db_dg06-04 stage_eidr_sfra1_vol-01 ENABLED 159383552 0 - - -
 

sunguru
Level 4
hostname# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS



hostname# vxmend -g stage_eidr_db_dg off %6
VxVM vxmend ERROR V-5-1-809 Plex %6 in volume stage_eidr_sdb1_vol is locked by another utility

Gaurav_S
Moderator
Moderator
   VIP    Certified


Hi Sunguru,

As you can see there are "tutil0" flags set on many components... (total 4 flags)

pl %6           stage_eidr_sdb1_vol ENABLED(SPARSE) 1155530752 - TEMPRM SDMVTMP -
sd stage_eidr_db_dg07-01 %6  ENABLED  524205824 631324928 -      SDMVDST -
pl stage_eidr_sdb1_vol-01 stage_eidr_sdb1_vol ENABLED 1363148800 - ACTIVE SDMV1 -
sd stage_eidr_db_dg03-01 stage_eidr_sdb1_vol-01 ENABLED 524205824 631324928 - SDMVSRC -

I believe we would need to clear those flags to continue further...

# vxedit -g <diskgroup> set  tutil0=""  <plex/subdisk>   (we are clearing flags by entering a blank character  "")

Try removing first from %6 plex & subdisk stage_eidr_db_dg07-0, once the flags are cleared you should be able to remove.... If still it doesn't work, try removing flags from stage_eidr_sdb1_vol-01 & stage_eidr_db_dg03-01 as well..

Gaurav

Dev_Roy
Level 6
Accredited Certified
Seems you have a good mirror volume for the volume is trouble i.e. stage_eidr_sdb1_vol-01, hence not to panic as even if fixing the first plex does not work then you can disassociate and remove the faulty plex and then you can add the mirror the mirrored volume ( stage_eidr_sdb1_vol-01) on the new disk where you actually tried to move the data of the earlier plex i.e. stage_eidr_sdb1_vol

First try ( not sure if this going to help)
1.> vxrecover -bsE -g <disk group name>
2.> vxtask list  --> and check if something is going on in the back ground.
3.> vxprint -qhtrg <disk group name>   --> check if any change in the plex status.

If above does not work then try:

1.> vxmend -g <dg name> -o force off <plex name>  --> This will force offline the plex
2.> vxsd -g <dg name> -o force dis stage_eidr_db_dg07-01
3.> vxplex -g <disk group name> -o force dis %6
4.> vxedit -g stage_eidr_db_dg -rf rm %6
5.> vxedit -g stage_eidr_db_dg -rf rm stage_eidr_db_dg07-01
6.> vxprint -qhtrg <disk group name>  --> Check what is the status of the plex.

If above commands throws any error then let us know and if it works then just remirror the volume with the desired disk.

Regards,
Dev


sunguru
Level 4


Hello

Cleared  all  "tutils" to "" and but no luck , pls advice


hostname# vxprint -htg stage_eidr_db_dg
DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         RVG          KSTATE   STATE    NVOLUME
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

dg stage_eidr_db_dg default  default  32000    1243543272.831.bos-stage-22
dm stage_eidr_db_dg03 c2t5006016039A03E36d4s2 auto 65536 524205824 -
dm stage_eidr_db_dg04 c3t5006016839A03E36d93s2 auto 73471 268337152 -
dm stage_eidr_db_dg05 c2t5006048AD53107C3d100s2 auto 65536 534683488 -
dm stage_eidr_db_dg06 c2t5006048AD53107C3d101s2 auto 65536 534683488 -
dm stage_eidr_db_dg07 c2t5006048AD53107C3d102s2 auto 65536 534683488 -
dm stage_eidr_db_dg08 c2t5006048AD53107C3d103s2 auto 65536 534683488 -
v  eidr_sfra3   -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl eidr_sfra3-01 eidr_sfra3  ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd stage_eidr_db_dg06-01 eidr_sfra3-01 stage_eidr_db_dg06 0 2097152 0 c2t5006048AD53107C3d101 ENA
v  stage_eidr_oraagent_vol - ENABLED  ACTIVE   12582912 SELECT    -        fsgen
pl stage_eidr_oraagent_vol-01 stage_eidr_oraagent_vol ENABLED ACTIVE 12582912 CONCAT - RW
sd stage_eidr_db_dg06-02 stage_eidr_oraagent_vol-01 stage_eidr_db_dg06 2097152 12582912 0 c2t5006048AD53107C3d101 ENA
v  stage_eidr_sdb1_vol -     ENABLED  ACTIVE   1363148800 SELECT  -        fsgen
pl %6           stage_eidr_sdb1_vol DISABLED(SPARSE) OFFLINE 1155530752 CONCAT - WO
sd stage_eidr_db_dg07-01 %6  stage_eidr_db_dg07 0 524205824 631324928 c2t5006048AD53107C3d102 ENA
pl stage_eidr_sdb1_vol-01 stage_eidr_sdb1_vol ENABLED ACTIVE 1363148800 CONCAT - RW
sd stage_eidr_db_dg05-01 stage_eidr_sdb1_vol-01 stage_eidr_db_dg05 0 524205824 0 c2t5006048AD53107C3d100 ENA
sd stage_eidr_db_dg06-03 stage_eidr_sdb1_vol-01 stage_eidr_db_dg06 14680064 107119104 524205824 c2t5006048AD53107C3d101 ENA
sd stage_eidr_db_dg03-01 stage_eidr_sdb1_vol-01 stage_eidr_db_dg03 0 524205824 631324928 c2t5006016039A03E36d4 ENA
sd stage_eidr_db_dg04-01 stage_eidr_sdb1_vol-01 stage_eidr_db_dg04 0 207618048 1155530752 c3t5006016839A03E36d93 ENA
v  stage_eidr_sfra1_vol -    ENABLED  ACTIVE   159383552 SELECT   -        fsgen
pl stage_eidr_sfra1_vol-01 stage_eidr_sfra1_vol ENABLED ACTIVE 159383552 CONCAT - RW
sd stage_eidr_db_dg06-04 stage_eidr_sfra1_vol-01 stage_eidr_db_dg06 121799168 159383552 0 c2t5006048AD53107C3d101 ENA

hostname#vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS

hostname#vxedit -g stage_eidr_db_dg -rf rm %6
VxVM vxedit ERROR V-5-1-818 Plex %6 is associated, cannot remove

hostname#_# vxplex -g stage_eidr_db_dg dis %6
VxVM vxplex ERROR V-5-1-1225 Volume stage_eidr_sdb1_vol is locked by another utility

 

Gaurav_S
Moderator
Moderator
   VIP    Certified

hmm.. thats unlucky,...  as per error message also


V-5-1-809

VxVM vxplex ERROR V-5-1-809 Plex plex in volume volume is locked by another utility.

The vxplex command fails because a previous operation to attach a plex did not complete. The vxprint command should show that one or both of the temporary and persistent utility fields (TUTIL0 and PUTIL0) of the volume and one of its plexes are set.

If the vxtask list command does not show a task running for the volume, use the vxmend command to clear the TUTIL0 and PUTIL0 fields for the volume and all its components for which these fields are set:

# vxmend -g diskgroup clear all volume plex ...

The same thing we did manually with vxedit command....

Is this node part of any cluster or is standalone host ? Also are you using CVM ? just want to know if we have option to restart vxconfigd daemon.... could be a possibility that state of plex & subdisk are holded in kernel & are not releasing...

Gaurav


sunguru
Level 4
Hi Gaurav

Thanks a lot for your excellent support , below command resolved the problem

have a goodone.

vxmend -g diskgroup clear all volume plex ...





Gaurav_S
Moderator
Moderator
   VIP    Certified

Good news... enjoy

so there was some more flag which was in hold & got cleared by vxmend..

Gaurav