vxvm metadata recover

[tanislavm]

Hi, I have an disk group imported on a certain box and one logical volume has metada corrupted. Is there a way to recover the corrupted metada of that logical volume in production and the real data to be still there?Maybe with filesystem mounted?Or I should unmounts this file system? If I try to recreate from the beginning this logical volume with vxassist make,then the real data will be affected?is this a good way to have logical volume back online and the real data still there? vxmend helps? tnx so much. marius

[Gaurav_S]

Hi,

Couple of questions before suggesting further, what is the current state of volume in vxvm ? can you post a "vxprint -htg <diskgroup> <volume> " output ?

did you try mounting the volume & you are unable to mount the volume ? what is the error you are getting. Is this the only volume in diskgroup having problem or all the volumes in diskgroup have problem ?

vxmend, vxassist is way to recreate a volume, you can recreate the exact volume by specifying the right options like "privlen, publen, privslice, pubslice" with vxdisksetup command but note that this is just defining the volume structure. These steps are used if you lost your diskgroup accidently, or some how diskgroup configuration got corrupted etc  however note, If there is a corruption in filesystem, recreating volume will not help.

If the volume is "enabled active" however ypou are unable to mount the filesystem on it, like any other filesystem, you will need to run "fsck" on it. I recommend to run a fsck with "-n" option first to ensure what changes fsck may make.

# fsck -F vxfs -o full -n /dev/vx/rdsk/<dg>/<vol>

G

[tanislavm]

Hi, I am able to see the output of vxprint -htg and only one logical volume has metadata corrupted. If this meta data is corrupted I assume I am not able to mount the logical volume.right? I like only to know theoretically now what I should to perform in this case to recover the logical volume. So for only this volume if I try to recreate the meta data with vxassist I will find the real data also? Or using vxmend,the same question. Could I perform those things in production? tnx so much, marius

[rsharma1]

As Gaurav suggested above, if there is a corruption in the filesystem alone, recreating volume wont help here. Provided you have the vxconfigbackup from a time when the metadata config was correct (see backups under/etc/vx/cbr) , for restoring/recreating the diskgroup metadata config (along with volume configs), you may also use vxconfigrestore command; this will not affect/alter the public region data.

[Daniel_Matheus]

Hello Marius,

Can you provide us with more details about the issue?

What exactly happened, how did you discover the metadata corruption (my feeling is rather that the corruption occured on the file system rather than on the volume).

Did this error come from the file system?

Is the file system still mounted and accessible?

Please attach related system messages to this thread.

Was there an outage in the SAN or similar that could have led to the corruption?

Please also provide us the output of #vxprint -g <diskgroup> -ht

Thanks,
Dan

[tanislavm]

Hi, I inly need a way to recover only one single logical volume with metadata corrupted,the rest of the logical volumes within disk group are fine.I presume that the file system is fine,so those are the circumstances. tnx so much, marius

[Gaurav_S]

Hi Marius,

How are you arriving to conclusion that only volume has metadata corrupted ? is the volume "enabled active" in vxprint -htg output as asked above ?  If yes, then metadata of volume or diskgroup is not corrupted ..

When you are saying, filesystem is fine , how did u arrive to that conclusion, did u tried mounting the volume & filesystem to mount point ?

can u paste followng outputs to get more clarity

# vxprint -qhtg <diskgroup> <volume>

# df -h |grep -i <volume>

# fstyp -v /dev/vxrds/<diskgroup>/<vol>

G

[tanislavm]

Hi, I only wish to know as a theoretical case how to fix this issue. tnx so much, marius

[Gaurav_S]

Hi,

Theoritically, there are two different scenarios which might be possible in the problem you are describing as "metadata corruption"

1. Filesystem metadata corruption:  In this case, the volume beneath is fine & is in "Enabled Active" state however the there were some IO errors (hard or soft) or there was some problem writing to data region, there are some possibilities where filesystem may get corrupted. In this case, you will need to execute fsck the filesystem with right options to fix the problem. The problem may fix or may not depending on extent of filesystem corruption.

2. VxVM private region corruption: vxvm disks have 2 regions i.e private region & public region. If in case diskgroup is lost or destroyed or corrupted, you may not be able to see the diskgroups or volumes in it. Another possibility is someone accidently deleted a volume. In such situations you will need to create the diskgroup or volumes manually. There are multiple outputs that can help here, for e.g if you have configuration backup in /etc/vx/cbr/bk or if you have veritas explorer or if you have a "vxprint -mvphsr" output , you can easily rebuild the diskgroup & the volumes.

The steps what I have explained in my previous comment like initializing the disk again with right values like puboffset, privoffset, privslice, pubslice or recreating the volume using vxmake command can fix above scenarios where volume was lost for some reason.

Now, if in case someone has touched public region, for e.g lets take for a scenario that storage lun beneath the volume was accidently decommissioned & assigned to other server & other server did write on the Lun. There are high chances that public region has been damaged which removes the chances of restoring the data by reassigning the Lun or recreating the volume once Lun is assigned back. In such cases, restoring from backup would be the choice.

Simple point to remember is, vxmake, recreating the volume can only help if underneath storage Lun or disk is untouched or intact & presumably superblock of filesystem on the public region is intact as well, there are high chances you can get the data.

G

[Marianne]

I only wish to know as a theoretical case how to fix this issue.

PLEASE!! No theory!!

Real life scenario depends on what exactly happed and the error message(s) when trying to fsck or mount the filesystem.

Recovering metadata with fsck is no guarantee that data will be intact.

Depending on what happened, you may need to recover from backup.

[tanislavm]

Hi G, Your comments are indeed of very high quality. Only to clarify something. When I create vxvm logical volume on a physical disk added under vxvm control,then the metadata in private region comprise information about each logical volume(i.e this volume belong to a certain disk group and is part of a certain raid).right? Then each logical volume has an supenrblock (if I have made an file ystem on it).right? tnx so much, marius

[Gaurav_S · Accepted solution]

Hi Marius,

Yes, once disk is initialized in Veritas, vxvm will put a private region on the disk which will contain a header or details of that disk. Some of the disks private region in disksgroup contains the whole diskgroup configuration, if you want to see which disks in a diskgroup contains whole diskgroup configuration, you need to execute "vxdg list <diskgorup>"

for e.g see the output below

# vxdg list codb_racdg049
flags:     shared cds
version:   140
local-activation: shared-write
cluster-actv-modes: nbcodb01=sw nbcodb02=sw nbcodb03=sw nbcodb04=sw
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=2 nlog=2
config disk c10t10d3 copy 1 len=24072 disabled
config disk c10t10d4 copy 1 len=24072 state=clean online <<<<<<<<<<< disk containing config copy
config disk c10t10d5 copy 1 len=24072 state=clean online  <<<<<<<<<<<<disk containing config copy

Once you create a filesystem on a logical volume, each filesystem will have its own superblock.

Hope this answers.

G

[tanislavm]

Hi G, The last thing is below. I think that in lvm case also I could recover an logical volume corrupted metadata by lvcreate and I also find the real data if I mount that logical volume.right? tnx so much, marius

[Gaurav_S]

Hi,

Logically yes, but I don't have much experience in doing this procedure on LVM ... but as said logically, if the disks data region is untouched, yes you should be able to get volume back by recreating volumes... again, if Filesystem on top of it has corruption, it may need fsck

G

[tanislavm]

Hi, I do not have the button below "mark as solution" so please close this in my behalf.tnx so much

[mikebounds]

You opened post as "I don't need a solution" so I have changed to "I need a solution", so now you can mark which comment helped you the most as solution

[mikebounds]

Please mark one of Gaurav's comment as solution as it was not me who solved your issue.

Mike