01-20-2014 04:06 AM
Hi, I have an disk group imported on a certain box and one logical volume has metada corrupted. Is there a way to recover the corrupted metada of that logical volume in production and the real data to be still there?Maybe with filesystem mounted?Or I should unmounts this file system? If I try to recreate from the beginning this logical volume with vxassist make,then the real data will be affected?is this a good way to have logical volume back online and the real data still there? vxmend helps? tnx so much. marius
Solved! Go to Solution.
01-20-2014 07:22 PM
Hi Marius,
Yes, once disk is initialized in Veritas, vxvm will put a private region on the disk which will contain a header or details of that disk. Some of the disks private region in disksgroup contains the whole diskgroup configuration, if you want to see which disks in a diskgroup contains whole diskgroup configuration, you need to execute "vxdg list <diskgorup>"
for e.g see the output below
# vxdg list codb_racdg049
flags: shared cds
version: 140
local-activation: shared-write
cluster-actv-modes: nbcodb01=sw nbcodb02=sw nbcodb03=sw nbcodb04=sw
detach-policy: global
dg-fail-policy: dgdisable
copies: nconfig=2 nlog=2
config disk c10t10d3 copy 1 len=24072 disabled
config disk c10t10d4 copy 1 len=24072 state=clean online <<<<<<<<<<< disk containing config copy
config disk c10t10d5 copy 1 len=24072 state=clean online <<<<<<<<<<<<disk containing config copy
Once you create a filesystem on a logical volume, each filesystem will have its own superblock.
Hope this answers.
G
01-20-2014 04:20 AM
Hi,
Couple of questions before suggesting further, what is the current state of volume in vxvm ? can you post a "vxprint -htg <diskgroup> <volume> " output ?
did you try mounting the volume & you are unable to mount the volume ? what is the error you are getting. Is this the only volume in diskgroup having problem or all the volumes in diskgroup have problem ?
vxmend, vxassist is way to recreate a volume, you can recreate the exact volume by specifying the right options like "privlen, publen, privslice, pubslice" with vxdisksetup command but note that this is just defining the volume structure. These steps are used if you lost your diskgroup accidently, or some how diskgroup configuration got corrupted etc however note, If there is a corruption in filesystem, recreating volume will not help.
If the volume is "enabled active" however ypou are unable to mount the filesystem on it, like any other filesystem, you will need to run "fsck" on it. I recommend to run a fsck with "-n" option first to ensure what changes fsck may make.
# fsck -F vxfs -o full -n /dev/vx/rdsk/<dg>/<vol>
G
01-20-2014 04:41 AM
01-20-2014 04:51 AM
As Gaurav suggested above, if there is a corruption in the filesystem alone, recreating volume wont help here. Provided you have the vxconfigbackup from a time when the metadata config was correct (see backups under/etc/vx/cbr) , for restoring/recreating the diskgroup metadata config (along with volume configs), you may also use vxconfigrestore command; this will not affect/alter the public region data.
01-20-2014 05:03 AM
Hello Marius,
Can you provide us with more details about the issue?
What exactly happened, how did you discover the metadata corruption (my feeling is rather that the corruption occured on the file system rather than on the volume).
Did this error come from the file system?
Is the file system still mounted and accessible?
Please attach related system messages to this thread.
Was there an outage in the SAN or similar that could have led to the corruption?
Please also provide us the output of #vxprint -g <diskgroup> -ht
Thanks,
Dan
01-20-2014 05:42 AM
01-20-2014 06:39 AM
Hi Marius,
How are you arriving to conclusion that only volume has metadata corrupted ? is the volume "enabled active" in vxprint -htg output as asked above ? If yes, then metadata of volume or diskgroup is not corrupted ..
When you are saying, filesystem is fine , how did u arrive to that conclusion, did u tried mounting the volume & filesystem to mount point ?
can u paste followng outputs to get more clarity
# vxprint -qhtg <diskgroup> <volume>
# df -h |grep -i <volume>
# fstyp -v /dev/vxrds/<diskgroup>/<vol>
G
01-20-2014 07:01 AM
01-20-2014 08:19 AM
Hi,
Theoritically, there are two different scenarios which might be possible in the problem you are describing as "metadata corruption"
1. Filesystem metadata corruption: In this case, the volume beneath is fine & is in "Enabled Active" state however the there were some IO errors (hard or soft) or there was some problem writing to data region, there are some possibilities where filesystem may get corrupted. In this case, you will need to execute fsck the filesystem with right options to fix the problem. The problem may fix or may not depending on extent of filesystem corruption.
2. VxVM private region corruption: vxvm disks have 2 regions i.e private region & public region. If in case diskgroup is lost or destroyed or corrupted, you may not be able to see the diskgroups or volumes in it. Another possibility is someone accidently deleted a volume. In such situations you will need to create the diskgroup or volumes manually. There are multiple outputs that can help here, for e.g if you have configuration backup in /etc/vx/cbr/bk or if you have veritas explorer or if you have a "vxprint -mvphsr" output , you can easily rebuild the diskgroup & the volumes.
The steps what I have explained in my previous comment like initializing the disk again with right values like puboffset, privoffset, privslice, pubslice or recreating the volume using vxmake command can fix above scenarios where volume was lost for some reason.
Now, if in case someone has touched public region, for e.g lets take for a scenario that storage lun beneath the volume was accidently decommissioned & assigned to other server & other server did write on the Lun. There are high chances that public region has been damaged which removes the chances of restoring the data by reassigning the Lun or recreating the volume once Lun is assigned back. In such cases, restoring from backup would be the choice.
Simple point to remember is, vxmake, recreating the volume can only help if underneath storage Lun or disk is untouched or intact & presumably superblock of filesystem on the public region is intact as well, there are high chances you can get the data.
G
01-20-2014 09:55 AM
I only wish to know as a theoretical case how to fix this issue.
PLEASE!! No theory!!
Real life scenario depends on what exactly happed and the error message(s) when trying to fsck or mount the filesystem.
Recovering metadata with fsck is no guarantee that data will be intact.
Depending on what happened, you may need to recover from backup.
01-20-2014 10:04 AM
01-20-2014 07:22 PM
Hi Marius,
Yes, once disk is initialized in Veritas, vxvm will put a private region on the disk which will contain a header or details of that disk. Some of the disks private region in disksgroup contains the whole diskgroup configuration, if you want to see which disks in a diskgroup contains whole diskgroup configuration, you need to execute "vxdg list <diskgorup>"
for e.g see the output below
# vxdg list codb_racdg049
flags: shared cds
version: 140
local-activation: shared-write
cluster-actv-modes: nbcodb01=sw nbcodb02=sw nbcodb03=sw nbcodb04=sw
detach-policy: global
dg-fail-policy: dgdisable
copies: nconfig=2 nlog=2
config disk c10t10d3 copy 1 len=24072 disabled
config disk c10t10d4 copy 1 len=24072 state=clean online <<<<<<<<<<< disk containing config copy
config disk c10t10d5 copy 1 len=24072 state=clean online <<<<<<<<<<<<disk containing config copy
Once you create a filesystem on a logical volume, each filesystem will have its own superblock.
Hope this answers.
G
01-21-2014 12:04 AM
01-21-2014 12:15 AM
Hi,
Logically yes, but I don't have much experience in doing this procedure on LVM ... but as said logically, if the disks data region is untouched, yes you should be able to get volume back by recreating volumes... again, if Filesystem on top of it has corruption, it may need fsck
G
01-21-2014 12:31 AM
01-21-2014 12:54 AM
You opened post as "I don't need a solution" so I have changed to "I need a solution", so now you can mark which comment helped you the most as solution
01-21-2014 04:00 AM
Please mark one of Gaurav's comment as solution as it was not me who solved your issue.
Mike