vxconfigbackup warning

[Jbert]

Whenever I try to backup the configuration, warning is shown as below. Also when i try to restore it, the operation fails.

/etc/vx/bin/vxconfigbackup -l /opt/configbackup
Start backing up diskgroup abcdg to /opt/configbackup/abcdg.1294657147.185.sys1_01 ...
VxVM vxconfigbackup WARNINIG V-5-2-3608 On disk diskgroup configuration for diskgroup abcdg is invalid, please check this dg
VxVM  NOTICE V-5-2-3100 Backup complete for diskgroup: abcdg

ls /opt/configbackup:

abcdg.1294657147.185.sys1_01

[Jbert]

What does this error mean and why this is shown?

[avsrini]

Hi Jbert,

Generally this means Diskgroup configuration copy saved on the disk is not matching

with the config copy on kernel. Was there any changes made to SAN recently?

Is Storage shared with other hosts?

Try vxconfigbackup after running # vxdg flush command.

This command will flush config copy from kernel to on-disk.

Regards

Srini

[Jbert]

Hi Srini,

I tried the vxdg flush command still the problem persists. The answer to your questions is 'No changes/sharing'. Now what?

Regards.

[Marianne]

Please post output of  'vxdg list abcdg'

Explanation of 'flush' option from man page:

      flush    Rewrites all  on-disk  structures  (disk  headers,
               configuration   copies,  and  kernel  log  copies)
               managed by VxVM for  the  specified  disk  groups.
               Also,  if  any  configuration  copies are disabled
               (for example, due to I/O failures),  this  command
               rewrites  those configuration copies, and attempts
               to enable them.

[Gaurav_S]

I too believe Srini is right, ondisk copy is not matching with kernel copy...

In addition to "vxdg list abcdg", can you also update if you are able to read the private region from the disk.. This can be done using:

# /etc/vx/diag.d/vxprivutil dumpconfig /dev/vx/rdmp/<da_name> 

da_name will be shown in the first coulmn of "vxdisk list" output (just in case you aren't aware).. You have to give the da_name of the disk which is part of abcdg moreover if multiple disks are involved in abcdg, checkout from "vxdg list abcdg" output, some disk would be showing "config state" as "clean online", run the vxprivutil on that disk..

Let us know if above command gives you any output..

Gaurav

[Jbert]

Hi Marianne,

Its a shared disk group. Stack: SFCFS

# vxdg list abcdg
Group:     abcdg
dgid:      1294657147.185.sys1_01
import-id: 33792.86
flags:     shared
version:   140
alignment: 8192 (bytes)
local-activation: shared-write
cluster-actv-modes: sys1_01=sw sys1_02=sw
ssb:            on
autotagging:    on
detach-policy: local
dg-fail-policy: leave
copies:    nconfig=default nlog=default
config:    seqno=0.1141 permlen=51576 free=51534 templen=20 loglen=4096
config disk emc0_01da copy 1 len=51576 disabled
config disk emc0_01db copy 1 len=51576 state=clean online
config disk emc0_01dc copy 1 len=51576 disabled
config disk emc0_01dd copy 1 len=51576 disabled
config disk emc0_01de copy 1 len=51576 disabled
config disk emc0_01df copy 1 len=51576 disabled
config disk emc0_01e0 copy 1 len=51576 state=clean online
config disk emc0_01e1 copy 1 len=51576 state=clean online
config disk emc0_01e2 copy 1 len=51576 disabled
config disk emc0_01e3 copy 1 len=51576 disabled
config disk tagmastore-usp0_1c8c copy 1 len=51576 disabled
config disk tagmastore-usp0_1c8d copy 1 len=51576 disabled
config disk tagmastore-usp0_1c8e copy 1 len=51576 disabled
config disk tagmastore-usp0_1c8f copy 1 len=51576 disabled
config disk tagmastore-usp0_1c9a copy 1 len=51576 disabled
config disk tagmastore-usp0_01ce copy 1 len=51576 disabled
config disk tagmastore-usp0_01cf copy 1 len=51576 disabled
config disk tagmastore-usp0_01d0 copy 1 len=51576 state=clean online
config disk tagmastore-usp0_01d1 copy 1 len=51576 state=clean online
config disk tagmastore-usp0_01d2 copy 1 len=51576 disabled
log disk emc0_01da copy 1 len=4096
log disk emc0_01db copy 1 len=4096 disabled
log disk emc0_01dc copy 1 len=4096
log disk emc0_01dd copy 1 len=4096 disabled
log disk emc0_01de copy 1 len=4096 disabled
log disk emc0_01df copy 1 len=4096 disabled
log disk emc0_01e0 copy 1 len=4096
log disk emc0_01e1 copy 1 len=4096 disabled
log disk emc0_01e2 copy 1 len=4096 disabled
log disk emc0_01e3 copy 1 len=4096 disabled
log disk tagmastore-usp0_1c8c copy 1 len=4096 disabled
log disk tagmastore-usp0_1c8d copy 1 len=4096 disabled
log disk tagmastore-usp0_1c8e copy 1 len=4096 disabled
log disk tagmastore-usp0_1c8f copy 1 len=4096 disabled
log disk tagmastore-usp0_1c9a copy 1 len=4096 disabled
log disk tagmastore-usp0_01ce copy 1 len=4096
log disk tagmastore-usp0_01cf copy 1 len=4096 disabled
log disk tagmastore-usp0_01d0 copy 1 len=4096 disabled
log disk tagmastore-usp0_01d1 copy 1 len=4096
log disk tagmastore-usp0_01d2 copy 1 len=4096 disabled
 

[Jbert]

Hi Gaurav,

The o/p is:

# /etc/vx/diag.d/vxprivutil dumpconfig /dev/vx/rdmp/tagmastore-usp0_01d0
VxVM vxprivutil ERROR V-5-1-1735 scan operation failed:
        Format error in disk private region

# /etc/vx/diag.d/vxprivutil dumpconfig /dev/vx/rdmp/emc0_01e0
VxVM vxprivutil ERROR V-5-1-1735 scan operation failed:
        Format error in disk private region

[Jbert]

I hope this could be useful:

# dd if=/dev/vx/rdmp/tagmastore-usp0_01d0 of=/dev/null bs=1024K count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.528346 seconds, 198 MB/s

# dd if=/dev/vx/rdmp/emc0_01e0 of=/dev/null bs=1024K count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.13942 seconds, 92.0 MB/s

[Gaurav_S]

ok, well that tells somehow private region on disk is corrupted...

Since its an SFCFS stack, I must ask, do you use IOFencing ?  I can see above that you have tried reading from the two disks & it wa successful however I believe you would have tried it from the same server..

Can you paste following outputs:

# modinfo  |grep -i vx

#/usr/sbin/vxfenadm -g /dev/vx/rdmp/tagmastore-usp0_01d0

#/usr/sbin/vxfenadm -r /dev/vx/rdmp/tagmastore-usp0_01d0

#/usr/sbin/vxfenadm -g /dev/vx/rdmp/emc0_01e0

#/usr/sbin/vxfenadm -r /dev/vx/rdmp/emc0_01e0

Also recommending:

-- Do not deport the DG as you may face problems in importing it.

-- If you can collect a veritas explorer thats too good, if not, minimum take following backups:

# vxdg list > vxdg_list.out

# vxdg list abcdg > vxdg_list_abcdg.out

# vxprint -qthg abcdg > vxprint_qthg.out

# vxprint -g abcdg -mvphsr > vxprint_mvphsr.out

# vxdisk list > vxdisk_list.out

# vxdisk -o alldgs -e list > vxdisk_alldgs_e_out

# vxdisk -s list > vxdisk_s_list.out

Just in case if private region needs to be rebuilt, above information would be handy..

Also, since there are two arrays here in the DG, was there any migration operation underway ? or the setup is like this from quite a long ?

Gaurav

[Jbert]

Hi Gaurav,

I tried dd command from both the nodes and its working fine.
 
vxfenadm o/p:

 # vxfenadm -d

I/O Fencing Cluster Information:
================================

 Fencing Protocol Version: 201
 Fencing Mode: Disabled
 Cluster Members:

        * 0 (sys1_01)
          1 (sys1_02)

 RFSM State Information:
        node   0 in state  8 (running)
        node   1 in state  8 (running)

  ------

# vxfenadm -g /dev/vx/rdmp/tagmastore-usp0_01d0

Reading SCSI Registration Keys...

Device Name: /dev/vx/rdmp/tagmastore-usp0_01d0
Total Number Of Keys: 0
No keys...

# vxfenadm -r /dev/vx/rdmp/tagmastore-usp0_01d0

Reading SCSI Reservation Information...

Device Name: /dev/vx/rdmp/tagmastore-usp0_01d0
Total Number Of Keys: 0
No keys...

  ------

 # vxfenadm -g /dev/vx/rdmp/emc0_01e0

Reading SCSI Registration Keys...

Device Name: /dev/vx/rdmp/emc0_01e0
Total Number Of Keys: 0
No keys...

# vxfenadm -r /dev/vx/rdmp/emc0_01e0

Reading SCSI Reservation Information...

Device Name: /dev/vx/rdmp/emc0_01e0
Total Number Of Keys: 0
No keys...

  ------

There was no migration and the setup quite old.

  ------

Files Attached...

[Jbert]

tar.gz file attached

[Jbert]

Hi Gaurav,

Any updates on this?

[Gaurav_S]

Hello,

Well to my understanding you have no choice but to re-build the private regions on the disk... Since the DG is up at the moment (& just config backup failing) .. its up to you if you want to rebuild it at this stage or later stage as it will require an outage.

Once the DG is deported, I am not very confident that diskgroup will be imported successfully again without re-constructing the private region. Since you have already ran vxdg flush that means kernel copy is flushed to ondisk copy & even that is not giving a readable private region.

For how to rebuild the diskgroup, you can get idea from this post:

https://www-secure.symantec.com/connect/forums/restore-dg

Article http://www.symantec.com/business/support/index?page=content&id=TECH22298

is also helpful to understand how will vxdisksetup command needs to be used.

Gaurav

[Jbert]

I ll try it out and respond...