Do we need fbsync can do a complete sync if Primary Site Disk fails

I am surprised the fbsync command worked, but I would think there is a good chance your primary data is corrupt - I can see 2 possibilties:

1. VVR is intelligent to know that all data on the replaced disks (assuming it is not mirrored) will need to be synced so it marks the DCM map completely dirty for these subdisks so that all this data is synced
2. VVR is not this intelligent and the data on your new disks only contains the data changes that were written on the DR site and so is corrupt

I would run a space optimised snapshot on the primary site and mount the data to check, which probably  won't mount if the are corrupt.  You could also try to mount the volumes read-only ("-o ro" I think) on the Primary - this is not supported as if you try to read the files, they could be changing with VVR and these changes won't be in the primary nodes cache, but they should mount and if they don't mount, then your data is almost certainly corrupt.

If your data is corrupt, then just run a "vradmin -f stoprep" and vradmin -a startrep" to resync the data.

Mike

 

First thanks for your words Mike.

One question raising in my mind. As I said that I UP my Application at DR Site(did not switched over) and while UP my Application at DR Site I feel that my Client restarted the Primary Site Machine as well, means that might be a Take Over, Because before execute the fbsync command I checked the repstatus and saw the Primary - Primary configuration.

So Let me think on your words in which you said that :

If your data is corrupt, then just run a "vradmin -f stoprep" and vradmin -a startrep" to resync the data.

 

is not the above command failed as the configuration was Primary - Primary configuration.

Supposed Environment:

SFHA/DR setup

Primary Site = Two Cluster Node's

DR Site = One Cluster Node

What could be the step(s) if suppose Primary Site both Server's shutdown abnormally due to power outage and suppose we start the Application at DR Site (via DR Site Cluster Java Console.)

Now when the Power restored and Primary Site Servers are UP, it found that the Primary Site central storage/SAN Lun's which shared by both nodes dead/corrupted/faulty. So we replaced the SAN Luns via the help of option 4 and 5 of vxdiskadm. Now our next activity will be that we have to synchronized our Primary Site.

Now when we see the status of Replication via repstatus command we can see a Primary - Primary configuration. In this situation what will be the roadmap ?

Everyone's comment will be appriciated please.

Zahid,

Suggested reading:

Veritas Volume Replicator 5.0MP3 Linux Administrator's guide -> Transferring the Primary

https://sort.symantec.com/public/documents/sf/5.0MP3/linux/html/vvr_admin/ch08.htm

or https://d1mj3xqaoh14j0.cloudfront.net/public/documents/sf/5.0MP3/linux/pdf/vvr_admin.pdf (p239)

Particularly the sections:

Taking over from an original Primary (p247)

https://sort.symantec.com/public/documents/sf/5.0MP3/linux/html/vvr_admin/ch08s03.htm
--------------------
The takeover procedure involves transferring the Primary role from an original Primary to a Secondary. When the original Primary fails or is destroyed because of a disaster, the takeover procedure enables you to convert a consistent Secondary to a Primary. The takeover of a Primary role by a Secondary is useful when the Primary experiences unscheduled downtimes or is destroyed because of a disaster.
--------------------

Failing back to the original Primary (p255)

https://sort.symantec.com/public/documents/sf/5.0MP3/linux/html/vvr_admin/ch08s04.htm
--------------------
After an unexpected failure, a failed Primary host might start up to find that one of its Secondaries has been promoted to a Primary by a takeover. This happens when a Secondary of this Primary has taken over the Primary role because of the unexpected outage on this Primary. The process of transferring the role of the Primary back to this original Primary is called failback.
--------------------

Also: Veritas Cluster Server Agents for Veritas Volume Replicator Configuration Guide (5.0MP3 Linux)

https://sort.symantec.com/public/documents/sf/5.0MP3/linux/html/vvr_agents_config/index.html

https://d1mj3xqaoh14j0.cloudfront.net/public/documents/sf/5.0MP3/linux/pdf/vvr_agents_config.pdf

(probably start at Overview of how to configure VVR in a VCS environment, and work from there)

For other versions/platforms, as always, look for the relevant documents on http://sort.symantec.com/documents - the concepts / overall procedures are basically the same though.

It is ok to run fbsync, if vradmin lets you run fbysnc after you have replaced some LUNs, because as you say you need to sort out the Primary-Primary config, but as I said earlier, I think there is a good chance that fbsync won't sync all the data on the replaced LUNs.

Before you run fbsync, I believe if you run vxrlink status at the old primary (where you replaced the LUNs), it should tell you how many bytes are outstanding on the DCM, so if fbsync is going to work, the DCM would have to contain at least the size of the replaced LUNs and if it doesn't the vradmin fbsync is not that intelligent as in point 2 of my first post.  I guess vradmin MIGHT mark all data dirty on DCM for replaced LUNs, as part of running fbsync so you could also run vxrlink status on DR before running fbsync and then after fbsync is run, these 2 DCMs are merged, so you can check if the dirty data from added LUNs is added, then, but I very much doubt it.  If you verify fbsync has not taken replaced volumes in to consideration, then there is no point letting fbsync finish, just run "vradmin -f stoprep" and vradmin -a startrep" to sync from scratch.

Mike

@ Mike

Do you see any complication if the Mount resource is already Online at DR Site and we execute the mount command from Primary Site with read only option ?

From Primary and Secondary Site the Mount Resource is the Parent of RVG-PRI resource, So in any case will the Mount Resource automatically Probe  ? and if yes, is'nt the Child Resource (which is RVG-PRI in  our case) will be Online automatically as the Parent(MOUNT Resource) got Online ?

Means I just want to make sure that in any case the RVG-PRI will be Offline at Primary Site as its already online at DR Site.

You could mount using a temporary mount point so VCS shouldn't recognise the resource as online.   In theory the worst that coulld happen is that you get a currency violation and VCS will umount the read-only mount (it won't online dependent resources), but I would freeze application service groups on both sides for a "belts and braces" approach.

Mike

Hmmm let me share with my OS team(as my Client cant afford extra down time in terms of Cuncurrency Voilation as it has around 100,000 of users) and do it this way and will share the result.

- Freeze Application Service Group on both Sites (Primary and DR).

- Try to mount the Volume to a temporary place like /mnt. for example

  #mount -t vxfs -o ro /dev/vx/dsk/DG/Volume /mnt

You can't run fsck on a readonly filesystem.  It could be you just need to specify filesystem type, or it maybe that the inode table was been updated on the primary while you were trying to mount on the secondary (this is why this method is not officially supported).  If you have an Enterprise license and a little free space in the diskgroup, then the better method is to run a space optimised snapshot and mount this.

Mike

I just did stoprep and then startrep and when sync completed, now trying to run the mount command with readonly but feel that some issue. See the below for reference:

[root@xxxxxx ~]# mount -o ro /dev/vx/dsk/DG/Volume /mnt/

mount: wrong fs type, bad option, bad superblock on /dev/vx/dsk/DG/Volume,

       missing codepage or other error

       In some cases useful info is found in syslog - try

       dmesg | tail  or so

 

#tail -f /var/logs/messages

Nov 28 15:43:06 xxxxxx kernel: lost page write due to I/O error on VxVM65533

Nov 28 15:43:06 xxxxxx kernel: JBD: recovery failed

Nov 28 15:43:06 xxxxxx kernel: EXT3-fs: error loading journal.

 

I think that we need to run fsck but in a situation where I have ext3 filesystem. Any suggesstion what would be the syntax to run the fsck in my situation

like

fsck -o full -y /dev/vx/dsk/DG/volume

OR

 

like fsck -f -y /dev/vx/dsk/DG/volume