secondary rvg unknown

[symsonu]

Hello,

I did the full sync and now the status is as below

root@atmas1o:/ericsson/hagcs/log# vradmin -l printrvg
Replicated Data Set: ossrvg
Primary:
        HostName: pallini-ossrvg        <localhost>
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_metamorphosi_ossrvg, detached=on, synchronous=off
Secondary:
        HostName: metamorphosi-ossrvg
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_pallini_ossrvg, detached=on, synchronous=off
Secondary:
        HostName: metamorphosi-ossrvg
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_pallini_ossrvg, detached=on, synchronous=off
Config Errors:
        metamorphosi-ossrvg: unknown

root@atmas1o:/ericsson/hagcs/log# vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  pallini-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               24
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   200.00 G
  Total secondaries:          1

Secondary:
  Host name:                  metamorphosi-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Data status:                inconsistent
  Replication status:         stopped (primary detached)
  Current mode:               N/A
  Logging to:                 SRL
  Timestamp Information:      N/A

Config Errors:
  metamorphosi-ossrvg:        unknown

root@atmas1o:/ericsson/hagcs/log#

root@atmas1o:/ericsson/hagcs/log# vxrlink -g ossdg status to_metamorphosi_ossrvg
VxVM VVR vxrlink INFO V-5-1-3539 Rlink to_metamorphosi_ossrvg is not currently attached (STALE).
You have new mail in /var/mail/root
root@atmas1o:/ericsson/hagcs/log#

Now if i start replication with below command, will there be any impact, as the primary is not buffering data as this status is there from last night.

vradmin -g ossdg -s startrep ossrvg

[symsonu]

How can be the primary site goes into inconsistent .

also , what is the impact of dissociating/assoc SRL in primary?

[symsonu]

Thanks Mike for good informative command

Also, i was thinking if mismatch of info field i.e rid and version causing this?


atmas2o{root} # vxprint -Vl
Disk group: ossdg

Rvg:      ossrvg
info:     rid=0.1280 version=0 rvg_version=30 last_tag=0
root@atmas1o:/ericsson/hagcs/log# vxprint -Vl
Disk group: ossdg

Rvg:      ossrvg
info:     rid=0.177060 version=25 rvg_version=30 last_tag=24
 

[mikebounds]

If you lost a volume on primary, then as VVR acknowledges write when writte to SRL then I think this would make rlink inconsistent, but you lost volume on secondary, so not quite sure how this happened.

Disassociating/assoc SRL on primary, cleans up the state and this is usually used when the SRL fails, but it may help in your case - see extract from VVR admin guide:

Primary SRL volume error cleanup and restart

If there is an error accessing the Primary SRL, the SRL is dissociated and the

RLINKs are detached. The state of the Primary and Secondary RLINKs is changed

to STALE. The RVG state does not change, but the RVG is put into PASSTHRU

mode that allows update of the Primary volume to continue until the error is fixed.

See “About RVG PASSTHRU mode” on page 413.

The SRL must be repaired manually and then associated with the RVG. While the

SRL is being repaired, no attempt is made to send data to the RLINKs. After the

SRL is replaced, all RLINKs must be completely synchronized. Attach the RLINKs

and perform a complete synchronization of the Secondaries.

On the Primary (seattle):

To cleanup after a Primary SRL error

1Dissociate the SRL from the RVG.

# vxvol -g hrdg dis hr_srl

2 Fix or replace the SRL volume.

3 Make sure that the repaired SRL is started before associating it with the RVG.

 If the repaired SRL is not started, start it:

# vxvol -g hrdg start hr_srl

4 Associate a new SRL with the RVG. After associating the new SRL, the RVG

PASSTHRU mode no longer displays in the output of the command vxprint

-lV.

# vxvol -g hrdg aslog hr_rvg hr_srl

Mike

[symsonu]

will there be any outage required for dissociating/associating SRL?

I mean , is it seamless?

Also if you could tell, on which node we need to run the start replication command?

primary or secondary ?

vradmin -g ossdg -a startrep ossrvg
 

[mikebounds]

In theory I think you should be able to do it online as you should be able to delete an RVG online and when you delete an RVG, it essentially disassociates the SRL and data volumes before deleting RVG object.  However, sometimes you can't delete RVG online when VVR gets into strange states, so disassociating the SRL online MAY not work, but if it doesn't the command will just fail, it won't offline your app.

If you are using VCS, then freeze your replication service group first.

You need to run startrep on the primary.

Mike

[symsonu]

are we sure that the configuration erro that is coming while replication start

and here in below output (marked bold) is due to inconsistencies in rlink in both primary

and secondary. This error saying something about configuration.

I need to be sure before executing SRL dissoc/assoc.

root@atmas1o:/ericsson/hagcs/log# vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  pallini-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               24
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   200.00 G
  Total secondaries:          1

Secondary:
  Host name:                  metamorphosi-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Data status:                inconsistent
  Replication status:         stopped (primary detached)
  Current mode:               N/A
  Logging to:                 SRL
  Timestamp Information:      N/A

Config Errors:
  metamorphosi-ossrvg:        unknown

[mikebounds]

I am not sure about the configuration error and neither is vradmin which is why it says "Config error unknown" and the reason for this is that the rlinks are not connected so the 2 sides are not communicating, but when they communicate when you start replication, VVR sees an issue.

The secondary being inconsistent is a normal state as this happens any time DCM is used before it has completed, but it is my understanding that the primary is not normally inconsistent, hence why I am suggesting "cleaning" the state by detached and reattaching SRL.

Alternatiley you could try force attaching rlinks as you have done earlier using "-f attach" - this is not valid in terms of using "-f" means the 2 sites are the same which they are not, but if there is a config error then I would think "-f" would fail and it MAY give more information about why it failed.  If "-f attach" works, then you can just use "vradmin stoprep" to stop replication as it will say it is "up-to-date" when really it is invalid.

Mike

[symsonu]

Hello Mike/Gaurav

I ran the below attach command on secondary to ses if it reports SRL header errors

# vxrlink -g ossdg att to_pallini_ossrvg

I donot get any error and rlink goes to Enabled Active from Detached SSTALE.

atmas2o{root} # vxprint -PV
Disk group: ossdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
rl to_pallini_ossrvg ossrvg  ENABLED  -        -        ACTIVE   -       -
rv ossrvg       -            ENABLED  -        -        ACTIVE   -       -

Then I continue it on primary as below with -a

# vxrlink -g ossdg -a att to_metamorphosi_ossrvg

atmas1o#vxrlink -g ossdg -a att to_metamorphosi_ossrvg
VxVM VVR vxrlink WARNING V-5-1-3359 Attaching rlink to non-empty rvg. Autosync will be performed.
VxVM VVR vxrlink INFO V-5-1-3614 Secondary data volumes detected with rvg ossrvg as parent:
VxVM VVR vxrlink INFO V-5-1-6183 3pp: len=51200000 primary_datavol=3pp
VxVM VVR vxrlink INFO V-5-1-6183 JUMP: len=25165824 primary_datavol=JUMP
VxVM VVR vxrlink INFO V-5-1-6183 eba_ebss: len=10485760 primary_datavol=eba_ebss
VxVM VVR vxrlink INFO V-5-1-6183 eba_ebsw: len=40960 primary_datavol=eba_ebsw
VxVM VVR vxrlink INFO V-5-1-6183 eba_rede: len=10485760 primary_datavol=eba_rede
VxVM VVR vxrlink INFO V-5-1-6183 eba_rsdm: len=40960 primary_datavol=eba_rsdm
VxVM VVR vxrlink INFO V-5-1-6183 eba_rtt: len=20971520 primary_datavol=eba_rtt
VxVM VVR vxrlink INFO V-5-1-6183 eniq_pm_vol: len=16005120 primary_datavol=eniq_pm_vol
VxVM VVR vxrlink INFO V-5-1-6183 ericsson: len=110852096 primary_datavol=ericsson
VxVM VVR vxrlink INFO V-5-1-6183 export: len=83968000 primary_datavol=export
VxVM VVR vxrlink INFO V-5-1-6183 fmsybdata: len=31043584 primary_datavol=fmsybdata
VxVM VVR vxrlink INFO V-5-1-6183 fmsyblog: len=15515648 primary_datavol=fmsyblog
VxVM VVR vxrlink INFO V-5-1-6183 home: len=113623040 primary_datavol=home
VxVM VVR vxrlink INFO V-5-1-6183 pm_storage: len=162775040 primary_datavol=pm_storage
VxVM VVR vxrlink INFO V-5-1-6183 pmsybdata: len=4481024 primary_datavol=pmsybdata
VxVM VVR vxrlink INFO V-5-1-6183 pmsyblog: len=1855488 primary_datavol=pmsyblog
VxVM VVR vxrlink INFO V-5-1-6183 polled_data: len=1024000 primary_datavol=polled_data
VxVM VVR vxrlink INFO V-5-1-6183 segment1: len=40960 primary_datavol=segment1
VxVM VVR vxrlink INFO V-5-1-6183 sgwcg: len=228884480 primary_datavol=sgwcg
VxVM VVR vxrlink INFO V-5-1-6183 sybdata: len=188743680 primary_datavol=sybdata
VxVM VVR vxrlink INFO V-5-1-6183 syblog: len=80912384 primary_datavol=syblog
VxVM VVR vxrlink INFO V-5-1-6183 sybmaster: len=12095488 primary_datavol=sybmaster
VxVM VVR vxrlink INFO V-5-1-6183 upgrade: len=33554432 primary_datavol=upgrade
VxVM VVR vxrlink INFO V-5-1-6183 versant: len=38912000 primary_datavol=versant
VxVM VVR vxrlink INFO V-5-1-3365 Autosync operation has started
 

=====================================================================

Now the status is as below

Primary :-->

root@atmas1o:/ericsson/hagcs/bin# vxprint -PV
Disk group: ossdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
rl to_metamorphosi_ossrvg ossrvg CONNECT -     -        ACTIVE   -       -
rv ossrvg       -            ENABLED  -        -        ACTIVE   -       -
root@atmas1o:/ericsson/hagcs/bin#

root@atmas1o:/ericsson/hagcs/bin# vxprint -lP
Disk group: ossdg

Rlink:    to_metamorphosi_ossrvg
info:     timeout=500 rid=0.198498
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=off
assoc:    rvg=ossrvg
          remote_host=metamorphosi-ossrvg IP_addr=10.40.240.52 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1389095739.49.atmas2o
          remote_rvg_version=30
          remote_rlink=to_pallini_ossrvg
          remote_rlink_rid=0.1284
          local_host=pallini-ossrvg IP_addr=10.40.240.50 port=4145
protocol: TCP/IP
flags:    write enabled attached inconsistent cant_sync connected asynchronous autosync resync_started
 

root@atmas1o:/ericsson/hagcs/bin# vxprint -lP
Disk group: ossdg

Rlink:    to_metamorphosi_ossrvg
info:     timeout=500 rid=0.198498
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=off
assoc:    rvg=ossrvg
          remote_host=metamorphosi-ossrvg IP_addr=10.40.240.52 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1389095739.49.atmas2o
          remote_rvg_version=30
          remote_rlink=to_pallini_ossrvg
          remote_rlink_rid=0.1284
          local_host=pallini-ossrvg IP_addr=10.40.240.50 port=4145
protocol: TCP/IP
flags:    write enabled attached inconsistent cant_sync connected asynchronous autosync resync_started

root@atmas1o:/ericsson/hagcs/bin# vxprint -g ossdg -l repstatus ossrvg
VxVM vxprint ERROR V-5-1-924 Record repstatus not found
Rvg:      ossrvg
info:     rid=0.177060 version=25 rvg_version=30 last_tag=24
state:    state=ACTIVE kernel=ENABLED
assoc:    datavols=3pp,JUMP,eba_ebss,eba_ebsw,eba_rede,eba_rsdm,eba_rtt,...
          srl=oss_srl_vol
          rlinks=to_metamorphosi_ossrvg
          exports=(none)
          vsets=(none)
att:      rlinks=to_metamorphosi_ossrvg
flags:    closed primary enabled attached autosync resync_started
device:   minor=15001 bdev=329/15001 cdev=329/15001 path=/dev/vx/dsk/ossdg/ossrvg
perms:    user=root group=root mode=0600

You have new mail in /var/mail/root

============================================

Secondary

atmas2o{root} # vxprint -PV
Disk group: ossdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
rl to_pallini_ossrvg ossrvg  CONNECT  -        -        ACTIVE   -       -
rv ossrvg       -            ENABLED  -        -        ACTIVE   -       -
atmas2o{root} # vxprint -Pl
Disk group: ossdg

Rlink:    to_pallini_ossrvg
info:     timeout=500 rid=0.1284
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=off
assoc:    rvg=ossrvg
          remote_host=pallini-ossrvg IP_addr=10.40.240.50 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1318259952.12.atmas2o
          remote_rvg_version=30
          remote_rlink=to_metamorphosi_ossrvg
          remote_rlink_rid=0.198498
          local_host=metamorphosi-ossrvg IP_addr=10.40.240.52 port=4145
protocol: TCP/IP
flags:    write enabled attached inconsistent cant_sync connected

atmas2o{root} # vxprint -g ossdg -l repstatus ossrvg
VxVM vxprint ERROR V-5-1-924 Record repstatus not found
Rvg:      ossrvg
info:     rid=0.1280 version=25 rvg_version=30 last_tag=24
state:    state=ACTIVE kernel=ENABLED
assoc:    datavols=JUMP,eba_ebss,eba_ebsw,eba_rede,eba_rsdm,eba_rtt,...
          srl=oss_srl_vol
          rlinks=to_pallini_ossrvg
          exports=(none)
          vsets=(none)
att:      rlinks=to_pallini_ossrvg
flags:    closed secondary enabled attached
device:   minor=25027 bdev=329/25027 cdev=329/25027 path=/dev/vx/dsk/ossdg/ossrvg
perms:    user=root group=root mode=0600

Please have a look and suggest.how is it going on ?

Management is on me ........

[mikebounds]

If you have inconsistent data then you can't synchronise which is what you are seeing, so normally you could have "inconsistent cant_sync" at secondary, but not at primary, which means you cannot use secondary to sync and you must sync from primary - i.e in with "inconsistent cant_sync" ONLY at secondary, VVR would prevent you doing a takeover at secondary and syncing back to old primary.

You have inconsistent data on both rlinks and I am 95% sure that an rlink only shows the status of its local data - i.e showing inconsistent at primary is not reflecting that it's associated secondary is inconsistent, it is saying the primary is inconsistent.

If your application is running, then I guess VVR is wrong and your data is ok, so you need to get rid of your inconsistent flag.  You could delete the RDS (delete both rvg and rlink) and as your links are connected at the moment you should be able to use vradmin delsec and vradmin delpri to do this and then recreate using vradmin createpri and addsec, but probably you only need to detach and reattach SRL, but it doesn't take long to do either, so you could delete and recreate VVR objects to make sure.

Mike

[mikebounds]

Alternativey, you could try disassociating and reassociating rlink at primary, but I don't know if this will clear state and I don't think I have done this before, so don't know it there will be any issues doing this - commands would be:

vxrlink -g ossdg dis to_metamorphosi_ossrvg
vxrlink -g ossdg assoc ossrvg to_metamorphosi_ossrvg

You would need to detach rlink first using vxrlink det to vradmin stoprep

Mike

[symsonu]

autosync is in progress

atmas1o{root} # vxrlink -g ossdg -i5 status to_metamorphosi_ossrvg

Thu Jan 09 11:36:52 2014
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546278432 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546227232 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546175776 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546124320 Kbytes remaining.

But, the thing is autosync is running....

[mikebounds]

If VVR is saying cant_sync, but autosync is showing Kbytes remaining is decreasing, then it is syncing, so when it finishes, VVR should no longer say it is inconsistent.

My guess is that the lower level vxrlink command doesn't check cant_sync flag so just goes ahead, but vradmin startrep does and complains about config error.

Mike

[Gaurav_S]

Hi,

Cant_sync flag will go away once the sync is complete & so will the inconsistent flag. I have observed this in many setups that flags remain in vxprint output untill the sync is finished. so I wouldn't really worry about the flags untill sync in finished.

In this whole exercise, not using the -a  flag for autosync at the first instance, rather force attaching the rlink using -f flag was the incorrect step which made VVR to believe that both the sites are consistent & forced to attach. I believe this was one of the strong reasons of needing this troubleshooting, we have forced VVR to believe both sites to be consistent when they were not, as soon sync is initiated with autosync, VVR has determined the differences & doing a sync now.

It would be worth to do a DR test post rlink is upto date with first possible opportunity just to ensure that DR is ready.

G

[mikebounds · Accepted solution]

Just a couple of points on Guarav's comment:

When you use -f on "vxrlink att" or vradmin startrep, you are telling VVR that both sites are the same, so if for instance you stopped your application and umounted filesystems while VVR was up to date and then detached rlinks or even deleted RDS, then when you attach rlink, you can use "-f" as you know both sites are the same if the filesystems remained umounted.  VVR does not check anything when you use -f, so if the secondary site is behind, then VVR will still report as consistent, even though as soon as VVR starts replicating, the secondary data will become and remain inconsistent and VVR will be unaware.  So using "-f" can make your data inconsistent, but it will not effect the VVR states.

When you use autosync, VVR does not determine differences, it syncs all blocks except when using smartmove where VVR syncs all used blocks in the filesystems, and does not sync unused blocks.

Mike