secondary rvg unknown

Hi,

Cant_sync flag will go away once the sync is complete & so will the inconsistent flag. I have observed this in many setups that flags remain in vxprint output untill the sync is finished. so I wouldn't really worry about the flags untill sync in finished.

In this whole exercise, not using the -a  flag for autosync at the first instance, rather force attaching the rlink using -f flag was the incorrect step which made VVR to believe that both the sites are consistent & forced to attach. I believe this was one of the strong reasons of needing this troubleshooting, we have forced VVR to believe both sites to be consistent when they were not, as soon sync is initiated with autosync, VVR has determined the differences & doing a sync now.

It would be worth to do a DR test post rlink is upto date with first possible opportunity just to ensure that DR is ready.

G

If VVR is saying cant_sync, but autosync is showing Kbytes remaining is decreasing, then it is syncing, so when it finishes, VVR should no longer say it is inconsistent.

My guess is that the lower level vxrlink command doesn't check cant_sync flag so just goes ahead, but vradmin startrep does and complains about config error.

Mike

autosync is in progress

atmas1o{root} # vxrlink -g ossdg -i5 status to_metamorphosi_ossrvg

Thu Jan 09 11:36:52 2014
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546278432 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546227232 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546175776 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink to_metamorphosi_ossrvg is in AUTOSYNC. 546124320 Kbytes remaining.

But, the thing is autosync is running....

Alternativey, you could try disassociating and reassociating rlink at primary, but I don't know if this will clear state and I don't think I have done this before, so don't know it there will be any issues doing this - commands would be:

vxrlink -g ossdg dis to_metamorphosi_ossrvg
vxrlink -g ossdg assoc ossrvg to_metamorphosi_ossrvg

Mike

If you have inconsistent data then you can't synchronise which is what you are seeing, so normally you could have "inconsistent cant_sync" at secondary, but not at primary, which means you cannot use secondary to sync and you must sync from primary - i.e in with "inconsistent cant_sync" ONLY at secondary, VVR would prevent you doing a takeover at secondary and syncing back to old primary.

You have inconsistent data on both rlinks and I am 95% sure that an rlink only shows the status of its local data - i.e showing inconsistent at primary is not reflecting that it's associated secondary is inconsistent, it is saying the primary is inconsistent.

If your application is running, then I guess VVR is wrong and your data is ok, so you need to get rid of your inconsistent flag.  You could delete the RDS (delete both rvg and rlink) and as your links are connected at the moment you should be able to use vradmin delsec and vradmin delpri to do this and then recreate using vradmin createpri and addsec, but probably you only need to detach and reattach SRL, but it doesn't take long to do either, so you could delete and recreate VVR objects to make sure.

Mike

Hello Mike/Gaurav

I ran the below attach command on secondary to ses if it reports SRL header errors

# vxrlink -g ossdg att to_pallini_ossrvg

I donot get any error and rlink goes to Enabled Active from Detached SSTALE.

atmas2o{root} # vxprint -PV
Disk group: ossdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
rl to_pallini_ossrvg ossrvg  ENABLED  -        -        ACTIVE   -       -
rv ossrvg       -            ENABLED  -        -        ACTIVE   -       -

Then I continue it on primary as below with -a

# vxrlink -g ossdg -a att to_metamorphosi_ossrvg

atmas1o#vxrlink -g ossdg -a att to_metamorphosi_ossrvg
VxVM VVR vxrlink WARNING V-5-1-3359 Attaching rlink to non-empty rvg. Autosync will be performed.
VxVM VVR vxrlink INFO V-5-1-3614 Secondary data volumes detected with rvg ossrvg as parent:
VxVM VVR vxrlink INFO V-5-1-6183 3pp: len=51200000 primary_datavol=3pp
VxVM VVR vxrlink INFO V-5-1-6183 JUMP: len=25165824 primary_datavol=JUMP
VxVM VVR vxrlink INFO V-5-1-6183 eba_ebss: len=10485760 primary_datavol=eba_ebss
VxVM VVR vxrlink INFO V-5-1-6183 eba_ebsw: len=40960 primary_datavol=eba_ebsw
VxVM VVR vxrlink INFO V-5-1-6183 eba_rede: len=10485760 primary_datavol=eba_rede
VxVM VVR vxrlink INFO V-5-1-6183 eba_rsdm: len=40960 primary_datavol=eba_rsdm
VxVM VVR vxrlink INFO V-5-1-6183 eba_rtt: len=20971520 primary_datavol=eba_rtt
VxVM VVR vxrlink INFO V-5-1-6183 eniq_pm_vol: len=16005120 primary_datavol=eniq_pm_vol
VxVM VVR vxrlink INFO V-5-1-6183 ericsson: len=110852096 primary_datavol=ericsson
VxVM VVR vxrlink INFO V-5-1-6183 export: len=83968000 primary_datavol=export
VxVM VVR vxrlink INFO V-5-1-6183 fmsybdata: len=31043584 primary_datavol=fmsybdata
VxVM VVR vxrlink INFO V-5-1-6183 fmsyblog: len=15515648 primary_datavol=fmsyblog
VxVM VVR vxrlink INFO V-5-1-6183 home: len=113623040 primary_datavol=home
VxVM VVR vxrlink INFO V-5-1-6183 pm_storage: len=162775040 primary_datavol=pm_storage
VxVM VVR vxrlink INFO V-5-1-6183 pmsybdata: len=4481024 primary_datavol=pmsybdata
VxVM VVR vxrlink INFO V-5-1-6183 pmsyblog: len=1855488 primary_datavol=pmsyblog
VxVM VVR vxrlink INFO V-5-1-6183 polled_data: len=1024000 primary_datavol=polled_data
VxVM VVR vxrlink INFO V-5-1-6183 segment1: len=40960 primary_datavol=segment1
VxVM VVR vxrlink INFO V-5-1-6183 sgwcg: len=228884480 primary_datavol=sgwcg
VxVM VVR vxrlink INFO V-5-1-6183 sybdata: len=188743680 primary_datavol=sybdata
VxVM VVR vxrlink INFO V-5-1-6183 syblog: len=80912384 primary_datavol=syblog
VxVM VVR vxrlink INFO V-5-1-6183 sybmaster: len=12095488 primary_datavol=sybmaster
VxVM VVR vxrlink INFO V-5-1-6183 upgrade: len=33554432 primary_datavol=upgrade
VxVM VVR vxrlink INFO V-5-1-6183 versant: len=38912000 primary_datavol=versant
VxVM VVR vxrlink INFO V-5-1-3365 Autosync operation has started
 

=====================================================================

Now the status is as below

Primary :-->

root@atmas1o:/ericsson/hagcs/bin# vxprint -PV
Disk group: ossdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
rl to_metamorphosi_ossrvg ossrvg CONNECT -     -        ACTIVE   -       -
rv ossrvg       -            ENABLED  -        -        ACTIVE   -       -
root@atmas1o:/ericsson/hagcs/bin#

root@atmas1o:/ericsson/hagcs/bin# vxprint -lP
Disk group: ossdg

Rlink:    to_metamorphosi_ossrvg
info:     timeout=500 rid=0.198498
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=off
assoc:    rvg=ossrvg
          remote_host=metamorphosi-ossrvg IP_addr=10.40.240.52 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1389095739.49.atmas2o
          remote_rvg_version=30
          remote_rlink=to_pallini_ossrvg
          remote_rlink_rid=0.1284
          local_host=pallini-ossrvg IP_addr=10.40.240.50 port=4145
protocol: TCP/IP
flags:    write enabled attached inconsistent cant_sync connected asynchronous autosync resync_started
 

root@atmas1o:/ericsson/hagcs/bin# vxprint -lP
Disk group: ossdg

Rlink:    to_metamorphosi_ossrvg
info:     timeout=500 rid=0.198498
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=off
assoc:    rvg=ossrvg
          remote_host=metamorphosi-ossrvg IP_addr=10.40.240.52 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1389095739.49.atmas2o
          remote_rvg_version=30
          remote_rlink=to_pallini_ossrvg
          remote_rlink_rid=0.1284
          local_host=pallini-ossrvg IP_addr=10.40.240.50 port=4145
protocol: TCP/IP
flags:    write enabled attached inconsistent cant_sync connected asynchronous autosync resync_started

root@atmas1o:/ericsson/hagcs/bin# vxprint -g ossdg -l repstatus ossrvg
VxVM vxprint ERROR V-5-1-924 Record repstatus not found
Rvg:      ossrvg
info:     rid=0.177060 version=25 rvg_version=30 last_tag=24
state:    state=ACTIVE kernel=ENABLED
assoc:    datavols=3pp,JUMP,eba_ebss,eba_ebsw,eba_rede,eba_rsdm,eba_rtt,...
          srl=oss_srl_vol
          rlinks=to_metamorphosi_ossrvg
          exports=(none)
          vsets=(none)
att:      rlinks=to_metamorphosi_ossrvg
flags:    closed primary enabled attached autosync resync_started
device:   minor=15001 bdev=329/15001 cdev=329/15001 path=/dev/vx/dsk/ossdg/ossrvg
perms:    user=root group=root mode=0600

You have new mail in /var/mail/root

============================================

Secondary

atmas2o{root} # vxprint -PV
Disk group: ossdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
rl to_pallini_ossrvg ossrvg  CONNECT  -        -        ACTIVE   -       -
rv ossrvg       -            ENABLED  -        -        ACTIVE   -       -
atmas2o{root} # vxprint -Pl
Disk group: ossdg

Rlink:    to_pallini_ossrvg
info:     timeout=500 rid=0.1284
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=off
assoc:    rvg=ossrvg
          remote_host=pallini-ossrvg IP_addr=10.40.240.50 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1318259952.12.atmas2o
          remote_rvg_version=30
          remote_rlink=to_metamorphosi_ossrvg
          remote_rlink_rid=0.198498
          local_host=metamorphosi-ossrvg IP_addr=10.40.240.52 port=4145
protocol: TCP/IP
flags:    write enabled attached inconsistent cant_sync connected

atmas2o{root} # vxprint -g ossdg -l repstatus ossrvg
VxVM vxprint ERROR V-5-1-924 Record repstatus not found
Rvg:      ossrvg
info:     rid=0.1280 version=25 rvg_version=30 last_tag=24
state:    state=ACTIVE kernel=ENABLED
assoc:    datavols=JUMP,eba_ebss,eba_ebsw,eba_rede,eba_rsdm,eba_rtt,...
          srl=oss_srl_vol
          rlinks=to_pallini_ossrvg
          exports=(none)
          vsets=(none)
att:      rlinks=to_pallini_ossrvg
flags:    closed secondary enabled attached
device:   minor=25027 bdev=329/25027 cdev=329/25027 path=/dev/vx/dsk/ossdg/ossrvg
perms:    user=root group=root mode=0600

Please have a look and suggest.how is it going on ?

Management is on me ........

I am not sure about the configuration error and neither is vradmin which is why it says "Config error unknown" and the reason for this is that the rlinks are not connected so the 2 sides are not communicating, but when they communicate when you start replication, VVR sees an issue.

The secondary being inconsistent is a normal state as this happens any time DCM is used before it has completed, but it is my understanding that the primary is not normally inconsistent, hence why I am suggesting "cleaning" the state by detached and reattaching SRL.

Alternatiley you could try force attaching rlinks as you have done earlier using "-f attach" - this is not valid in terms of using "-f" means the 2 sites are the same which they are not, but if there is a config error then I would think "-f" would fail and it MAY give more information about why it failed.  If "-f attach" works, then you can just use "vradmin stoprep" to stop replication as it will say it is "up-to-date" when really it is invalid.

Mike

are we sure that the configuration erro that is coming while replication start

and here in below output (marked bold) is due to inconsistencies in rlink in both primary

and secondary. This error saying something about configuration.

I need to be sure before executing SRL dissoc/assoc.

root@atmas1o:/ericsson/hagcs/log# vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  pallini-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               24
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   200.00 G
  Total secondaries:          1

Secondary:
  Host name:                  metamorphosi-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Data status:                inconsistent
  Replication status:         stopped (primary detached)
  Current mode:               N/A
  Logging to:                 SRL
  Timestamp Information:      N/A

Config Errors:
  metamorphosi-ossrvg:        unknown

In theory I think you should be able to do it online as you should be able to delete an RVG online and when you delete an RVG, it essentially disassociates the SRL and data volumes before deleting RVG object.  However, sometimes you can't delete RVG online when VVR gets into strange states, so disassociating the SRL online MAY not work, but if it doesn't the command will just fail, it won't offline your app.

If you are using VCS, then freeze your replication service group first.

You need to run startrep on the primary.

Mike