vvr replication not ok .....need help to fix it

[symsonu]

Hello,

Replication is not happening , however RVG  is enabled /active

and Rlink is conect/active ....

below are the details :-->

-> vxprint -Pl
Disk group: ossdg

Rlink:    to_sec_site_ossrvg
info:     timeout=500 packet_size=1452 rid=0.113591
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=ossrvg
          remote_host=sec_site-ossrvg IP_addr=10.52.214.68 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1444153050.96.secmas1o
          remote_rvg_version=30
          remote_rlink=to_prisite_ossrvg
          remote_rlink_rid=0.1263
          local_host=prisite-ossrvg IP_addr=10.52.214.67 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected asynchronous dcm_logging

-> vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  prisite-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               20
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   300.00 G
  Total secondaries:          1

Secondary:
  Host name:                  sec_site-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Data status:                consistent, behind
  Replication status:         logging to DCM (needs dcm resynchronization)
  Current mode:               asynchronous
  Logging to:                 DCM (contains 61101728 Kbytes) (SRL protection logging)
  Timestamp Information:      N/A

-> vxrvg -g ossdg cplist ossrvg
Name                    MBytes  % Log   Started/Completed
----                    ------  ------  -----------------
point1                  <Checkpoint overflowed>

-> vxrlink -g ossdg status to_sec_site_ossrvg

DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61101728 Kbytes (19%) of the Data Volume(s).

-> vxrlink -g ossdg stats to_sec_site_ossrvg

        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory   Delays  NW Bytes  NW Delay Timeout
1       0       0       0       0       0       0       100000    1        10

primas1o{root} #: vxprint -PV
Disk group: ossdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
rl to_sec_site_ossrvg ossrvg CONNECT  -        -        ACTIVE   -       -
rv ossrvg       -            ENABLED  -        -        ACTIVE   -       -

primas1o{root} #: vxrlink -g ossdg -i 5 status  to_sec_site_ossrvg

Mon Nov 30 17:24:01 2015
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443008 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443008 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443040 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443232 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443232 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443264 Kbytes (19%) of the Data Volume(s).
^Cprimas1o{root} #:

=====================================================================================================================

Secondary

=======================================================

-> vxprint -Pl
Disk group: ossdg

Rlink:    to_prisite_ossrvg
info:     timeout=500 packet_size=1452 rid=0.1263
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=ossrvg
          remote_host=prisite-ossrvg IP_addr=10.52.214.67 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1327928106.32.secmas1o
          remote_rvg_version=30
          remote_rlink=to_sec_site_ossrvg
          remote_rlink_rid=0.113591
          local_host=sec_site-ossrvg IP_addr=10.52.214.68 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected

-> vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  prisite-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               20
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   300.00 G
  Total secondaries:          1

Secondary:
  Host name:                  sec_site-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Data status:                consistent, behind
  Replication status:         logging to DCM (needs dcm resynchronization)
  Current mode:               asynchronous
  Logging to:                 DCM (contains 61442560 Kbytes) (SRL protection logging)
  Timestamp Information:      N/A

-> vxrvg -g ossdg cplist ossrvg
The cplist command can only be used on a primary rvg

-> vxrlink -g ossdg status to_prisite_ossrvg
The status command can only be used on a primary rlink

-> vxrlink -g ossdg stats to_prisite_ossrvg

        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory   Delays  NW Bytes  NW Delay Timeout
334089805       0       0       1927960 0       0       0       100000    1        10

Also, I ran the below command to check rlink bandwidth usgae and it is 0% utilised

Cprimas1o{root} #: vrstat -R

Mon Nov 30 17:43:49 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg: DCM contains 61543520 Kbytes.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o
1       0       0       0       0       0       0       100000    1        10

Mon Nov 30 17:43:59 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg: DCM contains 61543552 Kbytes.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o
0       0       0       0       0       0       0       100000    1        10
Bandwidth Utilization 0.00 Kbps.

I am wondering if rlink in connect/active state then why its not replicating  ?

Regards

S

[Gaurav_S · Accepted solution]

Hi,

You need to execute a resync command from primary

# vradmin -g <dg> resync <rvg>

That will flush out DCM logs to write & get in sync

G

[symsonu]

Hi Gaurav,

Why is it writing to DCM when Rlinks are in connect/Active state?

Regards

S.

[symsonu]

I ran the resync and DCM starts  to decay as shown below , During DCM resynch bandwidth utlistaion
was between 200 to 200 Mbps

primas1o{root} #: vradmin -g ossdg resync ossrvg

primas1o{root} #: vxrlink -g ossdg -i 5 status  to_sec_site_ossrvg

Tue Dec 01 06:16:31 2015
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 68094560 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67938880 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67738336 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67550848 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67363296 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67201024 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67114592 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67029664 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66943744 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66778144 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66558688 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66337760 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66119200 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65904896 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65686624 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65474080 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65274432 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65124640 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65031040 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64890144 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64674944 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64474400 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64291712 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64075296 Kbytes (20%) of the Data Volume(s).
^Cprimas1o{root} #: vrstat -R

Tue Dec 01 06:18:32 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg: DCM contains 63843296 Kbytes.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
23061   9941632 5       169     0       0       1024    787000    1        10
334112407       0       7       1921859 0       0       0       5000      63       20

Tue Dec 01 06:18:42 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg: DCM contains 63481664 Kbytes.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
1846    725440  5       11      0       0       0       786000    1        10
1628    0       5       30      0       0       0       5000      81       10
Bandwidth Utilization 283.38 Mbps.

Tue Dec 01 06:18:52 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg: DCM contains 63097952 Kbytes.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
1681    829824  8       15      0       0       0       787000    1        20
1859    0       7       32      0       0       0       5000      100      20
Bandwidth Utilization 324.15 Mbps.

Tue Dec 01 06:19:02 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg: DCM contains 62800448 Kbytes.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
1170    539136  5       20      0       0       0       786000    1        10
1355    0       5       29      0       0       0       5000      100      10
Bandwidth Utilization 210.60 Mbps.

T of the Data Volume(s).
^C
You have new mail in /var/mail//root
root@primas1o:/# vxrlink -g ossdg -i 5 status to_sec_site_ossrvg

Tue Dec 01 06:55:49 2015
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 3760 outstanding writes, occupying 165036 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 2297 outstanding writes, occupying 69886 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 2776 outstanding writes, occupying 126479 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 2423 outstanding writes, occupying 149798 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 1772 outstanding writes, occupying 85379 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 570 outstanding writes, occupying 23797 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 189 outstanding writes, occupying 576 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4467 Rlink to_sec_site_ossrvg is up to date

Here, replication became uptodate  and started wrinting to SRL, Howevernow bandwidth utilisation droped 10 times than earlier

and we can see writes piling up in SRL though slowly. Its been noticed that in few days it will again fill up SRL and starts dcm logging.

^C
root@primas1o:/# vxprint -Pl
Disk group: ossdg

Rlink:    to_sec_site_ossrvg
info:     timeout=500 packet_size=1452 rid=0.113591
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=ossrvg
          remote_host=sec_site-ossrvg IP_addr=10.52.214.68 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1444153050.96.secfmcmas1o
          remote_rvg_version=30
          remote_rlink=to_prisite_ossrvg
          remote_rlink_rid=0.1263
          local_host=prisite-ossrvg IP_addr=10.52.214.67 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected asynchronous

root@primas1o:/# vrstat -R

Tue Dec 01 06:56:42 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg:26 writes behind, occupying 1885 Kbytes(0%) on SRL.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
12756   1299138 5       108     0       0       0       796000    1        10
334514874       0       6       1931402 0       0       0       99000     1        10

Tue Dec 01 06:56:52 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg: up-to-date.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
134     5547    5       0       0       0       0       816000    1        10
251     0       5       0       0       0       0       99000     1        10
Bandwidth Utilization 2.16 Mbps.

Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg:183998 writes behind, occupying 2208411 Kbytes(0%) on SRL.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
2778    331098  8       22      0       0       0       594000    1        20
2668    0       10      37      0       0       0       5000      100      40
Bandwidth Utilization 129.33 Mbps.

#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
6053    81766   5       4       0       0       0       570000    1        10
2880    0       5       9       0       0       0       5000      100      10
Bandwidth Utilization 31.94 Mbps.

Tue Dec 01 07:12:22 2015
Replicated Data Set ossrvg:

Data Status:
sec_site-ossrvg:353995 writes behind, occupying 2357604 Kbytes(0%) on SRL.

Network Statistics:
        Messages                Errors                  Flow Control
        --------                ------                  ------------
#    Blocks  RT(msec) Timeout Stream Memory Delays  NW Bytes  NW Delay Timeout
primas1o - secfmcmas1o
525     65633   5       0       0       0       0       600000    1        10
4047    0       5       0       0       0       0       5000      100      10
Bandwidth Utilization 25.64 Mbps.

How can we improve the situation that writes donot pile up in SRL and eventually making it full

[Gaurav_S]

Hi,

Check out on network, there are couple of likely conditions here .... DCM comes into play when SRL overflows ... SRL can overflow because of 2 major reasons..

1. Application is writing too fast & SRL volume is too low on size which makes SRL overflow ..

2. You had network glitch which didn't allow data to be sent to DR site eventually filling up SRL ..

G

[symsonu]

Hi Gaurav,

Network team is saying there is no issue with network.

can you please comment why speed is fast during dcm resynch as compared to normal replication as notice in above logs

Regards

S.

[symsonu]

Do you think there is need to chnage tunables as shown below

primas1o{root} #:  vxrlink -g ossdg -i 5 -e stats to_sec_site_ossrvg

Tue Dec 01 15:32:53 2015
 Messages :
 --------
 Number of blocks sent                           : 3988248

 Errors :
 ------
 No memory available                             : 0
 No message slots available                      : 0
 No memory available in nmcom pool on Secondary  : 0
 Timeout                                         : 434
 Missing packet                                  : 183
 Missing message                                 : 228
 Stream                                          : 0
 Checksum                                        : 0
 Unable to deliver due to transaction            : 6

 Messages :
 --------
 Number of blocks sent                           : 125978

 Errors :
 ------
 No memory available                             : 0
 No message slots available                      : 0
 No memory available in nmcom pool on Secondary  : 0
 Timeout                                         : 86
 Missing packet                                  : 27
 Missing message                                 : 8
 Stream                                          : 0
 Checksum                                        : 0
 Unable to deliver due to transaction            : 0

 Messages :
 --------
 Number of blocks sent                           : 75960

 Errors :
 ------
 No memory available                             : 0
 No message slots available                      : 0
 No memory available in nmcom pool on Secondary  : 0
 Timeout                                         : 0
 Missing packet                                  : 0
 Missing message                                 : 2
 Stream                                          : 0
 Checksum                                        : 0
 Unable to deliver due to transaction            : 0

 Messages :
 --------
 Number of blocks sent                           : 22514

 Errors :
 ------
 No memory available                             : 0
 No message slots available                      : 0
 No memory available in nmcom pool on Secondary  : 0
 Timeout                                         : 0
 Missing packet                                  : 0
 Missing message                                 : 0
 Stream                                          : 0
 Checksum                                        : 0
 Unable to deliver due to transaction            : 0

 Messages :
 --------
 Number of blocks sent                           : 189265

 Errors :
 ------
 No memory available                             : 0
 No message slots available                      : 0
 No memory available in nmcom pool on Secondary  : 0
 Timeout                                         : 19
 Missing packet                                  : 11
 Missing message                                 : 11
 Stream                                          : 0
 Checksum                                        : 0
 Unable to deliver due to transaction            : 0

 Messages :
 --------
 Number of blocks sent                           : 252878

 Errors :
 ------
 No memory available                             : 0
 No message slots available                      : 0
 No memory available in nmcom pool on Secondary  : 0
 Timeout                                         : 1
 Missing packet                                  : 3
 Missing message                                 : 24
 Stream                                          : 0
 Checksum                                        : 0
 Unable to deliver due to transaction            : 0

 Messages :
 --------
 Number of blocks sent                           : 168502

 Errors :
 ------
 No memory available                             : 0
 No message slots available                      : 0
 No memory available in nmcom pool on Secondary  : 0
 Timeout                                         : 31
 Missing packet                                  : 10
 Missing message                                 : 13
 Stream                                          : 0
 Checksum                                        : 0
 Unable to deliver due to transaction            : 0

^Cprimas1o{root} #: vxtune
     Tunable              Value         Description
-----------------        --------      ---------------
vol_rvio_maxpool_sz        262144       RVIO Pool Size (KBytes)
vol_min_lowmem_sz           11568       Low Memory Threshold (KBytes)
vol_max_rdback_sz           65536       Readback Pool Size (KBytes)
vol_max_nmpool_sz           16384       NMCOM Pool Size (KBytes)
vol_max_wrspool_sz          16384       Write Shipping Pool Size (KBytes)
volpagemod_max_memsz        65536       Cache Size for FMR Metadata (KBytes)
primas1o{root} #: ssh secmas1o
Last login: Tue Dec  1 15:30:42 2015 from prisite-a-repla
You have new mail.
DISPLAY set to primas1o:0.0
secmas1o{root} #: vxtune
     Tunable              Value         Description
-----------------        --------      ---------------
vol_rvio_maxpool_sz        262144       RVIO Pool Size (KBytes)
vol_min_lowmem_sz             528       Low Memory Threshold (KBytes)
vol_max_rdback_sz           65536       Readback Pool Size (KBytes)
vol_max_nmpool_sz          131072       NMCOM Pool Size (KBytes)
vol_max_wrspool_sz          16384       Write Shipping Pool Size (KBytes)
volpagemod_max_memsz        65536       Cache Size for FMR Metadata (KBytes)
secmas1o{root} #: