11-30-2015 09:48 AM
Hello,
Replication is not happening , however RVG is enabled /active
and Rlink is conect/active ....
below are the details :-->
-> vxprint -Pl
Disk group: ossdg
Rlink: to_sec_site_ossrvg
info: timeout=500 packet_size=1452 rid=0.113591
latency_high_mark=10000 latency_low_mark=9950
bandwidth_limit=none
state: state=ACTIVE
synchronous=off latencyprot=off srlprot=autodcm
assoc: rvg=ossrvg
remote_host=sec_site-ossrvg IP_addr=10.52.214.68 port=4145
remote_dg=ossdg
remote_dg_dgid=1444153050.96.secmas1o
remote_rvg_version=30
remote_rlink=to_prisite_ossrvg
remote_rlink_rid=0.1263
local_host=prisite-ossrvg IP_addr=10.52.214.67 port=4145
protocol: UDP/IP
flags: write enabled attached consistent connected asynchronous dcm_logging
-> vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
Host name: prisite-ossrvg
RVG name: ossrvg
DG name: ossdg
RVG state: enabled for I/O
Data volumes: 20
VSets: 0
SRL name: oss_srl_vol
SRL size: 300.00 G
Total secondaries: 1
Secondary:
Host name: sec_site-ossrvg
RVG name: ossrvg
DG name: ossdg
Data status: consistent, behind
Replication status: logging to DCM (needs dcm resynchronization)
Current mode: asynchronous
Logging to: DCM (contains 61101728 Kbytes) (SRL protection logging)
Timestamp Information: N/A
-> vxrvg -g ossdg cplist ossrvg
Name MBytes % Log Started/Completed
---- ------ ------ -----------------
point1 <Checkpoint overflowed>
-> vxrlink -g ossdg status to_sec_site_ossrvg
DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61101728 Kbytes (19%) of the Data Volume(s).
-> vxrlink -g ossdg stats to_sec_site_ossrvg
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
1 0 0 0 0 0 0 100000 1 10
primas1o{root} #: vxprint -PV
Disk group: ossdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
rl to_sec_site_ossrvg ossrvg CONNECT - - ACTIVE - -
rv ossrvg - ENABLED - - ACTIVE - -
primas1o{root} #: vxrlink -g ossdg -i 5 status to_sec_site_ossrvg
Mon Nov 30 17:24:01 2015
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443008 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443008 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443040 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443232 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443232 Kbytes (19%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 61443264 Kbytes (19%) of the Data Volume(s).
^Cprimas1o{root} #:
=====================================================================================================================
Secondary
=======================================================
-> vxprint -Pl
Disk group: ossdg
Rlink: to_prisite_ossrvg
info: timeout=500 packet_size=1452 rid=0.1263
latency_high_mark=10000 latency_low_mark=9950
bandwidth_limit=none
state: state=ACTIVE
synchronous=off latencyprot=off srlprot=autodcm
assoc: rvg=ossrvg
remote_host=prisite-ossrvg IP_addr=10.52.214.67 port=4145
remote_dg=ossdg
remote_dg_dgid=1327928106.32.secmas1o
remote_rvg_version=30
remote_rlink=to_sec_site_ossrvg
remote_rlink_rid=0.113591
local_host=sec_site-ossrvg IP_addr=10.52.214.68 port=4145
protocol: UDP/IP
flags: write enabled attached consistent connected
-> vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
Host name: prisite-ossrvg
RVG name: ossrvg
DG name: ossdg
RVG state: enabled for I/O
Data volumes: 20
VSets: 0
SRL name: oss_srl_vol
SRL size: 300.00 G
Total secondaries: 1
Secondary:
Host name: sec_site-ossrvg
RVG name: ossrvg
DG name: ossdg
Data status: consistent, behind
Replication status: logging to DCM (needs dcm resynchronization)
Current mode: asynchronous
Logging to: DCM (contains 61442560 Kbytes) (SRL protection logging)
Timestamp Information: N/A
-> vxrvg -g ossdg cplist ossrvg
The cplist command can only be used on a primary rvg
-> vxrlink -g ossdg status to_prisite_ossrvg
The status command can only be used on a primary rlink
-> vxrlink -g ossdg stats to_prisite_ossrvg
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
334089805 0 0 1927960 0 0 0 100000 1 10
Also, I ran the below command to check rlink bandwidth usgae and it is 0% utilised
Cprimas1o{root} #: vrstat -R
Mon Nov 30 17:43:49 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg: DCM contains 61543520 Kbytes.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o
1 0 0 0 0 0 0 100000 1 10
Mon Nov 30 17:43:59 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg: DCM contains 61543552 Kbytes.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o
0 0 0 0 0 0 0 100000 1 10
Bandwidth Utilization 0.00 Kbps.
I am wondering if rlink in connect/active state then why its not replicating ?
Regards
S
Solved! Go to Solution.
11-30-2015 12:04 PM
Hi,
You need to execute a resync command from primary
# vradmin -g <dg> resync <rvg>
That will flush out DCM logs to write & get in sync
G
11-30-2015 12:04 PM
Hi,
You need to execute a resync command from primary
# vradmin -g <dg> resync <rvg>
That will flush out DCM logs to write & get in sync
G
11-30-2015 08:13 PM
Hi Gaurav,
Why is it writing to DCM when Rlinks are in connect/Active state?
Regards
S.
12-01-2015 01:56 AM
I ran the resync and DCM starts to decay as shown below , During DCM resynch bandwidth utlistaion
was between 200 to 200 Mbps
primas1o{root} #: vradmin -g ossdg resync ossrvg
primas1o{root} #: vxrlink -g ossdg -i 5 status to_sec_site_ossrvg
Tue Dec 01 06:16:31 2015
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 68094560 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67938880 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67738336 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67550848 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67363296 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67201024 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67114592 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 67029664 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66943744 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66778144 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66558688 Kbytes (21%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66337760 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 66119200 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65904896 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65686624 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65474080 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65274432 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65124640 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 65031040 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64890144 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64674944 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64474400 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64291712 Kbytes (20%) of the Data Volume(s).
VxVM VVR vxrlink INFO V-5-1-12887 DCM is in use on rlink to_sec_site_ossrvg. DCM contains 64075296 Kbytes (20%) of the Data Volume(s).
^Cprimas1o{root} #: vrstat -R
Tue Dec 01 06:18:32 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg: DCM contains 63843296 Kbytes.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
23061 9941632 5 169 0 0 1024 787000 1 10
334112407 0 7 1921859 0 0 0 5000 63 20
Tue Dec 01 06:18:42 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg: DCM contains 63481664 Kbytes.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
1846 725440 5 11 0 0 0 786000 1 10
1628 0 5 30 0 0 0 5000 81 10
Bandwidth Utilization 283.38 Mbps.
Tue Dec 01 06:18:52 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg: DCM contains 63097952 Kbytes.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
1681 829824 8 15 0 0 0 787000 1 20
1859 0 7 32 0 0 0 5000 100 20
Bandwidth Utilization 324.15 Mbps.
Tue Dec 01 06:19:02 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg: DCM contains 62800448 Kbytes.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
1170 539136 5 20 0 0 0 786000 1 10
1355 0 5 29 0 0 0 5000 100 10
Bandwidth Utilization 210.60 Mbps.
T of the Data Volume(s).
^C
You have new mail in /var/mail//root
root@primas1o:/# vxrlink -g ossdg -i 5 status to_sec_site_ossrvg
Tue Dec 01 06:55:49 2015
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 3760 outstanding writes, occupying 165036 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 2297 outstanding writes, occupying 69886 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 2776 outstanding writes, occupying 126479 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 2423 outstanding writes, occupying 149798 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 1772 outstanding writes, occupying 85379 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 570 outstanding writes, occupying 23797 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4640 Rlink to_sec_site_ossrvg has 189 outstanding writes, occupying 576 Kbytes (0%) on the SRL
VxVM VVR vxrlink INFO V-5-1-4467 Rlink to_sec_site_ossrvg is up to date
Here, replication became uptodate and started wrinting to SRL, Howevernow bandwidth utilisation droped 10 times than earlier
and we can see writes piling up in SRL though slowly. Its been noticed that in few days it will again fill up SRL and starts dcm logging.
^C
root@primas1o:/# vxprint -Pl
Disk group: ossdg
Rlink: to_sec_site_ossrvg
info: timeout=500 packet_size=1452 rid=0.113591
latency_high_mark=10000 latency_low_mark=9950
bandwidth_limit=none
state: state=ACTIVE
synchronous=off latencyprot=off srlprot=autodcm
assoc: rvg=ossrvg
remote_host=sec_site-ossrvg IP_addr=10.52.214.68 port=4145
remote_dg=ossdg
remote_dg_dgid=1444153050.96.secfmcmas1o
remote_rvg_version=30
remote_rlink=to_prisite_ossrvg
remote_rlink_rid=0.1263
local_host=prisite-ossrvg IP_addr=10.52.214.67 port=4145
protocol: UDP/IP
flags: write enabled attached consistent connected asynchronous
root@primas1o:/# vrstat -R
Tue Dec 01 06:56:42 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg:26 writes behind, occupying 1885 Kbytes(0%) on SRL.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
12756 1299138 5 108 0 0 0 796000 1 10
334514874 0 6 1931402 0 0 0 99000 1 10
Tue Dec 01 06:56:52 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg: up-to-date.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
134 5547 5 0 0 0 0 816000 1 10
251 0 5 0 0 0 0 99000 1 10
Bandwidth Utilization 2.16 Mbps.
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg:183998 writes behind, occupying 2208411 Kbytes(0%) on SRL.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
2778 331098 8 22 0 0 0 594000 1 20
2668 0 10 37 0 0 0 5000 100 40
Bandwidth Utilization 129.33 Mbps.
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
6053 81766 5 4 0 0 0 570000 1 10
2880 0 5 9 0 0 0 5000 100 10
Bandwidth Utilization 31.94 Mbps.
Tue Dec 01 07:12:22 2015
Replicated Data Set ossrvg:
Data Status:
sec_site-ossrvg:353995 writes behind, occupying 2357604 Kbytes(0%) on SRL.
Network Statistics:
Messages Errors Flow Control
-------- ------ ------------
# Blocks RT(msec) Timeout Stream Memory Delays NW Bytes NW Delay Timeout
primas1o - secfmcmas1o
525 65633 5 0 0 0 0 600000 1 10
4047 0 5 0 0 0 0 5000 100 10
Bandwidth Utilization 25.64 Mbps.
How can we improve the situation that writes donot pile up in SRL and eventually making it full
12-01-2015 06:44 AM
Hi,
Check out on network, there are couple of likely conditions here .... DCM comes into play when SRL overflows ... SRL can overflow because of 2 major reasons..
1. Application is writing too fast & SRL volume is too low on size which makes SRL overflow ..
2. You had network glitch which didn't allow data to be sent to DR site eventually filling up SRL ..
G
12-01-2015 07:29 AM
Hi Gaurav,
Network team is saying there is no issue with network.
can you please comment why speed is fast during dcm resynch as compared to normal replication as notice in above logs
Regards
S.
12-01-2015 07:41 AM
Do you think there is need to chnage tunables as shown below
primas1o{root} #: vxrlink -g ossdg -i 5 -e stats to_sec_site_ossrvg
Tue Dec 01 15:32:53 2015
Messages :
--------
Number of blocks sent : 3988248
Errors :
------
No memory available : 0
No message slots available : 0
No memory available in nmcom pool on Secondary : 0
Timeout : 434
Missing packet : 183
Missing message : 228
Stream : 0
Checksum : 0
Unable to deliver due to transaction : 6
Messages :
--------
Number of blocks sent : 125978
Errors :
------
No memory available : 0
No message slots available : 0
No memory available in nmcom pool on Secondary : 0
Timeout : 86
Missing packet : 27
Missing message : 8
Stream : 0
Checksum : 0
Unable to deliver due to transaction : 0
Messages :
--------
Number of blocks sent : 75960
Errors :
------
No memory available : 0
No message slots available : 0
No memory available in nmcom pool on Secondary : 0
Timeout : 0
Missing packet : 0
Missing message : 2
Stream : 0
Checksum : 0
Unable to deliver due to transaction : 0
Messages :
--------
Number of blocks sent : 22514
Errors :
------
No memory available : 0
No message slots available : 0
No memory available in nmcom pool on Secondary : 0
Timeout : 0
Missing packet : 0
Missing message : 0
Stream : 0
Checksum : 0
Unable to deliver due to transaction : 0
Messages :
--------
Number of blocks sent : 189265
Errors :
------
No memory available : 0
No message slots available : 0
No memory available in nmcom pool on Secondary : 0
Timeout : 19
Missing packet : 11
Missing message : 11
Stream : 0
Checksum : 0
Unable to deliver due to transaction : 0
Messages :
--------
Number of blocks sent : 252878
Errors :
------
No memory available : 0
No message slots available : 0
No memory available in nmcom pool on Secondary : 0
Timeout : 1
Missing packet : 3
Missing message : 24
Stream : 0
Checksum : 0
Unable to deliver due to transaction : 0
Messages :
--------
Number of blocks sent : 168502
Errors :
------
No memory available : 0
No message slots available : 0
No memory available in nmcom pool on Secondary : 0
Timeout : 31
Missing packet : 10
Missing message : 13
Stream : 0
Checksum : 0
Unable to deliver due to transaction : 0
^Cprimas1o{root} #: vxtune
Tunable Value Description
----------------- -------- ---------------
vol_rvio_maxpool_sz 262144 RVIO Pool Size (KBytes)
vol_min_lowmem_sz 11568 Low Memory Threshold (KBytes)
vol_max_rdback_sz 65536 Readback Pool Size (KBytes)
vol_max_nmpool_sz 16384 NMCOM Pool Size (KBytes)
vol_max_wrspool_sz 16384 Write Shipping Pool Size (KBytes)
volpagemod_max_memsz 65536 Cache Size for FMR Metadata (KBytes)
primas1o{root} #: ssh secmas1o
Last login: Tue Dec 1 15:30:42 2015 from prisite-a-repla
You have new mail.
DISPLAY set to primas1o:0.0
secmas1o{root} #: vxtune
Tunable Value Description
----------------- -------- ---------------
vol_rvio_maxpool_sz 262144 RVIO Pool Size (KBytes)
vol_min_lowmem_sz 528 Low Memory Threshold (KBytes)
vol_max_rdback_sz 65536 Readback Pool Size (KBytes)
vol_max_nmpool_sz 131072 NMCOM Pool Size (KBytes)
vol_max_wrspool_sz 16384 Write Shipping Pool Size (KBytes)
volpagemod_max_memsz 65536 Cache Size for FMR Metadata (KBytes)
secmas1o{root} #: