Forum Discussion

mhab11's avatar
mhab11
Level 5
11 years ago

How do I verify my data size that will be replicated through VVR

I need to find a way to figure out the Mb/sec (Mb=Megabits) that will be replicated to my offsite and the pipe size I need in order to stay insync. I have been testing VVR and have all of my setting in place with everything working correctly. But with a 10Mb VPN internet pipe I have the line pegged at 10Mb all the time and am never able to catchup. When I looked at Performance monitor, Physical and Logical disks I am only generating about 4Mb/sec of disk writes which is what is being replicated the the offsite. Am I looking at the wrong data for my replication size needed? Is there a way to verify the Mb/sec that are going to be written the my offsite prior to starting VVR?

NOTE: If you are having similar problems make sure to read the whole thread, there were mutliple solutions that were needed to verify and fix the problem of the data never catching up completely.

 

business continuity
Replicator Option
Windows Server (2003-2008)
  • mikebounds's avatar
    mikebounds
    11 years ago

    To see what you are writing, use vxstat:

    If you think you are writing 4Mb/sec then you should see:

    On the primary site, 4Mb/sec going to SRL volume and an additional 4Mb/sec for the total of all the data volumes  and on the secondary site the SRL is only used to keep track of writes so there should be a small amount to SRL and 4Mb/sec to data volumes.

    Maybe you have your units mixed up and your network is correct at 10Mbits/sec, but your disk is writing 4Mbytes/sec.  

    vxstat reports in 512k block so multiply by 2048 to get Mbytes or multiply by 16384 to get Mbits.

    Mike

  • Wally_Heim's avatar
    Wally_Heim
    Level 6
    11 years ago

    Hi mhab11,

    We do have a utility called vradvisor which can be used to properly size your configuration.  It is located in the \Tools\storage_foundation folder on the media in 6.0.1 release (I think it is in the same location on most release media.)

    It is best used with an I/O data collection of at least 2 weeks.  This would give the best results of what I/O your configuration generates on the volumes being replicated.  The longer the collection period the better the results.

    When you do your initial sync of your replicated volumes VVR has to copy all the data that is currently on the volume(s) to the remote site.  This process can overwhelm replication links and can take many days or weeks to complete..

    You can use the following command to see how much data is needing to be replicated in 5 second intervals:

       vxrlink -i 5 status <rlink_name>

    The rlink name is available by running this command:

       vxrlink -VPl

    The rate of replication and errors can be seen with this command.

       vxrlink -e -i 5 stats <rlink_name>

    I often pipe the stats command to a text file and look at it after its been running for 10+ minutes.

    Also the first line of the stats output should be ignored.  It is miscalculated and does not contain good data.

    if you need to check memory used by VVR you should run this command:

       vxmemstat -i 5

    I hope this helps.  Please share more configuration details and the outputs of these command if you are still having problems.

    Also open a case with Symantec Techncial Support if you need more indepth assistance with tuning your replication configuration or if you need alternate initial sync options.

    Thank you,

    Wally

     

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    To see what you are writing, use vxstat:

    If you think you are writing 4Mb/sec then you should see:

    On the primary site, 4Mb/sec going to SRL volume and an additional 4Mb/sec for the total of all the data volumes  and on the secondary site the SRL is only used to keep track of writes so there should be a small amount to SRL and 4Mb/sec to data volumes.

    Maybe you have your units mixed up and your network is correct at 10Mbits/sec, but your disk is writing 4Mbytes/sec.  

    vxstat reports in 512k block so multiply by 2048 to get Mbytes or multiply by 16384 to get Mbits.

    Mike

  • mhab11's avatar
    mhab11
    Level 5
    11 years ago

    Thanks, I did a vxsts -g diskgroup -i 2 to get a continues print so I could see what my range was. my Blocks Write ranges from 650 to 1000 for the volume I am replicating. So that is 10649600Mbps?? I did mean Mb not MB, MBx8=Mb. I do have 2 disk arrays with one VD presented from each in the volume. Would that be 2 writes is has to do therefor doubling the Blocks Write?

      Primary Server

           Operations       Blocks           Avg.Time
           Read  Write     Read  Write     Read    Write

    (D:)     5       78       2269    906        15      222

    Secondary server

           Operations       Blocks           Avg.Time
           Read  Write     Read  Write     Read    Write

             0        5           0      2421         0       16

     

    So you can see the secondary is more then Double the Primary, this is what I cannot figure out where the excess data is coming from.

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    Sorry should have been divide by 2048 for MBytes and then this need to be multiplied by 8 for Mbits (or divide original figure by 256 instead) - so

    For 900 blocks for 2 seconds this is 450 blocks a second which is 450/256 = 1.76 Mbits/s

     

    I don't understand your output as you should see entries for each volume - so you should see a line for srl volume as well as D:

    Can you provide full output of vxstat for at least 2 iterations on both systems and also output from vxprint

    Mike

  • mhab11's avatar
    mhab11
    Level 5
    11 years ago

    Primary Server

    Wed Dec 11 12:29:18 2013

    Volume  AvMail_Snap133  0       0       0       0       0       0
    Volume  AvMail_Snap134  0       0       0       0       0       0
    Volume  AvMail_Snap132  0       0       0       0       0       0
    Volume  AvMail (D:)     5       63      2544    714     11      182
    Volume  Volume          0       63      0       783     0       97

    Wed Dec 11 12:29:20 2013

    Volume  AvMail_Snap133  0       0       0       0       0       0
    Volume  AvMail_Snap134  0       0       0       0       0       0
    Volume  AvMail_Snap132  0       0       0       0       0       0
    Volume  AvMail (D:)     5       63      2544    714     11      182
    Volume  Volume          0       63      0       783     0       97

    Wed Dec 11 12:29:22 2013

    Volume  AvMail_Snap133  0       0       0       0       0       0
    Volume  AvMail_Snap134  0       0       0       0       0       0
    Volume  AvMail_Snap132  0       0       0       0       0       0
    Volume  AvMail (D:)     4       79      2064    934     7       166
    Volume  Volume          0       80      0       1025    0       76

    Wed Dec 11 12:29:24 2013

    Volume  AvMail_Snap133  0       0       0       0       0       0
    Volume  AvMail_Snap134  0       0       0       0       0       0
    Volume  AvMail_Snap132  0       0       0       0       0       0
    Volume  AvMail (D:)     4       79      2064    934     7       166
    Volume  Volume          0       80      0       1025    0       76

     

    Secondary Server

    Volume  REP             0       0       0       0       0       0
    Volume  AvMail          0       4       0       1970    0       16

    Wed Dec 11 12:34:08 2013

    Volume  REP             0       0       0       0       0       0
    Volume  AvMail          0       4       0       1970    0       16

    Wed Dec 11 12:34:10 2013

    Volume  REP             0       0       0       0       0       0
    Volume  AvMail          0       5       0       2421    0       16

  • mhab11's avatar
    mhab11
    Level 5
    11 years ago

    Primary


    Diskgroup = Archive
    TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
    dg Archive      Archive      -        -        -        -        -       -
    dm Harddisk11   LanAcesLogs2 -        838849977-        -        -       -
    dm Harddisk6    LanAcesLogs1 -        838849977-        -        -       -
    sd LanAcesLogs1-01 MailArchive-01   ENABLED  734003200 0        -        -
     -
    sd LanAcesLogs2-01 MailArchive-02   ENABLED  734003200 0        -        -
     -
    v  MailArchive  -            ENABLED  734003200-        ACTIVE   -       -
    pl MailArchive-01 MailArchive  ENABLED  734003200 -        ACTIVE   -       -
    pl MailArchive-02 MailArchive  ENABLED  734003200 -        ACTIVE   -       -

    Diskgroup = AvMailDiskGroup
    TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
    sd 0-1-01       AvMailVolume-03   ENABLED  1258291200 0        -        -
    -
    sd 0-1-02       AvMailVolume_DCO1-02   ENABLED  4096     0        -        -
       -
    sd 0-1-03       AvMailVolume-06   ENABLED  512      0        -        -       -
    sd 0-2-01       Snap1-01     ENABLED  1258291200 0        -        -       -
    sd 16-1-01      AvMailVolume-05   ENABLED  1258291200 0        -        -
    -
    sd 16-1-02      AvMailVolume_DCO1-04   ENABLED  4096     0        -        -
       -
    sd 16-2-01      Snap3-01     ENABLED  1258291200 0        -        -       -
    sd 8-1-01       AvMailVolume-04   ENABLED  1258291200 0        -        -
    -
    sd 8-1-02       AvMailVolume_DCO1-03   ENABLED  4096     0        -        -
       -
    sd 8-1-03       AvMailVolume-07   ENABLED  512      0        -        -       -
    sd 8-2-01       Snap2-01     ENABLED  1258291200 0        -        -       -
    dg AvMailDiskGroup AvMailDiskGroup     -        -        -        -        -
       -
    v  AvMailVolume -            ENABLED  1258291200-        ACTIVE   -       -
    pl AvMailVolume-01 AvMailVolume ENABLED  1258291200 -        ACTIVE   -       -
    pl AvMailVolume-02 AvMailVolume ENABLED  1258291200 -        ACTIVE   -       -
    pl AvMailVolume-03 Snap1        ENABLED  1258291200 -        ACTIVE   -       -
    pl AvMailVolume-04 Snap2        ENABLED  1258291200 -        ACTIVE   -       -
    pl AvMailVolume-05 Snap3        ENABLED  1258291200 -        ACTIVE   -       -
    pl AvMailVolume-06 AvMailVolume ENABLED  512      -        ACTIVE   -       -
    pl AvMailVolume-07 AvMailVolume ENABLED  512      -        ACTIVE   -       -
    sd Disk1-01     AvMailVolume-02   ENABLED  1258291200 0        -        -
    -
    dm Harddisk10   16-2         -        1260379512-        -        -       -
    dm Harddisk2    Mirror1      -        1363146689-        -        -       -
    dm Harddisk3    0-1          -        1260379512-        -        -       -
    dm Harddisk4    8-1          -        1260379512-        -        -       -
    dm Harddisk5    16-1         -        1260379512-        -        -       -
    dm Harddisk7    Mirror2      -        1363146689-        -        -       -
    dm Harddisk8    0-2          -        1260379512-        -        -       -
    dm Harddisk9    8-2          -        1260379512-        -        -       -
    sd Mirror1-01   AvMailVolume_DCO1-01   ENABLED  4096     0        -        -
       -
    sd Mirror1-02   REP-02       ENABLED  204800   0        -        -       -
    sd Mirror2-01   REP-01       ENABLED  204800   0        -        -       -
    sd Mirror2-02   AvMailVolume-01   ENABLED  1258291200 0        -        -
    -
    rv NYVC         -            -        -        -        ACTIVE   -       -
    rl ANP2NYC      NYVC         -        -        -        ACTIVE   -       -
    v  REP          -            ENABLED  204800   -        ACTIVE   -       -
    pl REP-01       REP          ENABLED  204800   -        ACTIVE   -       -
    pl REP-02       REP          ENABLED  204800   -        ACTIVE   -       -
    v  Snap1        -            ENABLED  1258291200-        ACTIVE   -       -
    pl Snap1-01     Snap1        ENABLED  1258291200 -        ACTIVE   -       -
    v  Snap2        -            ENABLED  1258291200-        ACTIVE   -       -
    pl Snap2-01     Snap2        ENABLED  1258291200 -        ACTIVE   -       -
    v  Snap3        -            ENABLED  1258291200-        ACTIVE   -       -
    pl Snap3-01     Snap3        ENABLED  1258291200 -        ACTIVE   -       -

    Diskgroup = WIN-MSBKUUROFN9-Dg0
    TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
    v  HarddiskVolume1 -            DISABLED 204800   -        Stopped
    -       -
    v  HarddiskVolume2 -            ENABLED  976564224-        ACTIVE   -       -
     

     

    Secondary


    Diskgroup = BasicGroup
    TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0

    Diskgroup = AvMailDiskGroup
    TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
    dg AvMailDiskGroup AvMailDiskGroup     -        -        -        -        -
       -
    v  AvMailVolume -            ENABLED  1258291200-        ACTIVE   -       -
    pl AvMailVolume-01 AvMailVolume ENABLED  1258291200 -        ACTIVE   -       -
    pl AvMailVolume-02 AvMailVolume ENABLED  512      -        ACTIVE   -       -
    pl AvMailVolume-03 AvMailVolume ENABLED  512      -        ACTIVE   -       -
    sd Disk1-01     AvMailVolume-01   ENABLED  629145600 0        -        -       -

    sd Disk1-02     REP-01       ENABLED  204800   0        -        -       -
    sd Disk1-03     AvMailVolume-02   ENABLED  512      0        -        -       -
    sd Disk2-01     AvMailVolume-01   ENABLED  629145600 0        -        -       -

    sd Disk2-02     AvMailVolume-03   ENABLED  512      0        -        -       -
    dm Harddisk2    Disk1        -        976768002-        -        -       -
    dm Harddisk3    Disk2        -        976768002-        -        -       -
    rv NYVC         -            -        -        -        ACTIVE   -       -
    rl NYC2ANP      NYVC         -        -        -        ACTIVE   -       -
    v  REP          -            ENABLED  204800   -        ACTIVE   -       -
    pl REP-01       REP          ENABLED  204800   -        ACTIVE   -       -

    Diskgroup = MHSWS001NYC-3-Dg0
    TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
    v  HarddiskVolume1 -            DISABLED 204800   -        Stopped
    -       -
    v  HarddiskVolume2 -            ENABLED  976562176-        ACTIVE   -       -

    C:\Windows\system32>

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    If VVR is catching up, then what is why you will see more writes at secondary - so VVR should catch up.  As Wally says you can use:

     

       vxrlink -i 5 status <rlink_name>

    The rlink name is available by running this command:

       vxrlink -VPl

    to verify that you are catching up so this should show you are catching up by about 1000 blocks a second  (i.e if you write 900 block at primary and 1900 - 2400 at secondary, then VVR should be catching up by the difference) - can you provide output of  vxrlink -i 2 status

    Mike

  • mhab11's avatar
    mhab11
    Level 5
    11 years ago

    Primary

    C:\Users\mhabicht>vxrlink -i 2 status ANP2NYC
    12/11/2013 14:19:06
    RLINK is in autosync. Less than 1% (923552 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (921120 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (918688 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (918080 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (918688 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (916864 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (913824 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (912608 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (913216 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (912000 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (910784 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (910176 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (908352 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (905312 Kbytes) remaining.
    RLINK is in autosync. Less than 1% (904704 Kbytes) remaining.
     

     

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    This shows VVR is catching up, but it is catching from DCM, not SRL and if you only have 10MB left in DCM, then it SHOULD be clear in a matter of minutes, however, I suspect you maybe seeing an issue which I saw in UNIX VVR 5.0 which was fixed by 5.1 in for VVR on UNIX so I am surprised it is not fixed in Windows.

    This is what I think the issue is:

    In earlier versions of VVR on UNIX (5.0, maybe even 4.1), the DCM autosync would not catch up if there was a modest amount of writes going on as the process was to replicate the all the dirty blocks in DCM and then check DCM again and replicate any new dirty blocks so this is flawed as unless you have a quiet period where not many block are written, VVR never catches up.  This was fixed in later version by essentially putting a checkpoint in SRL so that VVR could drain writes from DCM after one (or maybe 2) passes of the DCM.

    If this is the case, then if you can briefly pause the apps so it doesn't write then VVR should catch up and then it should stay up to date as your bandwidth seems high enough for how much you are writing.

    If this is the issue, then Wally might know whether the same change to use the SRL at the end of DCM autosync was implemented in Windows by 5.1SP2 and if not, if this change was implemented in later 5.1 SPs or in 6.x.

    Mike