Wanoptimization Issue

[Angel_Fontana]

Hi guys,

I've got an issue with Wanoptimization option. Although it's enabled, all the copies of my server in site B connected to Main site by 100Mb point to point circuit are using Non-Optimized traffic.

Furthermore, the copy (file server in vmware machine using another vmware machine as VM backup host in the same vmware host) has decreased the speed from 80 Mb/s to 50 Mb/s suddenly. I mean, previous weekends, the copy took 50 hours, now it's taking 77H copying the same data. I've just launched a test copying only O.S. disk and as you can see above it has a constant download speed of 50Mb/s

Other data: I've got set up resiliance network with between Master/Media server and file server, there is no "Accelerator" license (don't ask, long story...). File server is Windows 2012, Master and Media Server is Netbackup Appliance 5230, Vmware host ESX 5.1

So, I've got two problems, Wanoptimization seems not to be working, and the decreasing of speed.

Do you know where the problem could be?

Thanks in advance!

---

         260          0.00             0             0              6.53            52         52210
          270          0.00             0             0              6.63            53         53065
          280          0.00             0             0              6.59            52         52700
          290          0.00             0             0              6.73            53         53839
          300          0.00             0             0              6.54            52         52351
          310          0.00             0             0              6.28            50         50213
          320          0.00             0             0              6.47            51         51773
          330          0.00             0             0              6.36            50         50917
          340          0.00             0             0              6.75            53         53986
          350          0.00             0             0              6.70            53         53584
          360          0.00             0             0              6.75            53         53974
          370          0.00             0             0              6.75            54         54015
          380          0.00             0             0              6.88            55         55013
          390          0.00             0             0              6.48            51         51829
          400          0.00             0             0              6.70            53         53604
          410          0.00             0             0              6.74            53         53913
          420          0.00             0             0              6.78            54         54223
          430          0.00             0             0              6.73            53         53827
          440          0.00             0             0              6.71            53         53678
          450          0.00             0             0              6.83            54         54621
          460          0.00             0             0              6.86            54         54874
          470          0.00             0             0              6.41            51         51306
          480          0.00             0             0              6.64            53         53086
          490          0.00             0             0              6.74            53         53948
          500          0.00             0             0              6.79            54         54327
          510          0.00             0             0              6.72            53         53775
          520          0.00             0             0              6.80            54         54406
          530          0.00             0             0              6.90            55         55160
          540          0.00             0             0              6.73            53         53826
          550          0.00             0             0              6.47            51         51720
          560          0.00             0             0              6.77            54         54162
          570          0.00             0             0              6.85            54         54826
          580          0.00             0             0              6.67            53         53377
          590          0.00             0             0              6.02            48         48176
          600          0.00             0             0              6.51            52         52061
          610          0.00             0             1              6.39            51         51148
          620          0.00             0             0              6.92            55         55386
          630          0.00             0             5              6.68            53         53457
          640          0.00             0             0              6.37            50         50982
          650          0.00             0             0              5.46            43         43674
          660          0.00             0             0              6.32            50         50530
          670          0.00             0             0              5.91            47         47308
          680          0.00             0             0              5.27            42         42180
          690          0.00             0             0              5.31            42         42518
          700          0.00             0             0              5.70            45         45625
          710          0.00             0             0              5.52            44         44132
          720          0.00             0             0              6.14            49         49126
          730          0.00             0             0              5.04            40         40341
          740          0.00             0             0              4.88            39         39028
          750          0.00             0             0              5.67            45         45348
          760          0.00             0             0              4.81            38         38453
          770          0.00             0             0              4.49            35         35913
          780          0.00             0             0              6.49            51         51912
          790          0.00             0             0              5.50            44         44016
          800          0.00             0             0              5.10            40         40794
          810          0.00             0             0              5.20            41         41627
          820          0.00             0             0              5.12            40         40992
          830          0.00             0             0              5.19            41         41510
          840          0.00             0             0              4.85            38         38791
          850          0.00             0             0              5.00            40         40011
          860          0.00             0             0              5.98            47         47817
          870          0.00             0             0              6.17            49         49356
          880          0.00             0             0              5.96            47         47714
          890          0.00             0             1              5.61            44         44864
          900          0.00             0             0              5.57            44         44533
          910          0.00             0             0              6.18            49         49436
          920          0.00             0             0              5.23            41         41806
          930          0.00             0             0              4.95            39         39596
          940          0.00             0             0              5.27            42         42135
          950          0.00             0             0              5.80            46         46364
          960          0.00             0             0              4.92            39         39388
          970          0.00             0             0              5.81            46         46514
          980          0.00             0             0              5.43            43         43431
          990          0.00             0             0              5.14            41         41152
         1000          0.00             0             0              5.85            46         46830
         1010          0.00             0             0              5.97            47         47784
         1020          0.00             0             0              5.99            47         47959
         1030          0.00             0             0              5.54            44         44307
         1040          0.00             0             0              5.40            43         43205
         1050          0.00             0             0              5.67            45         45397
         1060          0.00             0             0              5.65            45         45185
         1070          0.00             0             0              6.24            49         49940
         1080          0.00             0             0              5.69            45         45510
         1090          0.00             0             0              5.80            46         46364
         1100          0.00             0             0              5.96            47         47668
         1110          0.00             0             0              5.14            41         41118
         1120          0.00             0             0              5.56            44         44441
         1130          0.00             0             0              6.40            51         51239
         1140          0.00             0             0              6.78            54         54251
         1150          0.00             0             0              6.64            53         53132
         1160          0.00             0             0              6.79            54         54331
         1170          0.00             0             0              6.34            50         50759
         1180          0.00             0             0              6.95            55         55589
         1190          0.00             0             0              7.01            56         56113
         1200          0.00             0             0              6.87            54         54969
         1210          0.00             0             0              6.92            55         55387
         1220          0.00             0             0              6.79            54         54352
         1230          0.00             0             0              6.81            54         54506
         1240          0.00             0             0              6.99            55         55940
         1250          0.00             0             0              6.89            55         55126
         1260          0.00             0             0              6.82            54         54521
         1270          0.00             0             0              6.90            55         55196
         1280          0.00             0             0              6.98            55         55849
         1290          0.00             0             0              6.99            55         55911
         1300          0.00             0             0              7.04            56         56291
         1310          0.00             0             0              7.00            56         56037
         1320          0.00             0             0              6.99            55         55882
         1330          0.00             0             0              7.03            56         56242
         1340          0.00             0             0              6.99            55         55947
         1350          0.00             0             0              6.92            55         55397
         1360          0.00             0             0              7.02            56         56177
         1370          0.00             0             0              6.94            55         55552
         1380          0.00             0             0              7.04            56         56359
         1390          0.00             0             0              6.93            55         55426
         1400          0.00             0             0              6.93            55         55428
         1410          0.00             0             0              6.97            55         55798
         1420          0.00             0             0              6.98            55         55878
         1430          0.00             0             0              7.01            56         56065
         1440          0.00             0             0              6.92            55         55390
         1450          0.00             0             0              6.89            55         55127
         1460          0.00             0             0              6.94            55         55538
         1470          0.00             0             0              6.94            55         55509
         1480          0.00             0             0              6.97            55         55782
         1490          0.00             0             0              6.89            55         55156
         1500          0.00             0             0              6.85            54         54839
         1510          0.00             0             1              6.84            54         54747
         1520          0.00             0             0              6.74            53         53950
         1530          0.00             0             0              6.81            54         54461
         1540          0.00             0             0              6.76            54         54071
         1550          0.00             0             0              6.86            54         54848
         1560          0.00             0             0              6.83            54         54616
         1570          0.00             0             0              6.86            54         54890
         1580          0.00             0             0              6.83            54         54662
         1590          0.00             0             0              6.88            55         55075
         1600          0.00             0             0              6.83            54         54677
         1610          0.00             0             0              6.77            54         54197
         1620          0.00             0             0              6.59            52         52711
         1630          0.00             0             0              6.38            51         51052
         1640          0.00             0             0              6.57            52         52535
         1650          0.00             0             0              6.72            53         53784
         1660          0.00             0             0              6.79            54         54349
         1670          0.00             0             0              6.79            54         54287
         1680          0.00             0             0              6.83            54         54613
         1690          0.00             0             0              6.75            54         54033
         1700          0.00             0             0              6.70            53         53612
         1710          0.00             0             0              6.88            55         55024
         1720          0.00             0             0              6.81            54         54487
         1730          0.00             0             0              6.75            53         53975
         1740          0.00             0             0              6.69            53         53540
         1750          0.00             0             0              6.81            54         54501
         1760          0.00             0             0              6.35            50         50832
         1770          0.00             0             0              6.68            53         53463
         1780          0.00             0             0              6.85            54         54801
         1790          0.00             0             0              6.82            54         54521
         1800          0.00             0             0              6.37            50         50958
         1810          0.00             0             0              6.78            54         54237
         1820          0.00             0             0              6.82            54         54561
         1830          0.00             0             0              6.70            53         53607
         1840          0.00             0             0              6.79            54         54354
         1850          0.00             0             0              6.72            53         53794
         1860          0.00             0             0              6.66            53         53302
         1870          0.00             0             0              6.53            52         52222
         1880          0.00             0             0              6.54            52         52328
         1890          0.00             0             0              6.85            54         54766
         1900          0.00             0             0              6.48            51         51867
         1910          0.00             0             0              6.77            54         54125
         1920          0.00             0             0              6.76            54         54063
         1930          0.00             0             0              6.66            53         53293
         1940          0.00             0             0              6.90            55         55214
         1950          0.00             0             0              5.18            41         41435
         1960          0.00             0             0              4.61            36         36889
         1970          0.00             0             0              4.35            34         34765
         1980          0.00             0             0              4.30            34         34412
         1990          0.00             0             0              4.26            34         34083
         2000          0.00             0             0              5.91            47         47274
         2010          0.00             0             0              6.83            54         54626
         2020          0.00             0             0              6.42            51         51385
         2030          0.00             0             0              6.48            51         51854
         2040          0.00             0             0              6.38            51         51001
         2050          0.00             0             0              6.88            55         55022
         2060          0.00             0             0              6.84            54         54717
         2070          0.00             0             0              6.51            52         52090
         2080          0.00             0             0              6.98            55         55827
         2090          0.00             0             0              6.97            55         55743
         2100          0.00             0             0              6.97            55         55737
         2110          0.00             0             0              6.97            55         55771
         2120          0.00             0             0              6.58            52         52652
         2130          0.00             0             0              7.02            56         56166
         2140          0.00             0             0              7.00            55         55983
         2150          0.00             0             0              6.87            54         54938
         2160          0.00             0             0              7.00            55         55965
         2170          0.00             0             0              7.02            56         56172
         2180          0.00             0             0              6.99            55         55933
         2190          0.00             0             0              6.57            52         52568
         2200          0.00             0             0              6.72            53         53731
         2210          0.00             0             0              6.63            53         53031
         2220          0.00             0             0              6.77            54         54169
         2230          0.00             0             0              6.71            53         53642
         2240          0.00             0             0              6.41            51         51310
         2250          0.00             0             0              6.42            51         51384
         2260          0.00             0             0              6.91            55         55314
         2270          0.00             0             0              6.94            55         55517
         2280          0.00             0             0              6.92            55         55384
         2290          0.00             0             0              6.77            54         54122
         2300          0.00             0             0              6.84            54         54683
         2310          0.00             0             0              6.96            55         55674
         2320          0.00             0             0              6.81            54         54487
         2330          0.00             0             0              6.57            52         52564
         2340          0.00             0             0              6.49            51         51922
         2350          0.00             0             0              6.62            52         52985
         2360          0.00             0             0              6.80            54         54377
         2370          0.00             0             0              6.73            53         53866
         2380          0.00             0             0              6.84            54         54750
         2390          0.00             0             0              6.50            51         51964
         2400          0.00             0             0              6.76            54         54089
         2410          0.00             0             0              6.81            54         54513
         2420          0.00             0             0              6.44            51         51558
         2430          0.00             0             5              6.78            54         54219
         2440          0.00             0             0              6.82            54         54586
         2450          0.00             0             0              6.87            54         54938
         2460          0.00             0             0              6.72            53         53742
         2470          0.00             0             0              6.82            54         54597
         2480          0.00             0             0              6.76            54         54073
         2490          0.00             0             0              6.26            50         50091
         2500          0.00             0             0              6.70            53         53575
         2510          0.00             0             0              6.70            53         53620
netbu.WANOptimization>

---

[WVT]

Did you try monitoring wanopt at appliance using traffic command?

[Angel_Fontana]

Hiya,

Yep, I did, it's the output that I copied in the message. this one:

netbu.WANOptimization> Traffic 10 0
                               OPTIMIZED                             NON-OPTIMIZED
  Time Offset      MB/sec        Mb/sec        Kb/sec            MB/sec        Mb/sec        Kb/sec
     (sec)          (avg)         (avg)         (avg)             (avg)         (avg)         (avg)
  -----------    ----------    ----------    ----------        ----------    ----------    ----------
           10          0.00             0             0              0.00             0             2
           20          0.00             0             2              0.01             0            65
           30          0.00             0             2              0.00             0            28
           40          0.00             0             0              0.00             0            11
           50          0.00             0             0              0.00             0             0
           60          0.00             0             0              0.00             0            21
           70          0.00             0             0              0.00             0             4
           80          0.00             0             0              0.00             0            22
           90          0.00             0             0              0.00             0             2
          100          0.00             0             0              0.00             0            21
          110          0.00             0             0              0.00             0             1
          120          0.00             0             0              0.00             0             3
          130          0.00             0             0              2.02            16         16130
          140          0.00             0             0              3.26            26         26097
          150          0.00             0             0              2.43            19         19402
          160          0.00             0             0              3.22            25         25770
          170          0.00             0             0              2.17            17         17345
          180          0.00             0             0              3.52            28         28181
          190          0.00             0             0              6.60            52         52765
          200          0.00             0             0              6.81            54         54462
          210          0.00             0             0              6.85            54         54787
          220          0.00             0             0              6.39            51         51097
          230          0.00             0             0              5.89            47         47152
          240          0.00             0             0              6.63            53         53062
          250          0.00             0             0              6.62            52         52958
          260          0.00             0             0              6.53            52         52210
          270          0.00             0             0              6.63            53         53065
          280          0.00             0             0              6.59            52         52700
          290          0.00             0             0              6.73            53         53839
          300          0.00             0             0              6.54            52         52351
          310          0.00             0             0              6.28            50         50213
          320          0.00             0             0              6.47            51         51773
          330          0.00             0             0              6.36            50         50917
          340          0.00             0             0              6.75            53         53986
          350          0.00             0             0              6.70            53         53584
          360          0.00             0             0              6.75            53         53974
          370          0.00             0             0              6.75            54         54015
          380          0.00             0             0              6.88            55         55013
          390          0.00             0             0              6.48            51         51829
          400          0.00             0             0              6.70            53         53604
          410          0.00             0             0              6.74            53         53913
          420          0.00             0             0              6.78            54         54223

[WVT]

Do you have appliances at both ends? Do you have output of traffic command when it was "working"?

[WVT]

Latency between sites?

[Angel_Fontana]

The output is when this copy (and only this copy) is working. I've only got one appliance in Main Site.

About latency, sorry, it's in Spanish but I think it's understandable:

Estadísticas de ping para 10.81.50.22:
    Paquetes: enviados = 3626, recibidos = 3626, perdidos = 0
    (0% perdidos),
Tiempos aproximados de ida y vuelta en milisegundos:
    Mínimo = 9ms, Máximo = 26ms, Media = 9ms

So, no packet losts, and Latency Average = 9ms

[WVT]

9ms would only need a tcp window of a little more than 100KB to utilize bandwidth. Might not meet threshold for WanOpt to kick in. Is everything hooked up exactly the same as when it was 80mb/s?

[sdo · Accepted solution]

A nice overview of NetBackup Appliance WAN Optimization here:

https://www-secure.symantec.com/connect/articles/netbackup-appliances-network-optimization

...which describes how the WAN optimization feature may not result in tangible gains in all topology and connectivity scenarios.

[Angel_Fontana]

So, I shouldn't be worried about optimization network as my latency is good, right?

About the speed problems, right, there wasn't any change. This is the complete copy. It's done in weekend when nobody is in the office so the circuit is completely free. 

[WVT · Accepted solution]

No, what I am saying is at 9ms, it may not meet the threshold for the algorithm they use. The way wanopt works in general is some amount of packets are placed onthe wire and an acknowledge is received for that group, then it is cranked up a little more until the other side says whoa and the two sides reach a happy medium. Response times are a consideration. If one packet is lost out of a group, the other side can say, just resend me packet 37, I got the rest okay. Great for lossy, high latency networks...maybe not so great for yours. There is no way Veritas would be doing anything outside of an existing RFC so if its enabled and isnt optimizing then some criteria is not being met or its a bug. By the way, off the top of my head, the speed you are getting is about what you would get using a 64k TCP window...which is usually the starting point for Windows. In order to fill 100mbps pipe at 9ms I would have to put about 110kB of unacknowledged data on the wire.

[Angel_Fontana]

Hiya,

I did another test this weekend. Instead of use vmware policy I changed it to Windows one. As there is two data volumes (E & F) I'm using multiple data streams. Copying from both units at the same time Netbackup is using more than 8600 Kb/s instead of 5400 as it does with vmware policy. But, none of the copies never goes faster than 4300Kb/s although one of them has finished.

To be honest, I was investigating about your comment "In order to fill 100mbps pipe at 9ms I would have to put about 110kB of unacknowledged data on the wire" and I don't know how to do that.

[sdo]

Although this is Windows article, it is anintroduction to one facet of RSS (Receive Side Scaling):

https://msdn.microsoft.com/en-us/library/windows/hardware/ff556942(v=vs.85).aspx

There's a lot of topic/discussion on the internet about the trade-off / balancing between packet size, bandwidth, latency, buffer-size.  All these technologies are converging in an attempt to fully utilize available network bandwidth in the most efficient manner - essentially covered by RFC1323:

https://en.wikipedia.org/wiki/TCP_window_scale_option

.

Most of this is supposed to be hidden by the OS (at both ends), the NICs (at both ends), the NIC drivers (at both ends), and of course the switching infrastructure in between.  So, there's no NetBackup setting for any of this (apart perhaps from client side buffer size):

https://www.veritas.com/support/en_US/article.TECH28339

.

What WVT is doing is showing you the fundamental breakdown of latency, wire speed and how much unacknowledged data one can get in flight.  Basicaly, WVT knows this so well that he is able to roughly calculate (in his head) the fundamental figures.

.

In your figures you say Kb/s.. do you really mean kilo-bits per second?  Or did you mean to say KB/s kilo-bytes per second (i.e. what NetBackup shows).  Remember, network guys/girls talk in bits, whereas storage and backup guys/girls talk in bytes.

[WVT]

So now we have more bandwidth being used because we have two flows going instead of one. But when one stream runs, the TCP window isnt scaling up to use more of the pipe. It doesnt mean anything is broken. Read up on Bandwidth Delay Product on interwebs.

[Angel_Fontana]

Hi guys,

Sorry, I was talking in KBytes :)

So, as storage and backup guy it's a little difficult to me understand the doc you sent me. 

BTW, My thought is why previous weeks Netbackup was using the full circuit (near to 100 Mb/s, bits right now) and since 3 weeks ago, it's copying on half of speed. The only thing that I know is I don't set up anything new in windows servers or netbackup or circuit. If I copy a 4GB files from my computer to the copied server I get almost 100Mb/s. I guess it's not the same copy from my windows machine to another windows machine than netbackup copies.

[sdo]

Has a network admin implemented any traffic shaping?  Maybe perhaps QoS for specific protocols/addresses/ports/timeframes?

What speed do you get if you copy the same 4GB file from the replication source server to the replication target server?

[Angel_Fontana]

Sdo, our network admin implemented a QoS for Call Manager (VoIP). I asked for stopped it and, theoretically, it was stopped. Do you know any case where QoS for VoIP interfered with Netbackup?

About speed, just a remind:

- Netbackup Appliance and my computer (windows 7) are in Main Site

- Server Copied (windows 2012) in Secondary Site

So, copying from server copied to netbackup share folder it takes 850sec., 3638 KB/s. I was looking to the graphics and the speed wasn't stable. I mean, it reached near to 10MB/s and decrease to 11KB, then increase to near of 10 MB again.

Coping from server copied to my computer share folder it takes 335sec., 9231 KB/s. The connection was stable all the time, near to 10MB

[sdo]

Not heard of any direct interference.  What I was concerned about was whether QoS was implemented in brutal manner affecting all point-to-point conversations.  However, if QoS was implemented for specific point-to-point address which are nothing to do with NetBackup then you should not have been affected, unless the CPU or buffering overhead required for the network WAN entry/exit points is so intense that it slows down all traffic, maybe?  Maybe when QoS is enabled on the devices that it was enabled on - maybe this means that all traffic has to be buffered through a second stage so that it can be inspected to see if any QoS rules apply, maybe?  Do you see all these 'maybes'?  These definitely mean that we do not understand what the QoS implementation is actually doing.  A known unknown, so to speak.  So, we're slightly in the dark right now, and we can only fully understand the problem if we fully understand all of the devices, all of the stages and and all of the network hops.  :)   Do you see how your network admin feels that he/she can confidently say that you won't have been affected.  But how can they/you prove that?

Apologies Angel, but it's still not clear to me that your copy test was actually between the two devices which suffered the original problem?  Can you re-state which two devices suffered inter-device comms throughput issues, and confirm that your test was between these two very same devices and in the same direction that was originally having a problem?  Maybe create a table/list of tests, clearing identifying source and target and copy method/protocol and observed average speed, and whether QoS was enabled for VoIP, e.g. maybe six columns of:

DateTime, Source, Target, Protocol, Avg.KB/s, QoS Y/N

.

Also, I wouldn't use Windows File Explorer for my copy testing, because Windows will try to do all sorts of buffering - and so what you see from a user perspective is a saw tooth performance graph visual artifact because Windows will itself read a large chunk of the file and empty its own buffer next.  So, instead try a speed test using FTP or SFTP or SCP.

[WVT]

It wont matter if QOS for VOIP is affecting anything because if you turn it off phone calls will sound like Mcdonalds drive thru speaker.