Bad performance - NBU Appliance > VMware NBD

[rcdauria]

Hi guys,

We are seeing some performance inssues in a new Netbacku Appliance environment, as follows:

Master Server: Clustered Windows VMs, running NBU 7.7.1
Media Server: Netbackup 5230 Appliance running version 2.7.1
Backup Source: VMware Virtual machine hosted on a ESXi 6.0 (no vCenter right now)
Backup destination: Netbackup Appliance MSDP (100TB, empty right now)

Backup network being used right now is a 10GB/s fibre connection with Jumbo Frame enabled. We are planing to set up a bond with 4 interfaces, but just to isolate the issue we haven't set it up yet.

The ESX server also uses 10GB/s fibre connections to the storage. However, please note we have two VM Kernel interfaces, one for the backup network and another one for the storage communitation. Therefore we have a exclusive backup network.

With no concurrencies, backup can't go faster than 70.000 KB/s.

We can assure the problem is not on the datastore we are reading, as we tested it and we can see a very nice performance there (200.000 KB/s).

To test the network path, we have run iperf tests from the netbackup appliance to the ESX host, and saw a 450.000 KB/s rate (around 3.5Gbits/s)

I was not sure why the iperf test was showing a 3.5Gbits/s (and not around 10Gibs/s), so just to be sure I didn't had a switch issue, I setup a adhoc connection betweet my two Netbackup 5230 Appliances (that is, a fibre directly connecting both appliances) and, curiosly and I dont know why, iperf didn't went faster than 4.5Gbits/s.

Anyway, summing it all up: I am not sure why, but the 10Gbps interfaces of the appliance can't go into full utilization, they actually never go above 50% (5Gbits/s).

But also the backup rate is far from that, so I am not sure if we have a Netbackup problem or a Network problem.

Any guesses? AFAIK, there is no much tunning to do when backing up VMs (NDB backups).

Thank you in advance,

Rafael

[Michael_G_Ander · Accepted solution]

Have rarely seen more than about 5 Gbit on 10 Gbit connections, think it is because 10 Gbit actually requires pretty many resources

Another thing that be could an issue is that the vmware kernel had a limitation of around 220 MB/s for vadp in the earlier versions like 5.0

Have you changed the raw buffer size to the recommend 1024 ? This I have seen improve the vmware backup speed greatly

[rcdauria]

Hi Michael, thanks for sharing your tougths.

Raw Buffer Size didn't help.

I still feel its strange we only reach 80 MB/s in a unique VM, but I noticed we can run like 3 or 4 VMs simultaneously, each of them running at 60-70MB/s.

So that will have to do :)

Thanks a lot,

Rafael

[sdo]

This NBU TN:

VMware Transport Modes: Best practices and troubleshooting​

http://www.veritas.com/docs/000094725

...says there are limits on the number of concurrent NBD backups, because NBD uses VMware NFC protocol, which has different limits depending upon the version of vSphere ESXi, listed here:

https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc_50%2FvddkDataStruct.5.5.html

.

We know that NBD uses VMkernel ports:

https://communities.vmware.com/thread/391810?start=0&tstart=0

.

And, I have heard (and I've been searching for documented evidence, but not found any yet) that each VMkernel port usage is limited to 1Gb/s (i.e. around 112MB/s), but, this next link clearly shows that NBD can deliver more total aggregated bandwith per VMkernel port - up to a point, but each stream gets slower because VMware deliberately caps any single usage of a VMkernel port to 1Gb/s, and - here's the rub - there is nothing that you can do about that particular "specific point":

http://www.mrvray.com/2015/06/getting-to-know-the-network-block-device-transport-in-vmware-vstroage-apis-for-data-protection/

.

Any single VMkernel port is hard capped for single usage at 1Gb/s - i.e. c. 112MB/s.

Any single VMkernel port struggles with more than four demanding concurrent work loads - i.e. c. 220 MB/s.

.

So, next actions?

.

Perhaps test your actual VMkernel port maximum speeds ?

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052119

.

And, perhaps use multi-NIC VMkernel ports ?

http://www.heathreynolds.com/2014/02/multi-nic-vmotion-on-esxi-55.html

[sdo]

And - I found so many articles saying the typical actual real world usage maximum out of any 10Gb NIC is about 600MB/s - so a little over 50% of what one might consider to be the wire electrical/light/signal speed maximum of a 10Gb link.   And your observations with your iperf tests correlate with that.