Forum Discussion

rcdauria's avatar
rcdauria
Level 4
9 years ago

Bad performance - NBU Appliance > VMware NBD

Hi guys,

We are seeing some performance inssues in a new Netbacku Appliance environment, as follows:

Master Server: Clustered Windows VMs, running NBU 7.7.1
Media Server: Netbackup 5230 Appliance running version 2.7.1
Backup Source: VMware Virtual machine hosted on a ESXi 6.0 (no vCenter right now)
Backup destination: Netbackup Appliance MSDP (100TB, empty right now)

Backup network being used right now is a 10GB/s fibre connection with Jumbo Frame enabled. We are planing to set up a bond with 4 interfaces, but just to isolate the issue we haven't set it up yet.

The ESX server also uses 10GB/s fibre connections to the storage. However, please note we have two VM Kernel interfaces, one for the backup network and another one for the storage communitation. Therefore we have a exclusive backup network.

With no concurrencies, backup can't go faster than 70.000 KB/s.

We can assure the problem is not on the datastore we are reading, as we tested it and we can see a very nice performance there (200.000 KB/s).

To test the network path, we have run iperf tests from the netbackup appliance to the ESX host, and saw a 450.000 KB/s rate (around 3.5Gbits/s)

I was not sure why the iperf test was showing a 3.5Gbits/s (and not around 10Gibs/s), so just to be sure I didn't had a switch issue, I setup a adhoc connection betweet my two Netbackup 5230 Appliances (that is, a fibre directly connecting both appliances) and, curiosly and I dont know why, iperf didn't went faster than 4.5Gbits/s.

Anyway, summing it all up: I am not sure why, but the 10Gbps interfaces of the appliance can't go into full utilization, they actually never go above 50% (5Gbits/s).

But also the backup rate is far from that, so I am not sure if we have a Netbackup problem or a Network problem.

Any guesses? AFAIK, there is no much tunning to do when backing up VMs (NDB backups).

Thank you in advance,

Rafael

7.7
Backup and Recovery
Linux
NetBackup
  • Michael_G_Ander's avatar
    Michael_G_Ander
    9 years ago

    Have rarely seen more than about 5 Gbit on 10 Gbit connections, think it is because 10 Gbit actually requires pretty many resources

    Another thing that be could an issue is that the vmware kernel had a limitation of around 220 MB/s for vadp in the earlier versions like 5.0

    Have you changed the raw buffer size to the recommend 1024 ? This I have seen improve the vmware backup speed greatly

  • Michael_G_Ander's avatar
    Michael_G_Ander
    Level 6
    9 years ago

    Have rarely seen more than about 5 Gbit on 10 Gbit connections, think it is because 10 Gbit actually requires pretty many resources

    Another thing that be could an issue is that the vmware kernel had a limitation of around 220 MB/s for vadp in the earlier versions like 5.0

    Have you changed the raw buffer size to the recommend 1024 ? This I have seen improve the vmware backup speed greatly

  • rcdauria's avatar
    rcdauria
    Level 4
    9 years ago

    Hi Michael, thanks for sharing your tougths.

    Raw Buffer Size didn't help.

    I still feel its strange we only reach 80 MB/s in a unique VM, but I noticed we can run like 3 or 4 VMs simultaneously, each of them running at 60-70MB/s.

    So that will have to do :)

    Thanks a lot,

    Rafael

  • sdo's avatar
    sdo
    Moderator
    9 years ago

    This NBU TN:

    VMware Transport Modes: Best practices and troubleshooting​

    http://www.veritas.com/docs/000094725

    ...says there are limits on the number of concurrent NBD backups, because NBD uses VMware NFC protocol, which has different limits depending upon the version of vSphere ESXi, listed here:

    https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc_50%2FvddkDataStruct.5.5.html

    .

    We know that NBD uses VMkernel ports:

    https://communities.vmware.com/thread/391810?start=0&tstart=0

    .

    And, I have heard (and I've been searching for documented evidence, but not found any yet) that each VMkernel port usage is limited to 1Gb/s (i.e. around 112MB/s), but, this next link clearly shows that NBD can deliver more total aggregated bandwith per VMkernel port - up to a point, but each stream gets slower because VMware deliberately caps any single usage of a VMkernel port to 1Gb/s, and - here's the rub - there is nothing that you can do about that particular "specific point":

    http://www.mrvray.com/2015/06/getting-to-know-the-network-block-device-transport-in-vmware-vstroage-apis-for-data-protection/

    .

    Any single VMkernel port is hard capped for single usage at 1Gb/s - i.e. c. 112MB/s.

    Any single VMkernel port struggles with more than four demanding concurrent work loads - i.e. c. 220 MB/s.

    .

    So, next actions?

    .

    Perhaps test your actual VMkernel port maximum speeds ?

    https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052119

    .

    And, perhaps use multi-NIC VMkernel ports ?

    http://www.heathreynolds.com/2014/02/multi-nic-vmotion-on-esxi-55.html

  • sdo's avatar
    sdo
    Moderator
    9 years ago

    And - I found so many articles saying the typical actual real world usage maximum out of any 10Gb NIC is about 600MB/s - so a little over 50% of what one might consider to be the wire electrical/light/signal speed maximum of a 10Gb link.   And your observations with your iperf tests correlate with that.