Slow VM backups using NBD

[Eric_Engberg]

I'm in the process of setting up VMware backups but the transfer rates using nbd are very very slow.  The media server and ESX servers each use single 1GB NIC's and even with multiple backups going across multiple ESX server's the NIC on the media server is frequently below 1% utilized.  Occasionally the utilization in Windows Task Manager for the network interface will look like a square wave bouncing between 0% and 50% utilization.  There are points where it can get close to 100% utilization with 8 going at once but even with 8 active backups it is frequently below 30-50%.

I don't think it is a problem with the network itself as normal client backups do not have a problem maxing out the NIC.  I tried the suggestion in TECH191183 to disable disk caching in vxDiskLib.ini but that did not seem to have any affect.  The ESX servers are v5.5.  There are multiple VMKernel ports on the ESX servers but backups are the only one using this NIC.  I've scheduled to update Netbackup 7.6.0.2 early next week to see if that helps speed things up.  I've included OS and Netbackup version info below.  If more info is needed please let me know.

• Master Server
  • RedHat 6.3
  • Netbackup 7.5.0.3
• Media Server/VMware Backup Host
  • Windows 2008 R2
  • Netbackup 7.5.0.3

[INT_RND]

NIC usage as reported by the OS is worse than unreliable. What is your actual throughput during the copy operations?

Remember that the theoretical maximum of 1GB ether is around 125MB/s. To reach this number you first have to have I/O that can saturate this on both sides. Then you need to tune every part of the network with the same settings. (For example you must enable jumbo frame on the source NIC, each switch/router along the way, and the destination NIC)

Here is some information about performance tuning Netbackup:

http://www.symantec.com/connect/sites/default/files/NBU7%206-BestPractices-OptimizingPerformance-Lat...

[RiaanBadenhorst]

How many vmguest are you backing up simultaneously? Have you done one at a time to verify you're not overloading the ESX/Datastores with IO requests?

[watsons · Accepted solution]

v5.5 requires an upgrade of Netbackup to support, check out: http://www.symantec.com/docs/TECH127089

7.6.0.1 with EEB3318872  -or-
7.6.0.2

[Eric_Engberg]

I'm sorry I meant to reply much earlier than now.  I was mistaken and we were running ESX 5.1 and not 5.5 but 5.1 was still outside of compatability for 7.5.0.3.  I upgraded to 7.6.0.2 and speeds were much better.