Poor AVVI Performance since vSphere 5 upgrade

[Simon_B_]

Did anyone else notice a large performance drop of AVVI backups while backing up from vsphere 5?

I am running a basic setup:

• BE2010 R3 with latest available hotfixes
• AVVI Backups without GRT
• Disabled Incr/Diff Backups and CBT to exclude this from the list of possible causes
• Targeted to Deduplication Folder
• no vCenter in place
• (edit) NBD Transport mode is used for all tests - SAN transport mode is not availible in this environment

I've upgraded one server from vsphere 4.1 to 5.0 and the Transfer rates dropped from 3500mb/min to something in between 500 and 800MB/s. 

So now I have two vSphere hosts, one at 4.1 and one at 5.0, both with similar VMs, similar hardware and similar connection to the BE media server. The VMs on 4.1 still backup with over 3000mb/min while the 5.0 VMs only get transfer rates < 1000mb/s.

To verify this result I've installed a fresh vSphere 5.0 and migrated some of these machines over to this one, same behaviour here.

Any ideas?

[lgarcia]

what type of transport and wich order is specifid for transport?

Did ou verify in the job log you are using san transport?

Luis

[Simon_B_]

Thanks for your reply, forgot to mention it. 

All tests were performed with NBD, SAN transport mode is not possible in this environment.

[lgarcia]

was SAN transport enabled in your vsphere 4 environment?

Are your network card on the vSphere 5 environment configured as well as your vsphere 4 old environment (speed/duplex/vswitch/vmnic)??

[Simon_B_]

The settings for both vsphere 4.1 and 5 are completely the same. Including the Transport mode (nbd). The Network configuration is the same, as written above when first observing this behavior it was an inplace upgrade  4.1->5.0 which also keeps all network settings.

[ZeRoC00L]

If you copy data from a Virtual Machine at vSphere 5 to your backup exec server, what is the throughput ?

If you copy data from a Virtual Machine at vSphere 4 to your backup exec server, what is the throughput ?

[Simon_B_]

Both around 50MB/s while copying 1GB via CIFS. Same speed in both directions (be -> VM and VM -> be)

[ZeRoC00L]

You get these results both from vSphere 4 aswell as vSphere5 ?

[Simon_B_]

yes

[ZeRoC00L]

Definitely strange.
I am not running on vSphere 5 yes, so I cannot test it for you.

What you can try is to download a trial of VEAAM Backup, and see if the same issue also occurs:
http://www.veeam.com/vmware-esx-backup.html?ad=menu

This can point if the issue is with vSphere or Backup Exec.

[Simon_B_]

I also thought about that but the problem is that the current veeam version does not yet support vsphere 5. According to the forums a patch is available but only through support.. 

I will give it a try..

[CraigV]

...using Veeam isn't the solution. I've never managed to get BE to run as fast as Veeam does. I'll escalate to support for you, otherwise check out the Known Issues section of this forum to see if it is already logged as an issue...

[Simon_B_]

Thanks CraigV,

just to clarify, we are not talking about switching something to veeam here. The idea is to test the behavior with veeam to see if the performance there also degrades when backing up from vsphere 5.

If this is the case this would point to a change on the vsphere code which degrades the nbd throughput. If there is no througput difference this would point to a problem with the implementation of VDDK 5 on BE side.

I already searchd BE KB and forums and also the Vmware site for known issues/changes regarding this but nothing came up

[Si]

I have just installed Veeam to compare the performance of SAN Transport backup of vSphere 5 with BE.

Veeam is a lot faster. Almost getting 9000MB/min like I should do (10GBE iSCSI network).

Raised a support case about it, no solution thus far....

[Si]

Note you need a patch from support for it to work with vsphere 5

[teiva-boy]

BackupExec will not come close to Veeam's speed!  When you have one company, which all they do is VMware, and another that is a jack of all trades..  Who do you think will be better?