Please comment on my VMware backup performance

[Andrew]

Hey All,
Today was my first test of NBU 7.1 VMware Protection in my lab.  I am looking for any comments on how good or bad my performance is.  Based on my hardware, it looks awful to me.

I am running  in a test network, using the following:

• Mater NBU Server: Server 2008 R2 Blade (HP BL460, 16GB RAM, quad core processor)
• The blade chassis has 6Gbps etherchannel connectivity, but it is bottlenecked to 1Gbps at one link.
• 4Gbps FC SAN (presented to the vSphere blades and the Master NBU Server)
• HP MSL 6000 series library, connected at 4Gbps
• VMware vCenter 4.1u1

I have tested backing up the same VM different ways, with the following results:

16GB VM, Server 2003, Full Backup (Indexed and White Space is ignored)

• SAN Transport Method:  8,424 KB/s, total time 34 Minutes
• LAN Transport Method: 30,000 KB/s, total time 12 Minutes (241 Mbps)

None of the performance indicators in vCenter appear pegged, so I am stuck figuring out why this is so slow.

Thanks,
Andrew

[pikachu]

disable mapped and exclude in the options

now re-run a san backup, same speed?

[Nicolai]

Individual VM backup speed seems in general to be pretty moderate. However if we run multiple VM backup at the same time - the consolidate performance can be pretty impressing.

Try 20-30 host at the same time. I believe see good combined numbers. 

[AAlmroth]

Some thougths...

For the tape scenario a possible reason would be that the data rate to the drive is not high enough which leads to shoe shining effect with re-position overhead. For example, a LTO5 drive would expect a minimum rate at 47MB/s to stream at its lowest rate. Any speed below this would give severe performance impact.

As the LAN speed is seems to improve the overall speed to tape, it could also be that you use same HBA ports for disk access and tape writes. This gives overhead as well. As you use Windows, please check that you use the proper device driver for the tape drives and the correct NetBackup data buffer size.

LAN based backup using the service console on the ESX servers usually gives 20-30MB/s if on 1GbE, so I would rate 30MB/s as expected at the higher end.

You could always run a test to a disk based storage unit on local disk to compare the performance from VMWare and remove the tape target from the equation. Then you can better see where the bottleneck is.

/A

[Andrew]

Thanks all, I will setup more tests today based on your feedback. 

I am going to start a separate thread asking about De-Duplication within the VMware Protection add-on.

Having de-dupe would speed up performance.

Thanks,
Andrew

[Andrew]

I ran some more tests today.

I created a Disk Storage Unit and ran the backup using the Fiber Channel transport method.  My speed improved to about 19,000 KBps (152 Mbps).  Using the same Disk Storage Unit, my network performance stayed about the same (~30,000KBps/240Mbps).

Some other notes:

• I am running the latest drivers from HP for my MSL and my HBA.
• I have NOT tried turning off mapping and white space yet, because in my end solution I will need to restore indiviual files (which I think Mapping is a requirement for).
• It is possible we are using the SAME HBA for disk reads/tape writes.  I had not considered that as an issue so far, because we are using a 4Gbps HBA, and I would think that we have much more bandwidth than is being used (152Mbps).

Right now I am running a much larger test (2 VMs this time, but around 200GB instead of 16GB).

Tomorrow I will run another test with mapping disabled and white space disabled, to see if that yields any performance boost.

Thanks

Andrew

[Mark_Solutions]

Do you have any tuning in place?

On the Media Server under \netbackup\db\config\ (create the config directory if it doesnt exist) create the following - all in upper case and with no file extention (no service restart needed):

For LTO4 or 5 :SIZE_DATA_BUFFERS with a value of 262144 in it

For all : NUMBER_DATA_BUFFERS with a value of 32 in it

SIZE_DATA_BUFFERS_DISK eith a value of 1048576 in it

NUMBER_DATA_BUFFERS_DISK with a value of 32 in it

Also, assuming a Windows Media Server, add the following to the registry:

HKLM\System\CurrentControlSet\Control\SessionManager\Memory Management\

DWORD named PagePoolSize with a hex value of FFFFFFFF (that is 8 F's)

DWORD named PoolUsageMaximum with a decimal value of 40

These keys require a reboot

Then try your backups and see if it imporves things

Hope this helps