Very Slow VM Backups

[Kev_Lamb]

Hi,

We are currently using Network based VM backups rather than SAN based, the Backup Host is a Windows 2008 server running NBU7.5.0.5 and has full access to the ESX servers, I have several policies set up including VIP's for covering the VM's.

The back bone is a 1Gb LAN, the Backup Host is SAN attached to an HP VLS12000 which is emulating LTO4 drives, this is on 4Gb F/C

When backing up the VM's I am only seeing on average 1.5M/s throughput, we have standard clients that are using the conventional NBU method whuch reach over 40M/s through the same medium.

Just wondered if there were any tuning or other paramaters that we need to look at with the VM backups to speed them up.

Kev 

[Dyneshia]

Please review :

http://www.symantec.com/docs/TECH191183  - After upgrading to NetBackup 7.5.0.x, VMware Backups are slower than expected

In addition, in your backup selection, are you using the display name instead of the host name ?

Do you have any non-VM backups using the same network that complete in a timely fashion ?

[Kev_Lamb]

Hi,

Apologies for not getting back sooner, I have been doing soem investigations withour network set-up on the Media Server and also the buffer sizes, I incresed the amount of Buffers to 256 from 16 which gave a better performance across the board but this was still slow on the VM's and other clients using my media server.

I moved several backups through the Linux Master server and saw a dramatic rise in the throughput, the Windows Media server still produces upto 50% less throughput though.

We believe that this is a problem with the actual server, this is an HP Blade which uses it's own internal switch through the blade chassis and then through to the main LAN, a call to HP is being opened about this and we are also looking at putting in a Mezanine Card and having a direct connection on the LAN to see if this improves the performance.

Will keep you updated

Kev

[Jaykullar]

Kev, do you have resource limits setup? VMware advise not to backup more than 4 machines for a single datastore.

[Kev_Lamb]

Hi Jaykullar,

Not sure what the rescource limits are to be honest with VM, I have passed this over to our VM Team, as for the amount of machines to datastores I am backing up well in excess of 4, in somke policies we are backing up over 20 machines per data store, I am assuming machines = clients in this scenario.

Kev

[huanglao2002]

If you want to use VIP,you must consider resource limit. If you use 1Gb network backup 20 virtual machines the average backup speed almost equal 5MB/s.

Can you test backup 1 virtual machine to check the backup speed?

[Ray_Esperanzate]

As stated previously, you probably want to set up resource limits.  The other thing you will need to consider is the activity on your vmkernel ports.  VMWare will reduce the backup speeds in an effort to make sure that there is enough bandwidth for other activities using the vmkernel port (vmotion, etc).

I also suggest reading this excellent presentation given by George Winter during vision 2013:

http://www.symantec.com/connect/sites/default/files/IA%20B29%20-%20Best%20Prac%20Backup%20%20Recov%20VMware_EM.pdf

[Kev_Lamb]

Hi Huanglao2002, I have just set-off a backup on it's own with nothing else running, I am getting almost 13MB/s, I am using buffers and these are set as:

NUMBER_DATA_BUFFERS=256

SIZE_DATA_BUFFERS=262144

Ray,
Thanks for the document, I will certainly look at the resource limits, just giot figure out the correct amounts to use.

Kev

[huanglao2002]

 I have just set-off a backup on it's own with nothing else running, I am getting almost 13MB/s.

As 1Gb/s networkwork,it's almost reach 80MB/s usually.

VMWARE ------->> backuphost ------------>>> media server

                    |                                   |

              bottleneck                      bottleneck

you can try to set a standard type test backup policy,backup data from backuphost to media server,if it's speed reach 80MB/s. I think "vmware->backuphost" is bottleneck. alse you should check "backuphost->media server" performance.

[Kev_Lamb]

Hi,

My Backup Host is also the RHEL Master / Media Server, I was using a Wintel server as the Media Server which was also the Backup Host but we have seen as issue with this server, it is an HP Blade and uses it's own internal switch which is then plumbed into the server switch, anything running through this seems slow, we are looking at changing the network on this so it is direct through to the main switch.

Moving the VIP backups to use the RHEL Master / Media server as the Backup Host has increased the speed compared to before using the Wintel Blade Server.

Is it best to have a seperate Backup Host from the Master/Media Server?

Obvioulsy based on our set-up running a backup from the Backup Host to the Media Server would not be an issue as it is the same server.

Kev

[huanglao2002]

If you are sure "vmware->backuphost" is bottleneck,Please involve vmware team for help.

[Jaykullar]

Hi Kev,

Resource limits are in >host properties>master server>properties>resource limits. Ensure your datastore is no more than 4, also look at limiting ESX resource limit.

http://www.symantec.com/business/support/index?page=content&id=HOWTO44509

What is the spec of your server? I would advise against having media/master which is acting as a backup host also.

How many concurrent streams do you have setup on your storage unit?

[Kev_Lamb]

Hi

I currently do not have Multiplex or Streaming set on the Storage Unit as we are backing up to a VLS which does not allow Multiplex, the Master Server is an HP DL580 G7 with 4x1867MHz Q/Core processors and 128Gb RAM , this is running RJEL5.7

I have looked at the resource tab and these are all set to No Limit sio I might try setting these up.

Thanks for the heads up regarding the the backup host, is this best to have a physical rather than virtual, I may have to hunt around for a client to push this through.

Kev

[Kev_Lamb]

Just read this:

For a more efficient backup, the NetBackup media server and the VMware backup host should be installed on the same host.

http://www.symantec.com/business/support/index?page=content&id=HOWTO44423

[Jaykullar]

I understand you do not have multipexing or multistreaming, but how many concurrent jobs for your storage unit setting? "Maximum concurrent jobs". As you have no resource limits only this setting or global attributes of maximum client jobs would control the amount of jobs running.

Yes the media server and backup host are ideally on the same host, however its best practice to keep this seperate from your master server, i.e not having master/media/backup host on the same host.

[Kev_Lamb]

Ah, the VTL is set as a robot and as such does not have Max Concurent Jobs only Max Write Drives which is set to 22, the global Max Jobs Per Client is set to 8

I see hwat you mean about the Master Server, unfortunatley the Media Server that we were using is the Blade which has undelying network issues, I may need to look at using another server for this function, I know it can now be either Wintel or Linux but can this be a VM or is that not really reccomended, I may need to source another physical server somewhere.

Kev

[huanglao2002 · Accepted solution]

Please reference the attachment.

the attachment copyright belong symc.

[Michael_G_Ander]

I have seen that tuning the flash backup read buffer on the backup increase the VMWARE backup speed

It is described in technote http://www.symantec.com/docs/HOWTO56178

Regards

Michael

[Kev_Lamb]

Hi Michael

Thanks for the link, I will have a look at these parameters and see if this gives me any further improvement.

Kev

[huanglao2002]

Hi Kevin

any update?