cancel
Showing results for 
Search instead for 
Did you mean: 

Backing up a large vmware environment?

Jaykullar
Level 5

Hi All,

I was just looking to see what others are doing with the growth of vm's and the need to back them up. I am using NetBackup 7.1.0.4.

I am currently have 1000 vm's to backup per site (i have two sites), I have a mixture of VMX4 & VMX7. I have two high spec'd MSDP's per site, the MSDP are tier 5 storage on an 8GB SAN. The ESX Luns are zoned to the MSDP servers also, so using SAN as transport type. The MSDP servers are windows.

I currently backup 2 clusters per day one per media server to spread the backup load over the week, this works fine even though my performance using the camel test only shows 130 - 140 mb/sec. (The lower end of the scale)

However this means I can be upto 6 days out of date when requiring a restore due to backing up a cluster once a week.

I have tried to split the VMX4 VMs are 500, and 500 VMX7s so I can do BLIB backups on the VMX7s, but trying to backup 500 VMs (a day) even on a BLIB and the 500 VMX4s one cluster a week with only 2 media servers is proving a difficult task.

I have been looking at adding more MSDP servers, but as the storage isnt the fastest I am also looking at appliances.

I know im not the only one out there with the challenge of backing up 1000's of VMs, I was wondering what you have done, and the best options to either add more media servers, or look at appliances.

 

Any help is appreciated.

 

Thanks

Jay

11 REPLIES 11

BTLOMS
Level 5

we too have that many Vms. But we also have a large MSDP drive and a super powerful server to perform dedupes :). Daily backups complete in about 12 to 14 hours. Fulls take about 24 hours. We have not split it like you have.

 

But i am curious what solutions others may have.

Jaykullar
Level 5

We to have good powerfull servers which never show much load on them during backups. How do you manage 1000 VM full backup in 24 hours? Do you have a SQLIO or Symantec camel test result on the speed of your SAN / MSDP?

Do you only use 1 media server do your VM backups?

 

Thanks

RLeon
Moderator
Moderator
   VIP   

With SAN Transport mode, all vm snapshots are sent in full to the media servers for media-server-deduplication.

See if HotAdd Transport mode plus client-side-dedup could fit into the picture.

Even though this method is not LAN free, but in some cases backups complete alot quicker than using SAN Transport mode - if data change rate is low and the dedup ratio is good.

You do not need to use a media server for the HotAdd mode snapshot mount host VM, a Nbu standard client would do. Then you just set this client up as the "Backup Host", and enable client-side-dedup on it.
Your VM backups will be deduplicated on the HotAdd VM before getting sent to the media servers.

The processor resources of the ESX hosting the HotAdd VM will be used during the deduplication process.
Also note that unlike the SAN Transport mode, HotAdd does not support vmdks over the size of 1TB.

Jaykullar
Level 5

That does make sense, however I do not want to put a lot of load on the ESX environment using client side dedup and wanted backups to be SAN based only.

I will look at testing this to see if it helps, I take it i need to do the followig:

  • Instal NBU Client on a VM which is in the Datacenter of where I want to backup

  • Add this NBU client into VM Backup Hosts on the master
  • Configure a policy to use this VM as a backup host, and snapshot option to use HotAdd

Is there any other configuration required to get HotAdd & Client side dedup working? Also I take it i can still backup to MSDP?

What kind of throughput is everyone able to run through there SAN? One site Symantec Camel test showed 60mb/sec and the other 130mb/sec.

I only run a maximum of 12 concurrent jobs due to bandwidth.

 

Thanks in advance.

RLeon
Moderator
Moderator
   VIP   

That's it really. I typically deploy Windows 2008R2 for the the HotAdd VM, seems to give the least trouble. SUSE is also supported as a backup host now (since 7.5.0.3) so you might also want to consider it.

In some smaller deployments we make the Master Server itself the HotAdd VM, and have physical MSDP Media servers. This is so we don't need 2 VMs (a nbu master and a HotAdd VM).
Also, since the VM Master is not used as a Media server, the VMware based HA features could be leveraged to give it some protection.

Make sure the ESX on which the HotAdd VM runs has visibility to all datastores within which you have VMs that you want to backup, so that snapshots from them could also be mounted by the HotAdd VM.

In the Nbu VM guide there is a section titled Notes on the hotadd transport mode with some useful information, such as the 1TB vmdk limit.

And here is a link to some other useful infomation:
http://www.symantec.com/business/support/index?page=content&id=TECH183072

If client-side-deduplication is not utilized, then the HotAdd mode is not really useful; at least to NetBackup.

 

Also I take it i can still backup to MSDP?

Yes, just like any other client-side-dedup clients.

Jaykullar
Level 5

Thanks for the info. I will test this method to see if it helps get these VMs backed up.

What storage are you backing up to and what speed do you typically see when concurrent jobs are running? I feel that my storage (tier 5 archive) maybe the a key issue here as even when running 475 BLIBs its still a struggle.

Thanks

RLeon
Moderator
Moderator
   VIP   

I do not currently have access to any of my sources to pull figures out from (.... that's why I've been seen roaming/sniffing around the forums for the past couple of days...). Perhaps someone here will share some of theirs?

If you lay the whole thing on a piece of paper, you can see that due to source deduplication, the LAN throughput's significance to backup performance is reduced. The higher the dedup ratio the more so, since less data actually traverse your environment.

Because the deduplicated data still represents the full data (Referring to the "full" stuff that gets dedup'd, including BLIB data.), it is possible that reported transfer speeds could be way over what you are seeing now in SAN transport lan-free backups, if the dedup ratio is good.
The figures could get even (marginally) better if your MSDP media servers have high speed connections to the ESX hosts, such as 10Gb LAN or many LACP nic aggregations, both sides.

Since the network can become less likely to be a bottleneck in this config, that leaves you with the HotAdd ESX host's processing horsepower, the VM datastores' storage performance, and the MSDP's dedup pool storage performance. The processing requirements on the MSDP media server is somewhat mitigated due to the dedup being done elsewhere.
You may find it useful to use tools/vendor utilities to test the max speed for each of the above, so you would have an idea of where a bottleneck could be and what kind of speed you are realistically aiming for. This applies to SAN transport too. E.g., if your MDSP dedup pool write speed is very slow then your backups will be slow no matter how fast you can send data to it. (Unless you always get 100% dedup ratio from client-side-dedups)

We do not have VMware environments your size. And since our ESXes cannot handle that many snapshots at the same time (or not allowed to...), I never had to set the MSDP concurrent jobs to anything higher than 20. This is storage performance dependent so you may have to test around with yours.

For very big environments, you may even want to consider the extremes and assign a HotAdd VM backup host to each ESX host to share the load.

Jaykullar
Level 5

Thanks - At the moment the request was to be SAN only, so I presented all the VMFS LUNs to the MSDP servers.

We do have alot of vCenter's with various datacenters, so I would need a HotAdd VM per datacenter to get this configuration to work, I could add multiple HotAdd VMs per ESX server to spread the load yes.

If I was to do a weekly full I have 50TB to backup in one site, Id need 500mb/sec write speed to get this through, the horsepower of HotAdd would certainly help, but I need to test this.

Also I have noticed the Full backups are deduping at 95% but the BLIBs are low as 70%,

Im thinking the idea solution would be a single large MSDP, a high spec media server (which we have now), 500mb/sec writeable storage on our current 8GB SAN with individual ports for MSDP, along with HotAdd VMs. I think i might be dreaming :)

RLeon
Moderator
Moderator
   VIP   

Deduplicious.

Antonio_Mart_n
Level 3

Maybe you can uptate Netbackup version to increase backup performace.

Jaykullar
Level 5

Antonio, I am looking at upgrading to 7.5.0.5 soon, but also want to P2V the master.