So I have a bit of a conundrum that I'm searching for advice on.
We have a disk volume on a Windows 2012 R2 Hyper-V Enabled Host that only hosts the files required for running the Virtual Machines (VDI's/VHDs/VM Configs etc). The actual Hyper-V installation is on a different drive (Host drive). The drive is genuinely only for storage of the VM's and related files. The BackupExec jobs backup the Hyper-V VM's, with GRT enabled for certain VM's, the backups are "full" backups as far as Hyper-V is concerned (afaik) and the actual files on disk are also backed up in case we need to restore invidual disks etc (aware this might be considered overkill).
We have been backing this up via backup exec to tape for a long time now, job rates of 7000 - 9000 MB/min generally.
While a majority of these VM's do not change much over time, one does hold an SQL DB but doesn't hold to many files (more just symoblic links) and another is a low use Sharepoint server.
To save space on the server it was decided to do Windows 2012 R2 deduplication, this was turned on and allowed to do background optimisation only, initially great as up to 71% space savings have been seen with a negligible performance impact.
However our Backup Exec jobs to tape have slowed to a crawl over the last two months, the job rate has come down to 800 - 1300MB/min, the job is now still running through to the next day almost overlapping itself and this does cause an obvious and noticeable performance hit in the VM it is backing up (not entirely unexpected). I have been through countless articles and run myself in circles about what I can actually do to solve this issue so I have come seeking advice, even if its as simple as turn deduplication off (as that is the easy option and we are not out of space even with it off).
So questions for my own clarity:
1) Why has backup exec slowed to a crawl?
-I suspect its having to blow up the VM's again from dedup, take a checkpoint or some such and then shrink again? - Apologies on this I'm more from a VMWare background in terms of virtualisation and its the first time i've used BackupExec in this capacity.
2) What is the recommended way, given we are using Backup Exec 15 FP5, to backup Hyper-V VM's so that we have the easiest/fastest recovery from tape storage, pretending that we had a total failure of drive/server?
3) The tapes are Encrypted and Compressed in Backup Exec using the tickable hardware options - is this going to cause an issue with dedup data?
- I notice that the data on tape is larger than the old jobs, some of this is natural growth, but i assume this is copying the dedup contents under System Volume Info as well... please correct me if i'm wrong.
4) Does Backup Exec 16 deal with this any differently?
5) The disk itself says it is 83% fragmented.. nothing is running too slowly though and Windows Disk managment says it was optimised a few days ago. Also some of the disks are thin provisioned so again with limited Hyper-V experience I do not want to risk blowing up some thin provisioned disks inadvertently by running SDelete or some such and exacerbating an existing problem. Is this going to be a serious issue for Backup speed - other than disk heads thrashing about?
Thanks for all the help in advance!
You should not enable encryption against storing within a Backup Exec Deduplication folder (but hardware encryption against tape should not affect performance too much)
and Backup Exec does have to re-hydrate (return to non-dedplicated state) data that has been deduplicated using the technology built into Windows which will have a performance and storage impact (if you write to Backup Exec's own deduplication storage and duplicate to tape we also have to re-hydrate)
Is it the rehydration process that is killing the backup times then?
What is the recommended way to backup this drive and its contents for restore, just flat files or Hyper-V aware backup?
Just looking for an up-to-date set of guidelines for backing these machines up for easy restore via backup exec, and if wanting to save space, if deduplication or any other method of space saving should/could be used where should it be applied?
Most articles referenced on the site have up to date dates, but the internal content of the articles on the subject clearly hasn't been updated.
Thanks for confirmation on re-hydation and the clarification on encryption use.
If we initially avoid the fact that you are using Windows Deduplication , then except for some configurations that may not support GRT capability) it is always recommended to backup VM's using the correct virtual agent (with GRT enabled) as then you can choose to either DR the whole VM or get back a single file/e-mail etc from inside the VM.
Where GRT is not possible (if configuration not supported) you should use a remote agent backup of the VM content itself
It is not recommended to just backup the file system of the volume that contains the VMs without using a virtual agent setup and you certainly don't need to do both as the virtual agent gives both DR for the VM and GRT in a single pass.
You can backup Hyper-V VMs into a Backup Exec Deduplication Storage location (which saves space against storing the backups themselves), although note that because Our Deduplication process is not the same as Windows Deduplication, this would re-hydrate and then deduplicate (so still a performance hit against what you have done). Also note that tape backups are always stored as rehydrated.
With regards the fact that you have now enabled deduplication on this volume as far as I have been able to ascertain (checked with a senior colleague) we do no official testing of such a setup where Windows Deduplication is enabled on the Hyper-V hosts' storage for VMs and this therefore means no official support. So GRT may or may not work , and at very least we would expect performance issues relating to having to rehydrate your data for the backup/catalog processes however because of no testing we could not confirm all the symptoms you might experience.
There is potentially another big issue - if you did have to restore the whole VM (that was backed up from the Windows Deduplication enabled volume) during the restore we would probably need the complete disk space (and not the deduplicated disk space) which could mean you don't have enough space to restore a VM . ( this is as per http://www.veritas.com/docs/100009880 )
Based on all this I would have to say that you should not enable Windows Deduplication on the volumes that contain your Hyper-V VMs.
As a further comment it looks like MS don't suggest you use Windows Deduplication against VMs either (there are some exceptions for VDI etc)
and if MS say don't do it (either because it is bad practice or because not supported) we definitely won't test or support such a setup.