04-11-2014 05:36 AM
Hi!
I am experimenting with Symantec System Recovery 2013 and host-level Hyper-V VM backups in Windows Server 2012 R2.
My observations are as follows:
1. SSR is capable of doing hot Hyper-V backups. In this case Hyper-V creates intermediate snapshots (checkpoints / avhdx) during the backup that are auto-merged into the VM being backed up after the backup. Unfortunately, the snapshots stay in the iv2i backup files and I am not sure how to handle them after the restore of the VM from the backup.
2. I tried to run a pre-backup script in SSR that shuts down the VM in order to perform a cold backup of the VM. After the backup I run a post-backup script in SSR to restart the VM. This works ok so far but Hyper-V still creates the snaptshots (checkpoints / avhdx) I described above although the VM is shutdown at the time of backup. Is this behavior expected?
3. I tried to turn off the "backup volume snapshot" integration service for the particular VM through the Hyper-V Manager. According to the docs this should force Hyper-V to move the VM into saved state and perform a cold backup instead of a hot backup. Unfortunately, this does not change anything. The VM continues to run (instead of stopping) and the execution flow is the same as described in 1. Does SSR somehow ignore the Hyper-V settings?
Thanks in advance.
Anguel
04-14-2014 02:04 AM
Hm, good questions - let me escalate this to support for you.
04-14-2014 03:25 AM
Thanks Markus,
I also noticed that after the restore the VM has the temporary AVHDX hard disks attached instead of the original VHDX disks, i.e. if my VM has a virtual harddisk that was test.vhdx after restore it is test_SOME_LONG_ID.avhdx (although the original test.vhdx is still there. I expect that this is probably due to the fact that during backup Hyper-V redirects its writes to that temporary avhdx and after backup SSR does not "fix" this.
BTW: I am also running SSR inside the VM to get granular file-level backups. But that should not be important. Or does SSR make use of it's own VSS manager instead of Microsoft's VSS manager?
Anguel
04-14-2014 03:47 AM
At least under W2K3 it has its own VSS provider. But vssadmin list writers should show in the VM.
04-14-2014 07:38 AM
Anguel,
1. SSR is capable of doing hot Hyper-V backups. In this case Hyper-V creates intermediate snapshots (checkpoints / avhdx) during the backup that are auto-merged into the VM being backed up after the backup. Unfortunately, the snapshots stay in the iv2i backup files and I am not sure how to handle them after the restore of the VM from the backup.
I'm not sure I understand the problem here. What exactly is the problem you mention regarding the incremental (iv2i) backups?
04-15-2014 01:17 AM
My question is actually the following:
When I perform a host-level backup of Hyper-V (it does not matter if my VM is shutdown or running at the time of backup) in the backup archive I always get the original virtual disk, e.g. MYDISK.vhdx and an additional MYDISK_SOME_LONG_ID.avhdx temporary snapshot.
Probably this is all related to:
http://www.aidanfinn.com/?p=15759
So on restore the VM is actually pointing to that *.avhdx snapshot instead of the original *.vhdx. So how should I proceed? Probably I should I just discard the *.avhdx because it may contain inconsistent data? This would mean that I would have to manually make the VM to point back to the *.vhdx files instead of the *.avhdx files. I don't see any explanations in the SSR docs so customers have to guess...
Anguel
04-15-2014 05:49 AM
Let me check on this and get back to you.
I'm going to be out of office for a few days so may not be able to get back to you until sometime next week.
04-15-2014 07:05 AM
Thank you in advance Chris!
BTW: I have now ordered 2 server licenses and 15 workstation licenses.
Best regards,
Anguel
05-12-2014 05:59 AM
Any news on this topic?
05-20-2014 08:36 AM
Sorry for the delay on this.
I have received a response on this and need to verify one of two things. I should be able to provide an update within a day or two.
05-21-2014 04:23 AM
Thanks Chris,
sounds promising :)
06-04-2014 10:57 PM
Anything still open here? If no please mark this post as solved!
06-10-2014 05:16 AM
Still open. I am still waiting for an answer from Chris (see above).
06-25-2014 01:06 AM
Still no comment on this from Symantec support? It is important to know how to deal with those hot backups performed on Hyper-V in Server 2012 R2. This is a supported configuration, so someone should know how this actually works. We need this information in order to be able to restore from these backups properly. I don't like to "guess" when I pay for a professional backup solution...
06-25-2014 01:22 AM
Anguel,
My apologies - this somehow slipped off my radar.
I do now have this confirmed with our engineering team and I hope the following information helps to answer your questions.
Here’s a summary of our Hyper-V support:
Backup of boot / system volume of Hyper-v host from DR perspective:
a) In order to backup boot / system volumes of host, install SSR on the host and backup boot/system volumes like regular (non-hyper-v) system. Restore of boot/system volume using SRD will restore these volumes. However, the VMs on the Hyper-V having VHDs on data volumes may be out-of-sync post restore.
Backup of Hyper-V VMs:
a) Backup VMs from within the guest and restore VMs using SRD (from within guest)
b) It is also possible to backup multiple VMs by installing SSR on Hyper-V host itself and then take volume level backup of the host volumes. (Note: SSR doesn’t support incremental backup of CSV volumes though). When applying such method to protect VM’s VHD(x) files, SSR recommends that the customer back up the related drives in the same backup job. i.e. drives containing all the VHDs of a VM to be protected and its boot volume (typically C:\). In case if customer has VM’s VHD(x) files located across multiple volumes and he is not really sure which related drives to be backed up together, SSR recommends that he backup all volumes of the Hyper-V host machine.
Restore of VMs from this host level backup needs to be done using granular restore of VHDs for VM. This can be done by mounting v2i files and copying VHD and AVHD files of VM from these V2is. After copying VHD and AVHD files the end user needs to point his VM to the latest avhdx.
06-25-2014 03:49 AM
Chris,
Thank you for the information. Well, I do backup the complete hyper-v host including any volumes containing the VMs and their VHDX (all in one job), so b) should apply to my case.
However, I still wonder that your engineer tells me to finally point my VM to the latest AVHDX, as this is some temporary file that is used while the main VHDX is locked during backup. See here http://www.aidanfinn.com/?p=15759
"Writes are temporarily redirected to the checkpoint’s AVHDX/AHVD file(s). This gives the backup requestor a clean & crash consistent copy of the virtual machine’s VHD or VHDX files that are safe to read. After the backup, the checkpoint is merged and the job is done."
Update: I also noticed that the AVHDX file inside the SSR backup file (after the backup is complete) is smaller than the AVHDX file that is on my volume towards the end of the backup. This means that the AVHDX contains some partial / temporary data during the backup that is IMHO inconsistent. Could you please check this with your engineer? Thanks.
Anguel
06-28-2014 11:22 AM
Anguel,
I have not used SSR 2013 to protect Hyper-V VM's as I use Microsoft DPM for that. I do use ssr to protect all of my servers though. You could manually merge the AVHDX file with the VHDX file within the Hyper-V Manager after the restore. I have had the need to do this [merge files] a few times over the years. Check out this link to see how. http://social.technet.microsoft.com/wiki/contents/articles/6257.manually-merge-avhd-to-vhd-in-hyper-v.aspx
Rob
06-30-2014 03:56 AM
From what I have been told ....
The AVHDX file inside the SSR backup is small because it contains only a few writes that Hyper-V writer made. While the AVHDX file on the main volume contains writes for the all I/O happening in live VM after check-pointing and hence will be much bigger in size.
06-30-2014 05:03 AM
Chris,
Thank you for the feedback. If I understand what you are describing the SSR snapshot contains some "point in time" of a file, and if this file grows towards the end of the backup those latest changes to the file are not reflected in the backup image anymore. This is expected as an image-based backup is not an instant operation but takes some time.
But the more important question remains: Are these AVHDX files really consistent so that we can point our VM after restore to them (as your engineer proposed) or should we instead point the VM to the main VHDX file (my propsal). IMHO the VHDX should be chosen, as it was "closed" at the beginning of backup and during the backup all writes are redirected to the temporary AVHDX, which IMHO is not considered consistent.
Anguel
07-01-2014 04:02 AM
robbieD wrote:
You could manually merge the AVHDX file with the VHDX file within the Hyper-V Manager after the restore.
Rob,
Thank you for your feedback. I know that merging the AVHDX with the VHDX files is possible. However, as described above as far as I understand the AVHDX is auto-created at the beginning of backup only to make sure that we have a consistent VHDX in the backup archive that does not contain some partially written file. Any writes during backup are therefore redirected to the temporary AVHDX which is finally auto-merged into the VHDX after the backup is finished. According to http://www.aidanfinn.com/?p=15759 this is a new technique used by Microsoft in Server 2012 R2 instead of the old VSS method.
Now the Symantec engineer Chris talked to proposes to point a VM restored from backup to its restored AVHDX which is IMHO not a good idea as it contains data that is written by the VM during backup and may be therefore inconsistent. Therefore, I think that it is more safe to point the restored VM to the restored VHDX and not to the AVHDX. Any feedback form Symantec, specifically regarding Windows Server 2012 R2, is welcome.
Anguel