SSR 2013: Host-level Hyper-V backups in Win Server 2012 R2

[Anguel]

Hi!

I am experimenting with Symantec System Recovery 2013 and host-level Hyper-V VM backups in Windows Server 2012 R2.

My observations are as follows:

1. SSR is capable of doing hot Hyper-V backups. In this case Hyper-V creates intermediate snapshots (checkpoints / avhdx) during the backup that are auto-merged into the VM being backed up after the backup. Unfortunately, the snapshots stay in the iv2i backup files and I am not sure how to handle them after the restore of the VM from the backup.

2. I tried to run a pre-backup script in SSR that shuts down the VM in order to perform a cold backup of the VM. After the backup I run a post-backup script in SSR to restart the VM. This works ok so far but Hyper-V still creates the snaptshots (checkpoints / avhdx) I described above although the VM is shutdown at the time of backup. Is this behavior expected?

3. I tried to turn off the "backup volume snapshot" integration service for the particular VM through the Hyper-V Manager. According to the docs this should force Hyper-V to move the VM into saved state and perform a cold backup instead of a hot backup. Unfortunately, this does not change anything. The VM continues to run (instead of stopping) and the execution flow is the same as described in 1. Does SSR somehow ignore the Hyper-V settings?

Thanks in advance.

Anguel

[criley]

Anguel,

Maybe this information will help. Again, this has come direct from our engineering team who have in turn got this information directly from a Microsoft trainer.

This is the workflow for VM backups on Windows 2012 R2 Hyper-V:

a) Initiate a snapshot on host OS. This will initiate a snapshot on guest OS.
b) During snapshot process in the guest a new provider (instead of volsnap) is engaged which manages to create a crash consistent copy of VM disks (VHDXs).
c) Now, this copy of VHDX is crash consistent but not application consistent. This is called check-pointing.
d) A new hot disk (AVHDX) gets added to the VM if the VM has SCSI controller. This AVHDX becomes child of VHDX in step b) above.
e) Before completion of snapshot process, Hyper-V writer needs to do some changes to the snapshot volume. These changes now get redirected to AVHDX.
f) Snapshot process now completes. The hot-added disk contains only a few redirected writes from Hyper-V writer to make it consistent.
g) So, the AVHDX + VHDX together makes an application consistent snapshot.
h) Now while the hot disk is added there to capture only the Hyper-V writer initiated writes, other writes happening inside the VM continue to occur after step # c above. These writes get written to another AVHDX.

[Anguel]

Chris,

This information is very helpful and interesting. So it looks like during a backup there is one VHDX file (main virtual disk) plus two AVHDX files (hot disks). When I look into my SSR backup archives they really seem to contain three files per virtual disk, e.g.:

1.) DISK.vhdx

2.) DISK_88278437-AEF5-47BB873847-34376A45.avhdx

3.) DISK-AutoRecovery.avhdx

So let's assume I need to restore such a VM from am SSR backup, which of these virtal disk files should I then point my restored VM to (for an application consistent restore)? As far as I understand this should be probably the DISK_88278437-AEF5-47BB873847-34376A45.avhdx. Is this correct?

Anguel

[criley]

I have confirmed that you need to point the VM to DISK-AutoRecovery.avhdx

[Anguel]

Chris,

Thank you. The answer is very interesting although unexpected :) The AutoRecovery.avhdx is NOT the file the VM is pointing to by default after restore. I am very busy at the moment but I will test if this works as soon as I have time.

I believe that such answers actually belong in the docs of any good backup program. It should not only do the backup but a user should also know how exactly to restore a VM he backed up with SSR in case of data loss. Thank you very much for the support once again.

Regards,

Anguel

[criley]

Yes, I agree with you - this is something that I plan to document in a public article.

[Roland_R_]

Chris, Anguel

after reading and confusion I did many own tests with Windows 2012R2 and last Version of SSR2013.

I have found the following way.

Example:
Name of VM: TestVM
Rootpath of the VM E:\TestVM
Backup running VM

1. Delete the VM in Hyper-V-Manager i want to restore.
2. Delete the Subfolders of the VM-Path (E:\TestVM\Snapshots, E:\Virtual Hard Disks, E:\Virtual Machines)
3. Restore the Subfolders with Recoverypoint-Browser
   There are three files like anguel described

   Mount the recoverypoint as Volume (I:) and copy the permissions with
   robocopy I:\TestVM E:\TestVM /E /COPY:ATSOU /DCOPY:DAT

   This way, because the combination of Recoverypoint-Browser + robocopy was much faster then
   copy Data and Permissions only with robocopy

4. Import the restored VM with Hyper-V-Manager
5. Stop the Hyper-V-Manager
6. Start the Hyper-V-Manager

After 6. Start the Hyper-V-Manager the snapshots were automatically merged and I have only the .vhdx file and no .avhdx.

I this the right way and an application consistent restore?

Please comment, thanks

[Anguel]

Hi Roland,

If I remember I did not try to explicitly delete and then reimport the VM through Hyper-V Manager but just stopped the Hyper-V Manager Service and then copied the restored VM over from the recovery point. I.e. I mounted the recovery point to e.g. X: and used something like

xcopy X:\VMs\WS2012R2ESS E:\VMs\WS2012R2ESS /O /X /E /H /K

But your robocopy solution also sounds nice and probably does the same. I still don't understand why SSR is not able to set the permissions automatically. I expect this from such a program.

Back to the main problem: It is nice to hear that importing the VM with Hyper-V Manager merges the VHDX snapshots. In my case this was not the case as I restored the VMs directly as described above and in my case the VM was then pointing to the temporary AVHDX files instead. Probably you have found the correct way to do an app consistent restore? I have spent a lot of time figuring out how SSR works with Hyper-V and now I have a productive system and don't have more time to spend until something breaks. Of course, I am very disapointed that Symantec is not able to document such things. It is their product, not ours and we are running a supported configuration. So probably Chris can give us some more feedback on your solution and I hope that someone at Symantec will finally sit down, reproduce our problem and tell us what is the official way to do an app consistent VM restore on Hyper-V. Everything else is just guesswork!

UPDATE: I have now unmarked the question. Reading Roland's posting I don't think it is solved completely.

Anguel