I've an issue with be 2015 FP3 and hyper-v 2012 R2.Full backup of hyper-v host with 2 VM (2012 R2) completed successfully, but checkpoint are not removed, so nexts backup failed. Individual backup of VM are correct, and checkpoint deleted. Any idea ?
Solved! Go to Solution.
OK as based on the results I have seen on test (Against Hyper-V 2012 R2)
The new/faster method will merge the older checkpoint into the VHDX file and only keep the newer checkpoint file (AVHDX) at the end of each backup. This checkpoint file will then exist until the end of the next backup when it itself is merged into the vhdx. So when using this method (apart from the first ever full backup), during a backup job 2 checkpoints and 2 AVHDX files will be seen however at the end of the job only the later checkpoint (avhdx file) will remain. As such, as long as regular backups are run, the checkpoint file will not keep growing as the original machine state (vhdx file) is still updated/merged/committed into regularly. Note: The new/faster method backup will create this checkpoint as the full job runs irrespective of whether only full jobs will be run.
If NO faster method backup jobs have been run and the old/standard method backup is run, this does not leave a checkpoint (or AVHDX file) present when jobs are NOT running but does create a checkpoint (briefly) and an avhdx file during backup operation which is merged into the machine state (vhdx file) at the end.
If changing from running the faster method back to the standard method a checkpoint (avhdx file) will be left in place if the powershell command of Get-VMSnapshot -VMName 'VM' -ComputerName 'HOST' | Remove-VMSnapshot has NOT been used when making the change. This checkpoint (AVHDX file) becomes the object that gets updated with the original machine state (VHDX files) remaining untouched and therefore the AVHDX file will grow but the vhdx file will no longer be modified - As this is potentially undesirable behaviour, if doing this change the Powershell command should be used.
If using the faster method and you see more than 1 Backup Checkpoint, when a job is not running (or more than 2 Backup Checkpoints, when a job is running) then something, outside of Backup Exec's direct control, blocked the removal of an earlier checkpoint and you should review the Hyper-V-VMMS event log (Event Viewer —> Applications and Services Logs —> Microsoft —> Windows —> Hyper-V-VMMS —> Admin) for errors. Also if this condition is experienced, be prepared to use the powershell command to remove the checkpoints to tidy up. If this is a regular problem and/or the powershell command fails to tidy up the snapshot then Microsoft should be involved to help understand and identify why the checkpoint(s) cannot be removed. Note: the first time a faster method full backup takes place then it is expected to see event ids 15070 and 10155 against unable to remove a checkpoint
At various points in time, whilst backup jobs are active, AutoRecovery Checkpoints (avhdx files) are also created but these are always removed before the job finishes.
Also no matter whether running standard or faster method jobs “ Backing up”, “Merge in Progress” and “Creating Checkpoint” statuses can seen in the Hyper-V manager against the VM.
EDIT MADE AFTER THIS POST WAS MARKED AS A SOLUTION:
A technote has been created to explain this and to give a variation on the Powershell Command to allow removal of the backup checkpoints and not all of the checkpoints - added to this post just to give te most up to date into in the solution post
It's full backup, not incremental.
I have an error and a warning in event viewer
error Hyper-V-VMMS 15070 n’a pas pu supprimer le point de contrôle.
warning Hyper-V-VMMS 10155 Échec de la suppression du point de contrôle de sauvegarde précédent de l’ordinateur virtuel « XXX-SRV » : Un ou plusieurs arguments sont incorrects (0x80070057).
Check for events which are logged prior to event id 15070.
If nothing found in Hyper-V VMMS logs, look into the Application logs of the host around the same time when 15070 was logged.
Do these VMs have this hotfix installed - https://support.microsoft.com/en-us/kb/2964439 ?
Are you able to merge the checkpoint and retry ?
0x80070057 is a very generic error. I would start off by combing through the event viewer as suggested earlier & double-check the permissions.
No others error in the events logs.
I tried to install the hot fixe, but I can't, it not applicable to my system. (VM and host are fully patched, not in production yet).
I'm able to merge the checkpoint, and as a workaroud, I launch a powershell script in post job task to remove checkpoint.
I've opened a case to the veritas support, but no solution at this time.
Im having something similar happening to me as well.
I created a new post about it but cant find it anywhere.
Since updating to Backup exec 15 FP3 im seeing server backup checkpoints in the Hyper-v checkpoint window.
Prior to this version i did not see any backup check points.
Is this something new or is something going wrong with the snapshot processing.
Is anyone else seeing this kind of behavior when backing up Virtual Machines ?
The new method for Incemental backups of Hyper-V uses a Microsoft Incremental process against Wndows 2012 and later and was available in BE 2014 but required powershell commands to setup and was not the default (so most customers did not use it)
Backup Exec 15 FP3 can enable this new method without manual use of Powershell and now uses this as a default (and I believe as long as the settings/environment are compatible can switch existing jobs to the new method as it is installed.)
The way the Microsoft Incremental method works is
As such if a backup is running there should be 2 snapshots/checkpoints, if no backup is running (as long as one backup has run) there should be 1 snapshot/checkpoint.
As there could be other processes (including manual ones) that might create snapshots/checkpoints it is possible you might see more snapshots/checkpoints so be careful when reviewing the list to understand what created them (look at snapshot names and/or creation times.
There should be settings to allow you to switch back to the legacy method in your job configuration. (Correction this is a Backup Exec Server setting and not a per job setting)
Thanks for clearing that up Colin. When i upgraded to FP3 it enabled this feature by default.
I am only running Full backups as well so its quite concering having snapshots lying around on the server that only get merged evey backup.
I can understand this being a good this for people why do incremental backups, but i do full daily backups so it shouldn't do this.
I tried swithing back to the default legacy method of backup however this still left the check point behind as a diff disk and everything was getting merged into that
not the parent .vdhx
So all my virtual machines now have recovery checkpoints next to them which i cannot merge and have an ever increasing differential disk.
Hello i exactly the same problem than steve 2 i want to make backup full and i don't want having check point behind two backups
in my case if i check standard method in virtual machine option the operation returns to normal on servers 2012 but not on 2012 R2
And i cannot merge recovery checkpoints on hyper-v administrator
I have been testing various scenarios with Hyper-V 2012 R2 and the faster (new) method today and more or less know the expected behaviour and t does match what I wrote earlier in this thread
Tomorrow I intend to try and switch back to the standard method and have at least 2 tests to perform but expect to see something similar to what you have mentioned
If the behaviour has changed between 2012 (not R2) and 2012 R2 then it is possible the Microsoft has changed the behaviour and it will take me a bit longer to verify that as I will need to provision a 2012 (pre R2) Hyper-V setup. However thanks for telling me about this difference.
Thanks for taking the time to investigate this issue. My setup is 2012r2.as well.
I also have been running some tests on temp backup server. The temp backup server is doing exacty the same as the production backup server.
The snapshots do not disappear when reverting back to the legacy method and continue to merge into the differential disk not the primary vhdx.
This leaves the primary parent vhdx in limbo.
Im my optionion there is no need for finger printing or the new faster method checkpoints if there is no incremental/differential backups selected.
Thanks for confirming you have the same issue.
Sorry for hi-jacking your thread but is this the same kind of issue you are experiancing ?
Just for info on my first test of switching back to standard method, whilst I was using Hyper-V 2012 hosts (in a cluster) I noticed what appears to be another difference - I had two VMs, one running 2012 R2 and one runnng 2008R2. Since the standard backup (where I deliberately started an incremental job, which then ran as a full) the checkpoints now show different statuses on the 2008R2 VM compared with the 2012R2 one.
I am running the incremental again and then will start a full. Once I have done this I intend to switch back to the 'faster' method to see what then happens to the checkpoints.
In the meantime I am provisioning a 2012 non R2 hyper-V setup as well.
Once I have my findings sorted out I will be feeding that back to internal stakeholders (engineering and developement etc) to see what is expected behaviour and what is not. Note: as we are in effect hooking into a Microsoft method to backup VMs incrementally it is possible that the bevahiour and any changes between OS versions is outside of our control.