cancel
Showing results for 
Search instead for 
Did you mean: 

BE 2015 FP3 and hyper-v 2012 R2

dhuygelaere
Level 3

Hi all,

I've an issue with be 2015 FP3 and hyper-v 2012 R2.Full backup of hyper-v host with 2 VM (2012 R2) completed successfully, but checkpoint are not removed, so nexts backup  failed. Individual backup of VM are correct, and checkpoint deleted. Any idea ?

1 ACCEPTED SOLUTION

Accepted Solutions

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

OK as based on the results I have seen on test (Against Hyper-V 2012 R2)

The new/faster method will merge the older checkpoint into the VHDX file and only keep the newer checkpoint file (AVHDX) at the end of each backup. This checkpoint file will then exist until the end of the next backup when it itself is merged into the vhdx.  So when using this method (apart from the first ever full backup), during a backup job 2 checkpoints and 2 AVHDX files will be seen however at the end of the job only the later checkpoint (avhdx file) will remain. As such, as long as regular backups are run, the checkpoint file will not keep growing as the original machine state  (vhdx file) is still updated/merged/committed into regularly. Note: The new/faster method backup will create this checkpoint as the full job runs irrespective of whether only full jobs will be run.

If NO faster method backup jobs have been run and the old/standard method backup is run, this does not leave a checkpoint (or AVHDX file) present when jobs are NOT running but does create a checkpoint (briefly) and an avhdx file during backup operation which is merged into the machine state (vhdx file) at the end.

If changing from running the faster method back to the standard method a checkpoint (avhdx file) will be left in place if the powershell command of  Get-VMSnapshot -VMName 'VM' -ComputerName 'HOST' | Remove-VMSnapshot  has NOT been used when making the change. This checkpoint (AVHDX file) becomes the object that gets updated with the original machine state (VHDX files) remaining untouched and therefore the AVHDX file will grow but the vhdx file will no longer be modified - As this is potentially undesirable behaviour, if doing this change the Powershell command should be used.

If using the faster method and you see more than 1 Backup Checkpoint,  when a job is not running (or more than 2 Backup Checkpoints, when a job is running) then something, outside of Backup Exec's direct control, blocked the removal of an earlier checkpoint and you should review the Hyper-V-VMMS  event log (Event Viewer —> Applications and Services Logs —> Microsoft —> Windows —> Hyper-V-VMMS —> Admin) for errors. Also if this condition is experienced, be prepared to use the powershell command to remove the checkpoints to tidy up. If this is a regular problem and/or the powershell command fails to tidy up the snapshot then Microsoft should be involved to help understand and identify why the checkpoint(s) cannot be removed. Note: the first time a faster method full backup takes place then it is expected to see event ids  15070 and 10155  against unable to remove a checkpoint

 

Note:

At various points in time, whilst backup jobs are active,  AutoRecovery Checkpoints (avhdx files) are also created but these are always removed before the job finishes.

Also no matter whether running standard or faster method jobs “ Backing up”, “Merge in Progress” and “Creating  Checkpoint” statuses can  seen in the Hyper-V  manager against the VM.

 

EDIT MADE AFTER THIS POST WAS MARKED AS A SOLUTION:

A technote has been created to explain this and to give a variation on the Powershell Command to allow removal of the backup checkpoints and not all of the checkpoints - added to this post just to give te most up to date into in the solution post

http://www.veritas.com/docs/000100786

 

 

 

View solution in original post

42 REPLIES 42

CraigV
Level 6
Partner    VIP    Accredited

Have you enabled Advanced Open File Option for the backup? If so, disable it and try again.

Thanks!

dhuygelaere
Level 3

Hi,

Yes, it's enable, for the complete job and for the the single VM backup. GRT too. Only the full job have the issue.

I'll try without AOFO and let you know.

 

Thanks

dhuygelaere
Level 3

Hi,

still the same issue.

VJware
Level 6
Employee Accredited Certified

Which incremental method is being used ? Backup Exec's own or Microsoft's ? (If i recall, the Microsoft's method do not remove checkpoints, they are merged though).

Secondly, any errors/warnings in the Hyper-V VMMS logs on the host ?

dhuygelaere
Level 3

It's full backup, not incremental.

I have an error and a warning in event viewer

error Hyper-V-VMMS 15070 n’a pas pu supprimer le point de contrôle.

warning Hyper-V-VMMS 10155 Échec de la suppression du point de contrôle de sauvegarde précédent de l’ordinateur virtuel « XXX-SRV » : Un ou plusieurs arguments sont incorrects (0x80070057).

VJware
Level 6
Employee Accredited Certified

Check for events which are logged prior to event id 15070.

If nothing found in Hyper-V VMMS logs, look into the Application logs of the host around the same time when 15070 was logged.

Do these VMs have this hotfix installed - https://support.microsoft.com/en-us/kb/2964439  ?

Are you able to merge the checkpoint and retry ?

0x80070057 is a very generic error. I would start off by combing through the event viewer as suggested earlier & double-check the permissions.

 

dhuygelaere
Level 3

No others error in the events logs.

I tried to install the hot fixe, but I can't, it not applicable to my system. (VM and host are fully patched, not in production yet).

I'm able to merge the checkpoint, and as a workaroud, I launch a powershell script in post job task to remove checkpoint.

I've opened a case to the veritas support, but no solution at this time.

thks

VJware
Level 6
Employee Accredited Certified

Do you run only full backups or incrementals as well ?

dhuygelaere
Level 3

Full backup only

steves_2
Level 4

Im having something similar happening to me as well.

I created a new post about it but cant find it anywhere.

Since updating to Backup exec 15 FP3 im seeing server backup checkpoints in the Hyper-v checkpoint window.

Prior to this version i did not see any backup check points. 

Is this something new or is something going wrong with the snapshot processing.

Is anyone else seeing this kind of behavior when backing up Virtual Machines ?

Capture4.JPG

Steve

 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

The new method for Incemental backups of Hyper-V uses a Microsoft Incremental process against Wndows 2012 and later and was available in BE 2014 but required powershell commands to setup and was not the default (so most customers did not use it)

 

Backup Exec 15 FP3 can enable this new method without manual use of Powershell and now uses this as a default (and I believe as long as the settings/environment are compatible can switch existing jobs to the new method as it is installed.)

The way the Microsoft Incremental method works is

  • A child snapshot (checkpoint) is created to collect the changes.
  • On Backup, a new child snapshot is created for current changes and then the parent snapshot is backed up.
  • When the backup is complete, the child and parent snapshot is merged leaving only the new combined snapshot.
  • This combined snapshot remains ion the system and is used as teh new parent for the next backup.

As such if a backup is running there should be 2 snapshots/checkpoints, if no backup is running (as long as one backup has run) there should be 1 snapshot/checkpoint.

As there could be other processes (including manual ones) that might create snapshots/checkpoints it is possible you might see more snapshots/checkpoints so be careful when reviewing the list to understand what created them (look at snapshot names and/or creation times.

 

There should be settings to allow you to switch back to the legacy method in your job configuration. (Correction this is a Backup Exec Server setting and not a per job setting)

 

 

 

dhuygelaere
Level 3

yes, but we are talking about FULL backup, not incremental ...

steves_2
Level 4

Thanks for clearing that up Colin. When i upgraded to FP3 it enabled this feature by default.

I am only running Full backups as well so its quite concering having snapshots lying around on the server that only get merged evey backup.

I can understand this being a good this for people why do incremental backups, but i do full daily backups so it shouldn't do this.

I tried swithing back to the default legacy method of backup however this still left the check point behind as a diff disk and everything was getting merged into that

not the parent .vdhx

So all my virtual machines now have recovery checkpoints next to them which i cannot merge and have an ever increasing differential disk.

 

Steve

 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

I believe it might always have one checkpoint even if only doing fulls - however I do need to check this point so thanks for letting me know this part of the behavior.

I also need to check the behaviour when you switch back to traditional method as the info from steves 2 is concerning.

 

 

 

 

CELTA63
Level 2

Hello i exactly the same problem than steve 2 i want to make backup full and i don't want having check point behind two backups

in my case if i check standard method in virtual machine option the operation returns to normal on servers 2012 but not on 2012 R2

And i cannot merge recovery checkpoints on hyper-v administrator

Delphine

 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

I have been testing various scenarios with Hyper-V 2012 R2 and the faster (new) method today and more or less know the expected behaviour and t does match what I wrote earlier in this thread

Tomorrow I intend to try and switch back to the standard method and have at least 2 tests to perform but expect to see something similar to what you have mentioned

If the behaviour has changed between 2012 (not R2) and 2012 R2 then it is possible the Microsoft has changed the behaviour and it will take me a bit longer to verify that as I will need to provision a 2012 (pre R2) Hyper-V setup. However thanks for telling me about this difference.

 

 

steves_2
Level 4

Hi Colin,

 

Thanks for taking the time to investigate this issue. My setup is 2012r2.as well.

I also have been running some tests on temp backup server. The temp backup server is doing exacty the same as the production backup server.

The snapshots do not disappear when reverting back to the legacy method and continue to merge into the differential disk not the primary vhdx.

This leaves the primary parent vhdx in limbo.

Im my optionion there is no need for finger printing or the new faster method checkpoints if there is no incremental/differential backups selected.

@ Delpine 

Thanks for confirming you have the same issue.

@ dhuygelaere

Sorry for hi-jacking your thread but is this the same kind of issue you are experiancing ?

 

Steve

 

Aleksey_Suslov
Level 3

Faced with the same problem.
After any planned job checkpoint not deleted, but for a single job no problem - checkpoint deteled.

Temporary solve the problem. 
powershell command Get-VMSnapshot -VMName 'VM' -ComputerName 'HOST' | Remove-VMSnapshot

 

 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

Just for info on my first test of switching back to standard method, whilst I was using Hyper-V 2012 hosts (in a cluster) I noticed what appears to be another difference - I had two VMs, one running 2012 R2 and one runnng 2008R2. Since the standard backup (where I deliberately started an incremental job, which then ran as a full)  the checkpoints now show different statuses on the 2008R2 VM compared with the 2012R2 one.

I am running the incremental again and then will start a full. Once I have done this I intend to switch back to the 'faster' method to see what then happens to the checkpoints.

In the meantime I am provisioning a 2012 non R2 hyper-V setup as well.

Once I have my findings sorted out I will be feeding that back to internal stakeholders (engineering and developement etc) to see what is expected behaviour and what is not. Note: as we are in effect hooking into a Microsoft method to backup VMs incrementally it is possible that the bevahiour and any changes between OS versions is outside of our control.