cancel
Showing results for 
Search instead for 
Did you mean: 

BE 2015 FP3 and hyper-v 2012 R2

dhuygelaere
Level 3

Hi all,

I've an issue with be 2015 FP3 and hyper-v 2012 R2.Full backup of hyper-v host with 2 VM (2012 R2) completed successfully, but checkpoint are not removed, so nexts backup  failed. Individual backup of VM are correct, and checkpoint deleted. Any idea ?

42 REPLIES 42

steves_2
Level 4

Hi Colin,

Thats for that but In my senario im not running any incremental backups.

The Backup server is 2012r2. The HV Host is 2012r2 and the VM'S are 2012r2.

These are Full backups only. Please can you try the following.

Do a full backup of 2012r2 using the new method.

Swap back to the legacy method and do a full backup.

Look at the Folder where the VHDX Files are stored and note the modified time of the original VDHX file and the AVHDX File.

Do another Legacy Full Backup and physically look at the File Location where the VHDX files are stored when the backup is running .

You will see 2 AVHDX Files when the backup is running.When it has finished the newer AVHDX File will merge into the Older AVHDX file.

Look at the original VHDX and it will have the same timestamp as before and the AVHDX file will have grown larger.

Repeat Legacy backup again and repeat above.

You should see what im getting. The Disks are not being merged into the original VHDX and the AVHDX file is growing out of control.

Once you have performed a new method backup and swap back to legacy the check points are never removed and continue to grow.

 

Steve

 

steves_2
Level 4

@ Aleksey Suslov.

 

Thanks for that tip about removing removing the checkpoint with a powershell command.

I shall try that on my test setup to see what happens.

 

steve

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

@Steves 2  --- Thanks for the suggested test steps - I was going to try the ful backups only scenario eventually although I have a feeling that as I saw a checkpoint get created when I had only run a full backup, that the behaviour will be more or less the same.

steves_2
Level 4

Hi Colin,

 

The difference is that if you start with no check points and do a full legacy, the check point is created during the backup then removed when finished. Everythinkg works great.

But as soon as you switch to new method and back to legacy then thats when things start going pear-shaped.

Maybe if you observe the legacy backup method first then you might get a better idea.

 

To be honest if you havent got any incremental backups selected i dont think it should be doing either new method of checkpoint or legacy method of finger printing. I think the option should be in the incremental setting.

Steve

CELTA63
Level 2

@ Aleksey Suslov

Thanks a lot

with your command and backup with standard option everything is in order on every server for full backup

Delphine

 

steves_2
Level 4

Ah thats good to know. Not got round to testing it yet.

Does the command actually do a merge and not just delete the checkpoint.?

 

Steve

CELTA63
Level 2

The command make fusion you can see on hyper v administrator

Delphine

 

CELTA63
Level 2

Yes sorry in french is "fusion" so command make merge :)

Delphine

steves_2
Level 4

Im guessing fusion could mean merge in a different country :)

Spanish maybe.

Steve

steves_2
Level 4

Thats great , so thanks to Aleksey Suslov's Remove-VMSnapshot , we can manually revert back to the old method of backing up and removing the issues the new version has caused.

 

Steve

 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

@Steves 2 the avhdx was merged back to the vhdx  when I tested with onky full son the WIndows 2012 Hyper-V host (against a Windows 2008 VM)

 

Because of other tests I have not yet done the same test on Windows 2012 R2 Hyper-V host, although based on other results, I don't expect to see the same when I do test it (hopefully tomorrow)

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

@Steves 2 and Aleksey Suslov - good to know that remove snapshot command resets things correctly.

steves_2
Level 4

Hi Colin,

 

I just wondered if you were able to replicate the issues we were having ?

 

Cheers

steve

 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

OK as based on the results I have seen on test (Against Hyper-V 2012 R2)

The new/faster method will merge the older checkpoint into the VHDX file and only keep the newer checkpoint file (AVHDX) at the end of each backup. This checkpoint file will then exist until the end of the next backup when it itself is merged into the vhdx.  So when using this method (apart from the first ever full backup), during a backup job 2 checkpoints and 2 AVHDX files will be seen however at the end of the job only the later checkpoint (avhdx file) will remain. As such, as long as regular backups are run, the checkpoint file will not keep growing as the original machine state  (vhdx file) is still updated/merged/committed into regularly. Note: The new/faster method backup will create this checkpoint as the full job runs irrespective of whether only full jobs will be run.

If NO faster method backup jobs have been run and the old/standard method backup is run, this does not leave a checkpoint (or AVHDX file) present when jobs are NOT running but does create a checkpoint (briefly) and an avhdx file during backup operation which is merged into the machine state (vhdx file) at the end.

If changing from running the faster method back to the standard method a checkpoint (avhdx file) will be left in place if the powershell command of  Get-VMSnapshot -VMName 'VM' -ComputerName 'HOST' | Remove-VMSnapshot  has NOT been used when making the change. This checkpoint (AVHDX file) becomes the object that gets updated with the original machine state (VHDX files) remaining untouched and therefore the AVHDX file will grow but the vhdx file will no longer be modified - As this is potentially undesirable behaviour, if doing this change the Powershell command should be used.

If using the faster method and you see more than 1 Backup Checkpoint,  when a job is not running (or more than 2 Backup Checkpoints, when a job is running) then something, outside of Backup Exec's direct control, blocked the removal of an earlier checkpoint and you should review the Hyper-V-VMMS  event log (Event Viewer —> Applications and Services Logs —> Microsoft —> Windows —> Hyper-V-VMMS —> Admin) for errors. Also if this condition is experienced, be prepared to use the powershell command to remove the checkpoints to tidy up. If this is a regular problem and/or the powershell command fails to tidy up the snapshot then Microsoft should be involved to help understand and identify why the checkpoint(s) cannot be removed. Note: the first time a faster method full backup takes place then it is expected to see event ids  15070 and 10155  against unable to remove a checkpoint

 

Note:

At various points in time, whilst backup jobs are active,  AutoRecovery Checkpoints (avhdx files) are also created but these are always removed before the job finishes.

Also no matter whether running standard or faster method jobs “ Backing up”, “Merge in Progress” and “Creating  Checkpoint” statuses can  seen in the Hyper-V  manager against the VM.

 

EDIT MADE AFTER THIS POST WAS MARKED AS A SOLUTION:

A technote has been created to explain this and to give a variation on the Powershell Command to allow removal of the backup checkpoints and not all of the checkpoints - added to this post just to give te most up to date into in the solution post

http://www.veritas.com/docs/000100786

 

 

 

steves_2
Level 4

Thanks Colin,

It was the changing from faster method  to standard method i was interested in.

Looks like it does the same for you as what we were experiencing.

The avhdx was being updated not the original vhdx.

Is this classed as a bug and will be fixed when swapping methods ?

Not that i would want to swap back to the newer method as i always do full backups.

 

Cheers

steve

 

 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

Yep in this situation the avhdx is getting uodated and not the vhdx

 

I have asked why this happens when swapping methods so at the moment I don't know if it counts as a bug or not.

 


 

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

Going back to the orginal posters problem now that we have covered the observable basics when it works.

The original question was:

I've an issue with be 2015 FP3 and hyper-v 2012 R2.Full backup of hyper-v host with 2 VM (2012 R2) completed successfully, but checkpoint are not removed, so nexts backup  failed. Individual backup of VM are correct, and checkpoint deleted. Any idea ?

So the first point is to be aware of is the checkpoints should keep changing timestamp but at least one checkpoint will always be present (unless you change to standard method and use the powershell command).

Second point is, in the Hyper-V-VMMS event log (form the Hyper-V host) it is expected to see an error state for unable to remove checkpont during the first full backup that uses the new/faster method.

 

To comment on the backup failure we would potentially need the event logs messages (covering what happens on the second or later backup and not the first ) along with the actual failure message in the Backup Exec job logs. This might turn out to be something on the Hyper-V side of things that needs MS input.

 

dhuygelaere
Level 3

Hi all,

Update to this issue. We revert back to legacy backup method (standard method) and merged  the checkspoint manually (powershell VMSnapshot -VMName 'VM' -ComputerName 'HOST' | Remove-VMSnapshot  ) . Now backup are ok, and no checkpoint left after the backup, so I removed the post job script.

@ Colin " Note: the first time a faster method full backup takes place then it is expected to see event ids 15070 and 10155 against unable to remove a checkpoint"

I had this error every backup (but I merged checkpoint after the backup so, maybe working as first time backup)

The original problem was :

- First backup complete successfully

- Second backup failed with these error:

Sauvegarde -- VRTSRV::\\HV-XXXX\Hyper-V?Virtual?Machine\YYYYY-SRV
Accès refusé au fichier C:\Hyper-V\Virtual Hard Disks\YYYYY-SRV\YYYYY-SRV_C.vhdx.
Erreur lors de la lecture du flux de données de sécurité.
V-79-57344-65033 - Répertoire introuvable. Impossible de sauvegarder le répertoire \C:\Hyper-V\Virtu...
 

Sauvegarde -
V-79-57344-34108 - Une erreur inattendue est survenue lors du nettoyage des volumes de cliché. Assur...

And I saw that my Hyper-V server running out of disk space. (due to the .avhdx growing out of control) And in the hyper-v console, checkpoint on each VM that I can't remove in the interface.

I checked the event log, and I found this:

Nom du journal :System
Source :       volsnap
Date :         04/01/2016 23:46:32
ID de l’événement :35
Catégorie de la tâche :Aucun
Niveau :       Erreur
Mots clés :    Classique
Utilisateur :  N/A
Ordinateur :   HV-XXXX
Description :
Les clichés instantanés du volume C: ont été annulés car le cliché instantané du volume n’a pas pu s’agrandir.
XML de l’événement :
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="volsnap" />
    <EventID Qualifiers="49158">35</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2016-01-04T22:46:32.298953500Z" />
    <EventRecordID>4531</EventRecordID>
    <Channel>System</Channel>
    <Computer>HV-XXXX</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\HarddiskVolumeShadowCopy5</Data>
    <Data>C:</Data>
    <Binary>000000000200300000000000230006C0000000000000000001000000000000000000000000000000</Binary>
  </EventData>
</Event>

 

In conclusion :

Full backup with the new fast method create .avhdx file (checkpoint are visible in the hyper-v console) that are NOT merged after the backup and growing out of control, and due to insufficient free space on hyper-v drive, the next backup failed.

Full backup with standard method create .avhdx file (but no chekpoint in the hyper-v console) that are merged after the backup, so my hyper-v drive is not running out of disk space

Question:

If it's a normal situation, how many free space do I need on the hyper-v to backup with the new method (fast)?

Ok Ok I know the answer, it depend how many I/O on the VM  etc etc ...

So I will backup with the standard method, no issue with it

Thanks

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

New technote covering the scenario is now available here

 

http://www.veritas.com/docs/000100786

 

dperlberg
Level 2

Colin:

what we be the desire for this behavior? one major thing is that once there are these checkpoints, you no longer are able to expand the disk/etc. so your basically dead in the water. i have spent the night re-checkpointing, exporting that checkpoint. importing that into a new HV machine to get it back to just a vhdx with no checkpoints. only to have the backup again cause this. i will be changing the setting discussed above - but this is a very big issue. why would they default this on and not make it very clear? here is to a long weekend of work trying to get things back how they were desired...

 

Dean