Hyper-V backups including deleted blocks...

[sdo]

Hyper-V host: Windows 2012 R2 in a farm using CSV volumes, with NetBackup Client v7.6.1.1.  Storage target is MSDP on 5230 Appliance v2.6.1.1.

Before doing anything major inside the VM - Windows File Explorer (within the VM) showed 34.4 GB C: drive (which was 15.7 GB populated by Windows OS), and a 19.9 GB D: drive (which was 0.8 GB populated by NetBackup Client v7.6.1.1) - thus total disk space provisioned was 54.3 GB, and total used space was 16.5 GB.

The steps I followed:

1) I ran a test full VM backup, which saved 22 GB (clearly 5.5 GB larger than the used space).

2) I then filled up the C: drive and D: drive to near full, and took another full VM backup which saved 53.8 GB.

3) Then I deleted the temporary files, and emptied the waste basket, and ran another full VM backup, and it saved 53.8 GB again - even though total used space was back down to 16.5 GB.

4) I waited 10 hours, and ren the full VM backup again, and again it saved 53.8 GB.

5) Checked within the guest VM, and no VSS shadows are present/left-over after backups.

.

Now, I know that Hyper-V VM backups do not have the feature of ignoring 'deleted' blocks, as per VMware ESXi.

And I did have client-side de-dupe enabled on the Hyper-V host.

.

The reason I'm posting this - is because I need to find out whether:

1) This behaviour is just something that all backup admins have to live with.

2) Am I correct in thinking that the NetBackup Client side de-dupe engine has to re-evaluate/re-finger-print all the deleted blocks every time a full backup is run?

3) Is there anything that can be done from an Hyper-V admin function perspective to reclaim the deleted blocks?

4) Is there anything that can be done from within the guest OS to trigger a 'deleted block' reclaim?

5) If there is no deleted block reclaim function at the Hyper-V host layer or at the Hyper-V guest layer - then am I correct in thinking that... because NTFS is a 'scattering file system' (i.e. it does not prefer to over-write deleted blocks with new blocks) - that thus, slowly, or quickly (depending on rate of change) that at the Hyper-V host storage layer that all VMs will thus incrementally grow their 'thin' provisioned space up to the maximum size?

6) If question 5) is true, then surely this have grave implications to not only storage capacity planning, but also to Hyper-V host CPU capacity planning - because this means that all 'thin' provisioned Hyper-V VMs will grow to max provisioned, and that all this "used at least once but since deleted" space has to be re-evaluated in CPU (i.e. consumes Hyper-V guest CPU, and thus consumes Hyper-V host CPU) and finger-print RAM (inside the guest) every time a full backup is run?

Looking forward to discussing this.  :)

[sdo]

7) I guess the above will also mean that a full Hyper-V VM restore will essentially be thick(er), i.e. a full Hyper-V VM restore of the third VM backup will also restore all the deleted blocks and thus consume the full amount of space of 53.8 GB at the Hyper-V host storage layer, even though the VM useful contents only totals 16.5 GB.

[sdo]

...just continuing to share some observations...

8) A full Hyper-V VM backup essentially backs-up everything, and so includes things like pagefile.sys, and so must therefore not be using the Windows 'FilesNotToBackup' registry key - and must also therefore not be excluding files listed in the NetBackup Client exclude list (which is to be expected I suppose).

9) A differential Hyper-V VM backup does not include (detect or re-form) the Shadow_Copy_Components: nor System_State: as discreet components available for restore - and so a 'differential Hyper-V VM' backup is of limited use for a full system point in time recovery.

10) No Accelerator , no TIR, no BMR for Hyper-V VM backups.

[Michael_G_Ander]

An option that might be worth a look is sdelete, which actually deletes the files rather than just delete the index for the files.  It also has an option cleaning up "free space"

Know that some storage admins I have worked with as used isdelete before reclaiming space(zeros)  in storage system.

Let us know the result if you try the scenario using sdelete

[sdo]

I've reached to our Hyper-V admins... we'll see what they have to say about the first set of points in the leading post.

[Michael_G_Ander]

Any news on this ?

Could be interesting to know how to reduce the Hyper-V backups sizes

[sdo]

Our Hyper-V admins say they don't want to get in to scripting up 'sdelete'.  So it goes.

[sdo]

11) Just thought of another point... that because the deleted blocks within a Hyper-V VM are being picked up, processed and evaluated by the client side de-dupe engine within the Hyper-V host, then this also means that a greater variance of data will be retained within the finger-print cache and also within the MSDP pool(s) themselves - i.e. as time progresses, more and more of the deleted blocks will become more and more unique, i.e. more and more of the deleted blocks will become more different to the deleted blocks within other Hyper-V guest VMs - and so more and more de-dupe space will be unnecessarily consumed within the MSDP pool.  And therefore more data transferred unnecessarily, than might otherwise be expected (e..g compared to using VMware ESXi BLIB/CBT where this would not happen).   Hmmm.  Another reason not to do too many Hyper-V VM level backups, especially on VMs with a medium to high rate of change.  IMO, fairly static VMs could still be a candidate for VM backups - but even they will slowly grow and accumulate deleted blocks.