backing up thin VMs

[KCSCsupport]

Using Backup Exec 2010 with all updates.  I am backing up several VMs using Agent for VMware Virtual Infrastructure.  One day, two of my VMs just started backing up as the provisioned size instead of the used space. What's the deal? The two VMDK files are 7GB and 8GB, but are backing up as 32GB (virtual partition size). Help

[RahulG]

Please refer http://support.veritas.com/docs/357058 - Backups of a virtual machine will backup the entire provisioned disk size instead of just the used space

[KCSCsupport]

that link is dead

[KCSCsupport]

that doc says your thin disk will only be backed up as thick for the following reasons:

• 
  Non NTFS partitions
• Raided disks
• Cluster disks
• Linux Partitions
• If the VM that the VMDK is attached to has a SNAPSHOT created before the backup
• There are others as well

none of those are true for me. also, this just suddenly started happening. before, they were being backed up as thin.  i had been changing the configuration of other VM's, but they were not even on the same host.

[KCSCsupport]

anybody ever seen/heard of this? my nightly backup is taking an extra hour. please help

it's funny that this happened right after i imaged another VM on another host.  I imaged it off with Ghost, killed the VMDK file, then put the image back (resulting in a smaller VMDK file because I had deleted a lot of stuff on this VM). That same day, I took a snapshot of a different VM on a different host (the same host my backup server is on). None of this was on the same host of the problem VMs.

The first backup after these events resulted in the following error:
-----------------------------------------------
Verify- \\data.kcsc.local\D: An inconsistency was encountered on the storage media in QUANTUM 1.
V-79-57344-33994 - The data being read from the media is inconsistent.
VerifyA selection on device VMVCB::\\backup.kcsc.local\VCGuestVm\(DC)KCSC(DC)\vm\data2 was skipped because of previous errors with the job.
A selection on device VMVCB::\\backup.kcsc.local\VCGuestVm\(DC)KCSC(DC)\vm\EngSvr was skipped because of previous errors with the job.
A selection on device VMVCB::\\backup.kcsc.local\VCGuestVm\(DC)KCSC(DC)\vm\dc was skipped because of previous errors with the job.
A selection on device VMVCB::\\backup.kcsc.local\VCGuestVm\(DC)KCSC(DC)\vm\update was skipped because of previous errors with the job.
A selection on device VMVCB::\\backup.kcsc.local\VCGuestVm\(DC)KCSC(DC)\vm\mail1 was skipped because of previous errors with the job.
A selection on device C: was skipped because of previous errors with the job.
-----------------------------------------------

From then on out, it has been backing up two of my VMs as thick instead of thin.

[shorrj]

Is your datastore VMFS or NFS?

[Jim_S]

I'm working through a similar issue with 1 of my VMs.  I removed the VM from inventory in vCenter and added it back and then removed and re-added it to the BackupExec 2010R2 selection list.  I'm running another full backup this evening and will let you know if that did anything to resolve the issue.

[RyanW]

Same here. Thin vmdk files, but being backed up in their entire provisioned size.  I wrote something on this a couple weeks ago and nobody really had anything for me.

[shorrj]

Jim, Ryan - Are your datastores NFS or VMFS?

[Jim_S]

I'm using VMFS. 

The particular VM that I'm having an issue with used to also have a vRDM, but I ditched that and Storage vMotioned it over to a thin VMDK (see https://www-secure.symantec.com/connect/forums/differential-vm-backups-backupexec-2010-r2-and-avvi for history on this).  It backed up once properly during the full backup cycle once and then the next Monday during a policy-based differential backup it did a full provisioned size backup and then was broken ever since. 

I removed the VM from inventory in vCenter and will find out shortly whether it is going to back up properly this afternoon or if it's going to try to back up the entire inflated size.

I had another VM that randomly did a full inflated size backup that was also a thin-provisioned VMDK, but it also randomly resolved itself. :)

BackupExec has thousands of capabilities, I just wish they were all properly documented, fully supported, and worked like enterprise software should: flawlessly.

[shorrj]

My problem is that VMFS-thin works fine but NFS-thin doesn't.  I wish you guys the best of luck in finding the solution. 

[RyanW]

VMFS.

[Jim_S]

Well, removing the VM from inventory in vCenter, adding it back, and performing a full backup didn't do the trick.  I also now have multiple VMs doing this :(

BackupExec 2010R2 Log excerpt:

            Server - DC01
                  Set Information -
                  VMVCB::\\vc01.fib.com\VCGuestVm\(DC)FIB01(DC)\vm\Imaging\DirDB
                        Backup Set Information
Family Name: "Media created 11/6/2010 6:00:03 PM"
Backup of "VMVCB::\\vc01.fib.com\VCGuestVm\(DC)FIB01(DC)\vm\Imaging\DirDB"
Backup set #1 on storage media #1
Backup set description: "Weekly Full Backup"
Backup Method: Full - Back up virtual machines
The option to enable the restore of individual files and folders from a virtual machine backup was selected for this backup.
The Remote Agent for Windows Systems must be installed on the virtual machine to use Granular Recovery Technology to restore data.

Backup started on 11/6/2010 at 7:35:16 PM.
Backup Set Detail Information
Media Label: IMG000263
GRT backup set folder: X:\VCB\IMG000263
Transport mode 'san' was used for the disk 'DirDB.vmdk'
Transport mode 'san' was used for the disk 'DirDB_1.vmdk'
Transport mode 'san' was used for the disk 'DirDB(1).vmdk'
Transport mode 'san' was used for the disk 'DirDB(2).vmdk'
Backup Exec has discovered and protected 'C:' on virtual machine '\\DIRDB.fib.com'.
Backup Exec has discovered and protected 'F:' on virtual machine '\\DIRDB.fib.com'.
Backup Exec has discovered and protected 'G:' on virtual machine '\\DIRDB.fib.com'.
Backup Exec has discovered and protected 'H:' on virtual machine '\\DIRDB.fib.com'.
Backup Exec has discovered and protected 'I:' on virtual machine '\\DIRDB.fib.com'.
Backup Exec has discovered and protected 'Y:' on virtual machine '\\DIRDB.fib.com'.
Backup Exec has discovered and protected Microsoft SQL Server data on virtual machine '\\dirdb.fib.com'.

Backup completed on 11/6/2010 at 11:21:30 PM.
Backup Set Summary
Backed up 13982953 files in 36940 directories.
Backed up 1 virtual machines
Backed up 4 virtual machine disks
Processed 600912914582 bytes in  3 hours,  46 minutes, and  14 seconds.
Throughput rate: 2533 MB/min
Compression Type: None

From the log we can see that BackupExec backed up approximately 600GB of data.  However, when I browse out to the GRT folder on the BackupToDisk path on the media server I see that it only actually backed up approximately 275GB (see attachment DirDB BE MediaCenterView.jpg).  If I look in vCenter I can see that the VM is using at most around 316GB.  This VM has mostly thick disks and 1 thin disk.

I've got another VM that BackupExec log says it backed up ~165GB, there's 112GB in the IMG folder, and vCenter shows 108.53 GB used.  This VM has all thin disks.

For yet another VM the BackupExec log says it backed up ~100GB, there's ~31GB in the IMG folder, and vCenter shows 80.4GB used.  This VM has 3 thick vmdks and 1 thin.

All of my VMs that are 100% thick disks are working fine.  Any VM with any thin disks is not.

I guess the obvious solution would be to storage vmotion all the thin disks into thick, but really shouldn't Symantec's product just work as advertised and (poorly) documented?

I guess really my main issue here isn't with the D2D part of the backup policy, as these problem VMs aren't really taking up any more space on my media server that I can see.  The problem arises when the job is duplicated to tape and it writes the data amount that the backup exec log shows and not the amount of data that is actually there.  This causes my D2T duplication job to take 2 extra LTO3 cartridges, which also means it takes a lot longer to run and is more expensive.  This job should easily fit on 2 tapes, and it's instead taking 4.

[Jim_S]

I'm to understand from a different thread that thin vmdks on NFS is a known issue.  Would anyone from Symantec be able to chime in as to whether issues with thin vmdks on iSCSI is a known issue also please?

[Colin_Weaver]

I tested with a Windows 2008 R2 VM stored on an iSCSI (VMFS) Datasore on an Equalogic Array last week.

The VM  had 2 vmdk files, 1x 40GB Thick Disk and 1x 60GB Thin disk. The thick disk had 16.4GB used and the thin disk had 16.9GB used

Backup Job log reported 33GB backed up  and when I checked the Backup to Disk Location the VMDK files stored there were 12GB and 16.95GB  (note I think the difference on the smaller drive is because of the swap file being somehow excluded )

I was not testing Thin disks specifically my test was actually against iSCSI performance in general - however my figures etc at least show that we can't easily reproduce iSCSI thin disks backing up as full (unlike the NFS issue which we can reproduce.)

[Jim_S]

So it would seem that opening a support case would be my only possible means to resolution at this point then?

[itsmorefun]

Same problem for me but i use FC between ESX and SAN (and ethernet between esx and backup exec).

ESXi 4.1Backup Exec  2010

:(

[harsi]

Put me on the list, same problem here.

All thin provisioned disks are backed up as full.

They are all located on a VMFS datastore.

Sorry but this product is full of bugs.

[pkh]

This is a known issue.  You can add your "me too" here

https://www-secure.symantec.com/connect/issue/backup-virtual-machine-backs-entire-disk-space-instead...

[harsi]

Great to see that Symatec is aware of the problem since January 2010.

So we have a chance this will be addressed in 2012?

Mabye time to ask for a refund because this renders the agent useless.

Strange enough that this issue did not arouse when we were testinig the product. It has appeared all of a sudden.