Background: I have BE 2010 R2 installed on a physical Windows 2008 R2 server that has access to the iSCSI SAN and my vCenter 4.1 server. I'm trying to do a Full AVVI backup of my vSphere infrastructure over the weekend using a Disk-to-Disk-to-Tape policy I created and then have Differential Disk-to-Disk-toTape backups happen during the weekdays. I'm using the SAN method and have GRT enabled for SQL. I have the Remote Agent for Windows servers pushed out to all Windows servers on the network, and I've checked to make sure the VMware VSS provider and synch driver are uninstalled.
After some tweaking, my Full backups are running OK (793GB in ~ 13 hours for the toDisk portion, and then another 9 hours to duplicate that to LTO3 via an HP Autoloader). It's a little slower than I expected, but I'll work on tweaking it later.
The primary issue I'm having now is that my daily Differential backups are taking almost as long (11 hours and 6 hours respectively for the disk and tape portions) and have nearly as much data in them. After poring over the logs and troubleshooting this myself I've found that it is primarily due to a rather large virtual compatibility RDM that always gets backed up with the Full method. When I look at the DataStore for this VM I can see the .chk files related to the virtual RDM, so vSphere is change tracking the blocks on it, however BackupExec refuses to perform a differential backup on it.
I had a heck of a time finding anything about this in the documentation. I couldn't see anywhere in the Administrator Guide anything about RDMs breaking differential or incremental backups. I had to search google for quite a while before I found a AVVI agent FAQ that stated that RDMs (both virtual and physical) were not supported for differential or incremental backups using the AVVI. I can totally understand the physical RDMs not working, hell they don't work at all since you can't snapshot a VM witha physical RDM, but I can't understand why a virtual RDM causes any issues, especialy when vSphere 4.1 clearly is tracking the changed blocks.
My current plan of action is to create a new datastore and then use Storage vMotion to change the virtual RDM into a vmdk on a VMFS datastore, but I don't really want to do this as it will cause some replication issues on my NetApp SAN due to all the changed blocks at the SAN level this will cause. My current workaround is to disable differential backups and use standard network agent backups to just backup data, but this is not an ideal solution long-term.
Does anyone have any ideas for a different solution for this virtual RDM issue? Can anyone from Symantec please chime in as to why virtual RDMs are handled in this way?
Hmm - I am wondering if you might have just helped pinpoint why some customers have reported AVVI Differential backups running as fulls.
I will try and setup a test of this (can't promise a timescale) just to see if I get the same results against a Virtual RDM
You know much of this could be the fault of VMware's vStorage API, which has a pretty severe set of limitations that backup vendors must integrate with... The whole physical RDM thing is 100% vmware, and not Symantec's fault. For a virtual RDM, I wouldnt be surprised if it were the same.
I've done a little more digging around and I noticed that Veeam 4.0 didn't properly support virtual-mode RDMs, but in the Veeam 4.1 release they now support change-block tracking in virtual-mode RDMs. I posted on the VMware Community Forums asking for a clarification on whether vSphere 4.1 fully supports CBT on vRDMs: http://communities.vmware.com/thread/289166
Sure, I fully understand the physical RDM issue since VMware can't/won't snapshot a physical RDM. However, it looks like the virtual-mode RDM appears to be tracking changed blocks.
On the attached picture you can see the datastore for one of the VMs in question. DirDB_2.vmdk is the RDM mapping for the vRDM and you can plainly see the associated CTK file for that VMDK.
I compared some log files from a full and the subsequent differential backup. My issue does directly relate to the VM with a vRDM:
|Server||Full Backup from Policy - 10/13/2010 6:19AM||Differential Backup from Policy - 10/13/2010 7:00 PM||Notes|
|Full bytes||Full time||Diff bytes||Diff time|
|DirDB||286,934,302,987||3:33:45||560,905,966,583||7:38:40||Windows 2003 w/ SQL 2005 and File storage. File Storage on vRDM|
As you can see the Full backup size is actually smaller than the differential backup size. The full backup is just backing up the used blocks, whereas the differential backup is backing up the entire VMDK series with used and free space accounted for.
So basically not only is it not using CBT, it's doing a FULL size backup of this VM not even taking into account zeroed blocks.
Per the Designing Backup Solutions for VMware vSphere document (attached):
Limitations on Changed Block Tracking
Changed Block Tracking does not work in any of the following cases:
Virtual hardware version is 6 or earlier.
The virtual disk is a “physical compatibility RDM.”
The virtual disk is attached to a shared virtual SCSI bus.
Changed Block Tracking can be enabled on virtual machines that have disks like these, but when queried for
their change ID, these disks always return an empty string. So if you have a virtual machine with a regular
“O/S disk” and a pass‐through RDM as a “data disk” you can track changes on the “O/S disk” only.
So according to VMware's documentation Change Block Tracking does work on virtual-mode RDMs, just not physical mode. Also VMware's vDR appliance supports differential backups using changed block tracking and does work on virtual-mode RDMs.ware's
Just for completeness I have tested this with Backup Exec 2010 R2 and VMware ESX 4.1.0,260247
Differential backup is still currently running but it looks like not only is it doing a full backup of the vRDM, BUT perhaps more crucially it is NOT ignoring the white space and is therefore trying to backup the complete size of the vRDM (making it very very slow to complete)
I guess for the time being Symantec stand by the information in the FAQ which states that vRDM is not supported by AVVI for Differential and Incremental Backups.
As such please visit the Ideas section at the bottom of the Forums menu at the top of this web page and submit an enhancement request.