cancel
Showing results for 
Search instead for 
Did you mean: 

VMware incrementals fail

Chris_Rudin
Level 3

BE 2010, AVVI option licensed,  ESXi 4 Update 2 with vSphere Server.

I can perform full backups without issue.  However, the daily incrementals fail with:

V-79-57344-38324 - Incremental\\Differential backup methods are not supported for this VM. The bac...


VSphere says:

[2010-07-13 20:07:15.491 04812 info 'App'] [VpxLRO] -- ERROR task-internal-15027 -- vm-988 -- vim.VirtualMachine.queryChangedDiskAreas: vmodl.fault.InvalidArgument:

With the faulting property being 'ChangeID'

The incremental backup does work on one machine though.  All of the vms have been stunned and have ctk files in their directories.  I am at a loss as to why 1 machine works and the other 9 fail with that error.

Example snip from one of the VMXs:

scsi0:0.ctkEnabled = "TRUE"
scsi0:1.ctkEnabled = "TRUE"
ctkEnabled = "TRUE"

This machine failes with the above.

Any ideas?

15 REPLIES 15

AmolB
Moderator
Moderator
Employee Accredited Certified
Hi Chris

 Refer to the below link 

http://support.veritas.com/docs/355917

Chris_Rudin
Level 3

All virtual machines are version 7.  Also the incremental backup was created via a policy.  The same policy that handles the full backups.  I also set the rule that sets "Full backup must run before incrementals" in the policy and the incremental job started running after the full backup as expected.

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

Backup Exec itself sets the ctkEnabled settings you have mentioned as long as teh security is correct;

The following rather lengthy information about security settngs might be useful for troubleshooting this.

When Checking VMWare Config & permissions (particularly for Incremental and DIfferential problems)
 
To check ESX version:
1) Login vSphere Client to vCenter or ESX Host.
2) Click on object that represents vCenter of Host
3) Look in title bar of right hand pane (above the tabs) for the version.
 
To Check Hardware Version 7
1) In vSphere client click on a VM and then click the Summary tab
2) Review VM Version Settings in General section.
 
 
To Review Change Tracking Settings (Differential and Incremental Backups):
Note: this should be set by BE so will only be set if jobs inside of Differential or incremental policy have been run (or can be set manually)
1) Shutdown the VM
2) In the vSphere Client, edit the settings of a virtual machine
3) Click the Options tab
4) Click on the (Advanced-->) General settings
5) Click the Configuration Parameters... button (this will be grayed out if you have not shutdown the virtual machine)
6) Check the list of parameters for one called ctkEnabled and one for each VMDK in the format scsiX:Y.ctkEnabled and make sure all these settings are listed as True. Note if they are not present they can be manually created however it is an indication that the security might be incorrect as Backup Exec should be able to create and set these to true
 
To Review/set the security (ESX Host connection):
1) Login the vSphere client directly to the ESX host
2) Click on a virtual machine.
3) Click the Permissions tab
4) Verify that the root user (or specific ESX user) is listed and given the "Administrator" role
5) If the user is NOT listed make a note of the entries in the Defined in column for existing users
6) Select the object in the left hand pane that matches that seen in the “Defined in” column against the existing users and click the Permissions tab for this object
7) Right click in blank space on the permission screen and select Add Permission
8) Use this screen to add your user and make sure you set the "Assigned Role" to "Administrator" and that "Propagate to Child Objects" is enabled.
9) Repeat steps 2 though 4 to verify that the specific user (root or custom) is now listed with the "Administrator" role.
10) Make sure this user is also used in the backup selection list.
Note: If you set the permissions too low in the object tree you won't have permissions to the datastores that hold the virtual machines
 
To Review/set the security (vCenter connection)
1) Login the vSphere client directly to the vCenter
2) Click on a virtual machine.
3) Click the Permissions tab
4) Verify that the Windows based user (or Windows group the user is a member of) is listed and given the "Administrator" role
5) If the user is NOT listed make a note of the entries in the "Defined in" column for existing users
6) Select the object in the left hand pane that matches that seen in the "Defined in" column against the existing users and click the Permissions tab for this object
7) Right click in blank space on the permission screen and select Add Permission
8) Use this screen to add your user (or group) and make sure you set the "Assigned Role" to "Administrator" and that "Propagate to Child Objects" is enabled. Note: You can select from both Domain and Local (vCenter Server) Windows users at this point.
9) Repeat steps 2 though 4 to verify that the specific Windows user (or group) is now listed with the Administrators role.
10) Make sure this user is also used in the backup selection list.
11) Additionally verify that at a Windows level this same user has local administrator permissions over the Operating System of the vCenter server
Note: If you set the permissions too low in the object tree you won't have permissions to the datastores that hold the virtual machines
 

Chris_Rudin
Level 3

Even after updating the security the issue remains.  1 Virtual machine backed up incrementally, the rest spit out:

V-79-57344-38324 - Incremental\\Differential backup methods are not supported for this VM. The bac...


The error message really doesn't give me anything to go on.

I can see in the VM log that it did snapshot last night:

Jul 20 01:12:12.386: vmx| SnapshotVMX done with snapshot 'SYMC-INCR 19-07-2010 20:11': 26

But I see these in the vpxd.logs


[2010-07-19 20:12:41.267 04692 info 'App'] [VpxLRO] -- ERROR task-internal-76910 -- vm-1528 -- vim.VirtualMachine.queryChangedDiskAreas: vmodl.fault.InvalidArgument:

(vmodl.fault.InvalidArgument) {
   dynamicType = <unset>,
   faultCause = (vmodl.MethodFault) null,
   invalidProperty = "changeId",
   msg = "A specified parameter was not correct.
changeId",
}

This seems to be an error that is preventing the deltas from being built to be backed up.

Again, full backups run to completion so I really doubt this is a 'security' issue.

Chris_Rudin
Level 3

I found this on vmware's site:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021607



Reverting to a pre-existing snapshot can cause incremental backups based on Changed Block Tracking to become inconsistent

Symptoms

Reverting a snapshot for a virtual machine that has Changed Block Tracking (CBT) enabled to a snapshot older than its last incremental backup can cause inconsistencies in incremental backups of that virtual machine.
Note: Backup products that take advantage of CBT may be affected. Check with your backup vendor.
 
The following scenario must occur, in sequence, before there is a possibility of this issue occurring:
< >A virtual machine with hardware version 7 must have a snapshot present and must have been previously backed up by a backup product leveraging CBT.A backup product performs an incremental backup of the virtual machine and leverages CBT to determine changed blocks since the last backup.After the incremental backup is complete, you manually revert a snapshot on the virtual machine.A backup product performs an incremental backup of the virtual machine and leverages CBT to determine changed blocks since the last backup.

Resolution

If you need to revert a snapshot of a virtual machine that is being backed up by backup application that uses CBT to a snapshot older than its last incremental backup, you need to initiate a full backup of that virtual machine before using incremental backups again.
 
To workaround this issue, perform a full virtual machine backup after a snapshot revert operation. If the backup application does not allow this as an option, you need to remove the CTK files for that virtual machine. The CTK files are stored with the virtual machine on the datastore and can be removed via the Datastore Browser. This delete operation can be safely done while the virtual machine is running.

I tried this one VM, will see what happens.

MTodorov
Level 2

I have the same problem - vSphere 4.0, all machines are version 7. Differential backup is done only for the first machine, all the others fail with:

V-79-57344-38324 - Incremental\\Differential backup methods are not supported for this VM. The backup job failed, since the option 'to fall back to a full backup' was not selected.

Did you find a solution for this problem?

I will try to reorder selection list and see if differential backup will be done for some other machine if that machine is first on the list.

Chris_Rudin
Level 3

After about 60 days worth of tech support, the case was closed with something like: "Fix in progress from Symantec"

Kind of surprising that this bug wasn't found as it happens right out the box.  The work around was to backup using selection lists that only have a single virtual machine in them.  I did this for the key machines and left the ones where I am not concerned about incrementals in another list that are backed up as full images.

Still waiting for a patch.

DavideDG
Not applicable

Hello,

just to add my comments: I have the very same problem at a customer's site.

According to Symantec tech Support, the problem arises when there are selection lists with more than 1 VM, so the workaround was the same: use multiple selections with just 1 VM in it.

But, since we have 25 VM, this means 50 Backup Jobs (full+diff) to handle and, much worse, since we backup directly to a tape library, this means that the backup times have grown due to robotic operations (load, unload, rewind/verify).

Backup Exec version is 2010-R2 fully patched.

And still waiting a patch (that I was told was already being worked ... in early august).

Regards.

Jim_S
Level 4

I'm having this issue also, among the MANY other issues I'm having with BackupExec 2010 R2 using AVVI. 

JT_T
Level 5
Partner Certified

http://www.symantec.com/docs/TECH140486

As you can read from the link's article, this issue is something that Symantec might or might not fix.

WTF? Customers are buying BE to use it with VMware environments, but Symantec doesn't give a sh**t about it.

Workaround? Yes there is but use it with more than 5 VM's? No way.

 

 

BTW, do you have vSphere Server installed on BE Server?

 

Edit: Fixed link

Jim_S
Level 4

I agree that the workaround is horrible and not a real workaround in any sense of the word.  Symantec is truly sucking when it comes to the AVVI agent.  I'm 100% not impressed and am considering just shelfing it and going back to Windows agent backups inside the VMs since it actually works consistantly.

Symantec really needs to get their ducks in a row on this AVVI product.  They are still selling it as some miracle solution, and it just doesn't work as advertised.  It also doesn't work as documented.  And the documentation sucks to boot.  I am an UNHAPPY CUSTOMER.

 

My vCenter server is completely separate from my BE media server.

bleachlpb
Level 2

I am having the same problem, and it is especially frustrating when I see that this has been happening with other folks for almost a year, and it isn't fixed. 

Regardless, I am hoping that this problem gets resolved.  My environment is similar to everyone else's: 

  • VMware 4.0 and all VMs are HW7. 
  • ctkEnabled setting is there and set to true - this is set automatically when I push RAWS to my VM. 
  • VM is running Windows Server 2008 R2
  • BackupExec server running on Windows Server 2008 R2.  This server received Windows patches most recently in mid-December.
  • BackupExec server, and agents are fully-patched up to current via LiveUpdate
  • Checked permissions for my Backup Exec service account, it has Administrator rights throughout my VMware environment. 
  • I can see the ctk files in the directory for my VM in the datastore OK. 
  • I only have 1 VM per backup selection list
  • I am using a policy for full/diff VMware backups
  • The full/diff options are being set under VMware in the job properties, not under the standard/general tab (for file-level backups). 

Anything I'm missing??

Preaching to the choir, but having diff/incr features would be extremely useful and save a LOT of storage for my daily backups.  Which makes it that much more aggravating that this doesn't work.  Add me to the list of unsatisfied customers.

I should have done my research before I upgraded.  I suspect that I will be looking into the Avamar solution shortly if Symantec doesn't get their act together. 

Lars_Oeschey
Level 3

This really is weird. While running test with BE2010, I had some VMs backuped. Everything worked fine, even with differential. So we bought the product.

Then, for some days we only had two machines in the backup set, everything still was fine. Then I added all the machines that need to be backup'ed, now only the two machines would backup, but none of the newly added. I suspected that maybe it's a problem with ctkEnabled not being set, so I deleted the jobs from the policy and recreated them. Now no backup at all works, even the two machines that worked before don't backup any more.

I can't tell you how much that piece of crap annoys me, let alone what we paid for 21 ESX licenses. I'm thinking about recommending legal steps to my employer, who is one of Europes largest automobile manufacturer.

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

For all customers in this thread who are having problems with Incremental VMware backups, there is a limit to what research we can do with you via a public/community supported forum - as such if you are having problems please log a formal Technical Support case.

Note: Any support cases will be handled quicker if you are already using Backup Exec 2010 R3 and still have the problem as our engineering and development teams are now concentrating on 2010 R3 and future versions.

Lars_Oeschey
Level 3

of course I would use support, if your support system would work. The Mysupport site wants to verify my contact, and doesn't accept the technical Support ID of ours. Having gone through phone support once, made me never do that again, because although my english is not perfect at all, I still prefer a phone partner that I can understand at least. So, we still have two "fail" here.