I’m wondering if anyone knows if there are any potential issues with NetBackup and Vmware’s recent KB about their Change Block Tracking bug?
It looks like some other backup products might be affected but Google find very few results as related to NetBackup.
Netbackup TECHNOTE: http://www.symantec.com/docs/TECH225910
The VMware KB article is fairly new and may have not yet resulted in any NBU Support calls.
The problem with issues like these is that the problem is only noticed at restore time.
You found this article and can therefore be proactive to prevent issues.
IMHO, if you have vmdk's that were extended and experience the symptoms listed in the doc, then the answer is YES. Any backup product will be affected as a result.
Best to talk to VM admins to check extended vmdk's and apply workaround if symptoms are seen.
So, backup and restore issues are easy enough to avoid.
Thanks for sharing the KB with us. It now on our bug watch list !
Short answer: yes, but we don't know enough to document them yet. We're working with VMware. Stay tuned!
Netbackup TECHNOTE: http://www.symantec.com/docs/TECH225910
...and in case you are noticing that this link doesn't work at the moment, we're rewriting this document and will republish it a little later this week (we expect) as a Tech Alert.
To subscribe and get notifications of Alerts as they are published, please use this link:
http://www.symantec.com/business/support/index?page=content&key=15143&channel=ALERTS
I'm curious if using the 'Accelerator Forced Rescan' option would be a workaround until an offical fix is issues (either via VMware and/or Symantec). From NetBackup 7.6 Feature Briefing - Accelerator support for Virtual Machines.pdf
...Figure 4, is an optional setting called Accelerator forced rescan. This is basically a safety net to make certain nothing has gone awry with VMware CBT. Accelerator relies on CBT to accurately provide a list of changed blocks and if something goes amiss, it’s possible some blocks might not get backed up.
Based on this, it seems 'reasonable' that this would catch all the data regardless.
Then, the question begs, let's say this 'works', does it sort of 'reset' CBT when you do this? (e.g. the next 'accelerated backup' would be fine because a new baseline has been created?) OR is it still using the 'broken' CBT information even after a forced rescan?
AK
Some info we have gotten through the partner network, you question is answered further down:
Here is the VMware article:
http://kb.vmware.com/kb/2090639
Briefly, the issue is described as follows:
During vStorage API for Data Protection (VADP) style backups of VMware VMs, when incremental backups are desired the NetBackup “BLIB” option is selected. NetBackup then automatically enables the CBT mechanism for all protected VMs. If any protected VMDK(s) is extended in size past the 128GB boundary (or any power of 2 above that – e.g. 256GB, 512GB, 1024GB) after CBT is enabled, the CBT mechanism becomes corrupt. Subsequent backups of the impacted VMs won’t accurately backup the correct (in-use) blocks of data. In many cases this is exclusively extra data (blocks) but in some cases backups can miss blocks that are actually in use. Either data within the restored VM will be corrupt or the restored VM will fail to boot.
VMware has indicated that they plan on providing a fix for all currently supported versions of ESXi. VMware has not yet announced the exact nature of this fix or exactly when this fix will become available but indications are that VMware will provide this fix in the next VMware “update” release.
Some additional information:
Q: Does this issue apply if the VMDK is extended more than 128GB?
A: Yes. This issue does not necessarily impact all VMs. The issue only occurs if a VMDK is extended past the 128GB boundary (e.g. 120GB resized to 130GB) or any power of 2 above 128.
Q: Does this issue apply if my VMDK is already larger than 128GB?
A: No. The initial size of the VMDK does not matter. What matters is if the VMDK is extended beyond the affected limits after it is initially configured and CBT has been enabled.
Q: Is this issued caused by a NetBackup bug?
A: No. This is not a NetBackup bug. This is a VMware bug. Any vendor that uses VMware’s VADP for backups can be impacted by this issue.
Q: Why doesn’t NetBackup provide a fix for this?
A: NetBackup engineering has determined that there is no way a backup vendor can provide a reliable solution that covers every contingency configuration. We’ve studied competitor workarounds for this. Their solutions cover some use cases but there are still configurations where their solutions will not accurately detect and fix the issue and, in our opinion, give the user a false sense of security. We have determined that VMware must provide a fix for this. VMware agrees with this assessment.
Q: What should I tell my customer if they request an immediate solution?
A: The only solution that will insure that this issue is fixed for every VM is to manually disable CBT. This can be performed from within the vSphere interface or a PowerCLI script can be written to do this. Once CBT is disabled, NetBackup will automatically re-enable CBT during the next scheduled backup.
Q: My customer is using NetBackup Accelerator for VMware. Will running a backup enabling the “forced rescan” option take care of this issue?
A: No. Enabling “forced rescan” will not reset or disable CBT. CBT must be manually disabled.
Q: Is there a way I can test to determine if previous backups are corrupt?
A: Instant Recovery for VMware (IRV) can be used to test if the VM image is bootable. However, this will not detect if there is corruption of data inside the VM.
Q: If CBT is reset, will I ever encounter this issue again?
A: Yes. It is possible that this issue can subsequently occur after CBT is reset. If after the CBT reset the VMDK is extended past one of the affected boundaries, CBT can once again become corrupt mandating that CBT once again be reset.
Q: Will disabling CBT force a full backup of my VMs?
A: Yes. You customer should be aware that once CBT is disabled, the next backup run will force a full backup. All of the data within the VM will be backed up regardless of whether the backup schedule is a differential, cumulative or full backup schedule.
Q: Does my customer need to log a call with VMware regarding this issue?
A: We recommend that your customer does log a call with VMware. We’ve discussed this issue with VMware and have indicated that this has the potential of creating a significant data loss scenario. Having your customer log a call with VMware will reinforce our stance on this.
What I think is a bit unclear here: Would a regular non-Accelerated full backup be a functional workaround?
VMware describes the bug as CBT and incremental backup related. So if we disable Accelerator for VMware backups and only run full backups, are we then in the clear?
Got an answer from someone else: Regular full backups without accelerator should not run into this issue.
Of course sub-sequent incrementals are still affected and running full backups every day is not an option for most, so you still need to reset the CBT tracking to be safe:
http://www.symantec.com/business/support/index?page=content&id=TECH197311
