Quite frequently we see mapped full VM backups that run into problems when parsing active filesystems on our Linux guests.
What typically happens is that a VM backup job will throw a VFM error 3, usually while processing /var, and then simply stops reading data. When this happens, the backup job will never exit and stays running until killed.
The log will show errors like below:
8/25/2011 10:40:46 AM - begin writing
8/25/2011 10:43:39 AM - Error bpbrm(pid=5208) from client vmxxx.fq.dn: ERR - Unable to read metadata for index: 128353, VFM error = 3.
We are pretty sure that this is caused by inconsistencies in the file system - there is no reliable quiesce mechanism for Linux after all - but I consider it to be a bug that the ext3 handler simply stops processing instead of either skipping that file, or marking the job as failed.
But since we are seeing this fairly frequently I figure there may be other customers experiencing this issue as well. It seems to affect primarily Linux VMs, and ones that tend to have a lot of write activity such as a mail spooler or database node.
Anyone experiencing this problem that feel like chiming in with their experiences and possible workarounds?
(What we've done so far is to move known bad VMs into a different policy that does not perform file-level backups, eg a Full VM instead of Mapped Full VM).
