There is no RDM configured for any VMs.
Every VM is stored as vmdk on a VMFS 6 Datastore and Disks are in "Dependent" mode.
To make things clear:
The issue isn't limited to a couple of VMs. It happens on all of them, occasionally. Some Jobs will work for weeks without any issue, then fail one day, sometimes 2 in a row, then work the next day. I had 3 out of 26 virtual backups fail between yesterday and today.
Every VM has its own Backup Job. All backups go to one of two disk based storages, it doesn't seem to matter which one.
Also forget to mention, they all seem to fail at the END of the Backup. On a Full Backup the failed ones show pretty much the same runtime / bytes copied as the succeeded ones.
It looks like I solved it. Haven't had any failed backups over the last 2 days.
The issue seemed to have been a registry change I made back in February of 2016.
I set values for SAN/10GB NBD according to this article:
https://www.veritas.com/support/en_US/article.TECH185691
But couldn't see any difference in speed, so I bumped the Size of Read Buffers and Size of Write Buffers keys up to 16392 a month later. Again couldn't see much change, but it didn't have any negative impact at the time, so I left it.
Dropping it back down to 8196 and restarting BE services seems to alleviate my current issue though.
Still weird how this was OK for almost 2 years.
