I've finally got all my backups working correctly EXCEPT a legacy drive inside a VMware guest server. I had excluded this single VMDK from the nightly backup job and have set up a single one-time backup of this drive as a seperate job. However I cannot get the one-time job to succeed.
The job runs for the entirety of the 11.7TB of data inside that 12TB drive but results in the error I included below. The job takes about 2 days to complete unsuccessfully.
Currently the job is setup to use the same storage space as the pre-mentioned nightly backup job of this guest VM's other drives. So I know this isn't a permissions issue. I'm assuming as well, since I simply copied the nightly job and then unselected the drives from that job and added the single 12TB drive that the configuration is likely not the issue.
Backup- VMVCB::\\<vmguest>.example.com\VCGuestVm\(DC)ha-datacenter(DC)\vm\ VMGUSET-79-57344-38366 - VDDK-Warn: VDDK_PhoneHome: HTTP response error in VddkVacCurlHttpsPost with http status 400 at 1058. VDDK-Warn: VDDK_PhoneHome: VddkVacPushData : Failed to post data to VC/PH at line 1022. VDDK-Warn: VDDK_PhoneHome: VddkVacPushVendorProductData : Failed to push vendor product data at line 1326. VDDK-Warn: VDDK_PhoneHome: VddkVacWriteMember : Empty field to serialize for key edition at line 188. VDDK-Warn: VDDK_PhoneHome: VddkVacPushProductInstanceData : Failed to push product instance data at line 1219. VDDK-Warn: VDDK_PhoneHome: VddkVacPushEnvironmentData : Failed to push env data at line 1136. VDDK-Warn: VDDK_PhoneHome: VddkVacPushSessionData : Failed to push session data at line 1709. VDDK-Warn: VixDiskLib: VixDiskLibIsLegacyConnParams: the instance of VixDiskLibConnectParams is NOT allocated by VixDiskLib_AllocateConnectParams. The new features in 6.7 or later are not supported. VDDK-Warn: HostAgent is not a VirtualCenter. Error 20 at 3710. VDDK-Warn: VDDK_PhoneHome: Invaild transport mode used at line 271 VDDK-Warn: VixDiskLibProvider::NameBasedProvider::Get: Unable to open disk \\UNC\PATH\TO\STORAGE\IMG000027\GUESTVM_1-000002.vmdk, openflags = 4 - VixError 0x3e86. VDDK-Warn: DiskLibProvider_GetDisk: Open failed - index 0. VDDK-Warn: ERROR 2 opening disk 0. VDDK-Warn: VixDiskLibProvider::NameBasedProvider::Get: Unable to open disk 1:\IMG000027\GUESTVM_1-000002.vmdk, openflags = 4 - VixError 0x3e86.
The line "The new features in 6.7 or later are not supported.", seems of note as the GUESTVM I'm trying to backup is on a ESXi Server 6.7. But again, that same GUESTVM does backup correctly if I exclude that one 12TB vmdk. My backup exec server is on an older ESXi 5.5 host running a Windows Server 2012R2 guest. The GUESTVM I'm backing up is a Windows Server 2016 install.
I notice that in the \\UNC\PATH\TO\STORAGE location when the job starts, it creates 8 GUESTVM_1-000002 vmdk that look to split. The first VMDK reaches 1.98TB and then it starts filling the second. This one ends up around 700GB when the job ends.
GRT Enabled/Disabled makes no difference. At this point anything over 1TB is quite likely to fail. Even my weekly Full backups of current data which is around 1.7TB fails 4 out of 5 times.
The legacy data I'm trying to backup (which is again 12TB) has failed 100% of the time. The most I've ever gotten was 2.97TB before failure. The failure is always "An unexpected network error occured". I've looked and looked and cannot find anything happening on my network that could potentially impact this. However it becomes increasing difficult to track when a backup takes almost a week of continuous traffic. A debug log of this job would be astronomically large in debug files.
It seems my unexpected network interruption is around 2:30 am every day. However I still can't figure out what is causing the issue. The backup server does nothing around then, nor does the NAS i'm backing up to. Between the two devices is Backup Exec <-> Ethernet <-> Switch <-> Fiber <-> Switch <-> Ethernet <-> NAS.
I'm going to attempt to try this on an older NAS that is locally within the same building and see if removing some of the network hops makes a difference. Backup Exec <-> Ethernet <-> Switch <-> Ethernet <-> NAS
I have found the culprit of the 2:30am breakage and resolved it. However that just means I'm back to my OP.
I find it incredibly odd that ALL my backups work except this one. My last attempt got a little over 9TB backed up (based on the filled VMDK files at the destination and the backup set result. After that it broke again with the OP errors. Why do my other backups work and not this one? They are all targeting a ESXi 6.7 host :(
Honestly, it does make me feel better I'm not the only one out there pulling hair out though over this.