10-12-2015 08:11 AM
I have three physical servers in my environment:
1. SBS 2011 Server with Backup Exec 2014 SBE (fully patched 14.1 Rev 1786)
2. 2012 R2 Server hosting virtual machines
3. 2012 R2 Server hosting replicas
We backup to external USB hard drives connected to a powered USB hub. I have no problem backing up SQL, Exchange, files, etc. However, when I try to backup a virtual machine (whether a replicating one or not), the following happens:
1. The VHD of the virtual machine appears to complete and is visible on the external drive
2. A virtualmachine-AutoRecovery.avhd file is created along with two corresponding XML files
3. The backup hangs backing up one of the Virtual Machine's XML files (e.g. 9C10E96E-7F11-4F6A-BDCF-FED56587044A.xml)
Now here's where things get ugly and weird:
4. Shortly after (< 1 min) the server comes to a screeching halt. CPU and network activity go to zero - the system is unresponsive and needs to be power cycled.
5. If the system reboots with the external hard drive still attached, Windows will hang on bootup with a black screen and mouse cursor.
6. If the drive is reconnected after Windows boots, the SBS 2011 server hangs again.
7. The Hyper-V server indicates the virtual machine is still being backed up, and the Hyper-V Management Service needs to be killed and restarted.
So far, I have tried:
1. Different manufacturer USB drives (WD and Toshiba)
2. Different USB host controler (server's USB 2.0 and a USB 3.0 add-in card)
3. Different powered USB hubs
4. External drive is excluded from AV (Vipre Business Premium) scanning
5. Every BIOS, driver, and firmware update I can find for both physical servers
Normally, I would think the AV process was the culprit, but there's nothing in the Active Protection, Application, or System Logs. We're also using a Hyperoo client to backup the virtual machines to the cloud, but this runs on the Hyper-V server directly: not the SBS 2011 server. The cloud backup has no issues.
10-12-2015 10:16 AM
I was able to use the Windows Backup on the Hyper-V server to create a backup to a share on the SBS server without issue, so the problem doesn't appear to be with the virtual machine. Also, the Hyper-V services on the SBS box are disabled.
10-14-2015 11:46 AM
Tried removing antivirus - no change
Reinstalled agents on Hyper-V host and virtual machines - no change
Selected only VM's and no host components - no change.
I did notice a strange shuffle right before the hang: My server indicated that the USB hard drive bounced and was remapped as two drives. As the server was grinding down, it still showed the drive mapped to a single drive letter.
I suspect the backup software is attempting some sort of permission write that's going awry, which would explain why the disk crashes the system after it's reattached. If I format the drive on another machine, I can use it again and Backup Exec recognizes it right away.
11-13-2015 05:29 PM
Wow - 1 month and no posts. Guess this discussion is dead.
08-13-2016 09:10 PM - edited 08-13-2016 09:49 PM
Same/similar problem using Backup Exec 15 revision 1180 Feature Pack 4 on Windows Server 2012 R2 fully patched.
The following log entry repeated for the duration of the hang. Didn't start capturing prior to the hang.
2889 BESERVER 3872 8/14/2016 12:07:23 AM 4720 -1 ActiveState::doStatusEvent('192.168.11.9 Backup 00013-Incremental'): JobStatus=16 OperationStatus=1 EngineStatus=15 ActiveResourceContainerID=5F7D8ACB-8B2E-46C2-BB01-D5C6DFA9BD96 AbortStatus=0
Stopping the BE Remote Agent, Reporting, and Event Notifier installed on the hypervisor (192.168.11.9) made no difference in terms of the "hung" state of the VirtualMachine XML file being processed.
And an attempt to stop the Backup Exec services also seemed to fail after a few tries over a 10 minute interval.
472 BENGINE 2420 8/14/2016 12:35:49 AM 1984 [server] - Cancelling any pending device requests 473 BENGINE 2420 8/14/2016 12:35:50 AM 0 cancelling any pending device requests 474 BENGINE 2420 8/14/2016 12:35:50 AM 0 475 BENGINE 2420 8/14/2016 12:35:49 AM 1984 [server] - DeviceManager: Canceling pending requests. 476 BENGINE 2420 8/14/2016 12:35:49 AM 1984 [server] - Shutting down JobEngine 477 BENGINE 2420 8/14/2016 12:35:50 AM 0 shutting down JobEngine 478 BENGINE 2420 8/14/2016 12:35:50 AM 0 479 BENGINE 2420 8/14/2016 12:35:49 AM 1984 [server] + backupjob.cpp (3347): 480 BENGINE 2420 8/14/2016 12:35:49 AM 1984 [server] | Attempting to abort active BSD.
After reenabling RAW services on the hypervisor, BE detected the agent, and then proceeded to shutdown. Maybe a zombie connection to the hypervisor confused BE. Anyway, I have a log of the activity should Veritas be interested.