Forum Discussion

Cursed_Network's avatar
Cursed_Network
Level 2
10 years ago

Backup Exec 2014 Hyper-V hangs SBS 2011 server

I have three physical servers in my environment:

1. SBS 2011 Server with Backup Exec 2014 SBE  (fully patched 14.1 Rev 1786)

2. 2012 R2 Server hosting virtual machines

3. 2012 R2 Server hosting replicas

We backup to external USB hard drives connected to a powered USB hub.  I have no problem backing up SQL, Exchange, files, etc.  However, when I try to backup a virtual machine (whether a replicating one or not), the following happens:

1. The VHD of the virtual machine appears to complete and is visible on the external drive

2. A virtualmachine-AutoRecovery.avhd file is created along with two corresponding XML files

3. The backup hangs backing up one of the Virtual Machine's XML files (e.g. 9C10E96E-7F11-4F6A-BDCF-FED56587044A.xml)

Now here's where things get ugly and weird:

4. Shortly after (< 1 min) the server comes to a screeching halt.  CPU and network activity go to zero - the system is unresponsive and needs to be power cycled.

5. If the system reboots with the external hard drive still attached, Windows will hang on bootup with a black screen and mouse cursor.

6. If the drive is reconnected after Windows boots, the SBS 2011 server hangs again.

7. The Hyper-V server indicates the virtual machine is still being backed up, and the Hyper-V Management Service needs to be killed and restarted.

So far, I have tried:

1. Different manufacturer USB drives (WD and Toshiba)

2. Different USB host controler (server's USB 2.0 and a USB 3.0 add-in card)

3. Different powered USB hubs

4. External drive is excluded from AV (Vipre Business Premium) scanning

5. Every BIOS, driver, and firmware update I can find for both physical servers

Normally, I would think the AV process was the culprit, but there's nothing in the Active Protection, Application, or System Logs.  We're also using a Hyperoo client to backup the virtual machines to the cloud, but this runs on the Hyper-V server directly: not the SBS 2011 server.  The cloud backup has no issues.

2014
Backing Up
Backup and Recovery
Backup Exec
Troubleshooting
Windows Server (2003-2008)

4 Replies

Sort By
  • Cursed_Network's avatar
    Cursed_Network
    Level 2
    10 years ago

    I was able to use the Windows Backup on the Hyper-V server to create a backup to a share on the SBS server without issue, so the problem doesn't appear to be with the virtual machine.  Also, the Hyper-V services on the SBS box are disabled.

    • bvj's avatar
      bvj
      Level 0
      9 years ago

      Same/similar problem using Backup Exec 15 revision 1180 Feature Pack 4 on Windows Server 2012 R2 fully patched. 

      The following log entry repeated for the duration of the hang. Didn't start capturing prior to the hang.

      2889	BESERVER	3872	8/14/2016 12:07:23 AM	4720	-1 ActiveState::doStatusEvent('192.168.11.9 Backup 00013-Incremental'): JobStatus=16 OperationStatus=1 EngineStatus=15 ActiveResourceContainerID=5F7D8ACB-8B2E-46C2-BB01-D5C6DFA9BD96 AbortStatus=0


      Stopping the BE Remote Agent, Reporting, and Event Notifier installed on the hypervisor  (192.168.11.9)  made no difference in terms of the "hung" state of the VirtualMachine XML file being processed.

      And an attempt to stop the Backup Exec services also seemed to fail after a few tries over a 10 minute interval.

      472	BENGINE	2420	8/14/2016 12:35:49 AM	1984	[server]             - Cancelling any pending device requests
473	BENGINE	2420	8/14/2016 12:35:50 AM	0	cancelling any pending device requests
474	BENGINE	2420	8/14/2016 12:35:50 AM	0
475	BENGINE	2420	8/14/2016 12:35:49 AM	1984	[server]             - DeviceManager: Canceling pending requests.
476	BENGINE	2420	8/14/2016 12:35:49 AM	1984	[server]             - Shutting down JobEngine
477	BENGINE	2420	8/14/2016 12:35:50 AM	0	shutting down JobEngine
478	BENGINE	2420	8/14/2016 12:35:50 AM	0
479	BENGINE	2420	8/14/2016 12:35:49 AM	1984	[server]             + backupjob.cpp (3347):
480	BENGINE	2420	8/14/2016 12:35:49 AM	1984	[server]             | Attempting to abort active BSD.


      After reenabling RAW services on the hypervisor, BE detected the agent, and then proceeded to shutdown. Maybe a zombie connection to the hypervisor confused BE.   Anyway, I have a log of the activity should Veritas be interested.

  • Cursed_Network's avatar
    Cursed_Network
    Level 2
    10 years ago

    Update:

    Tried removing antivirus - no change

    Reinstalled agents on Hyper-V host and virtual machines - no change

    Selected only VM's and no host components - no change.

    I did notice a strange shuffle right before the hang:  My server indicated that the USB hard drive bounced and was remapped as two drives.  As the server was grinding down, it still showed the drive mapped to a single drive letter.

    I suspect the backup software is attempting some sort of permission write that's going awry, which would explain why the disk crashes the system after it's reattached.  If I format the drive on another machine, I can use it again and Backup Exec recognizes it right away.