Backup Exec Puting Hyper-V VM into Saved State

[tss_tim]

Hello,

We are running Backup Exec 2010 R3 on a Windows Severer 2008 R2 server. We backup our Hyper-V machines using multiple Hyper-V policies (one a night to distribute the backups throughout the week). During the Monday backups one of our Exchange 2010 DAG members to go to saved state when ever the backup runs. This obviously causes the databases that are running on that server to fail over to the other DAG member, whilst this is not noticeable to most users I don't want to have my DAG fail-over weekly because of backups.

I found the option within the policy that says 'Exclude virtual machines that must be put in a saved state to backup up'. This has obviously fixed the problem of the machine going to saved state, but means that the machine is not being backed up.

I have checked the following Symantec documents:

http://www.symantec.com/business/support/index?page=content&id=TECH126115
http://www.symantec.com/business/support/index?page=content&id=TECH200150

I can confirm that this server meets all of the requirements to be able to do an on-line backup:

• Hyper-V SCVMM 2008 R1 tools may cause this issue. 

SCVMM 2008 R1 was never installed in this environment and SCVMM 2008 R2 guest services are installed on all of our VMs.

• The Hyper-V integration services are installed and the VSS integration service is enabled

The Hyper-V backup integration service is installed and running:

• All virtual machine disks are configured within the guest operating system as NTFS-formatted basic disks.

All of our disks are basic and NTFS:

• Each virtual machine volume must specify itself as the storage location for its shadow copies.

If I search for the shadow storage location when a backup is not running then I get "No items found that satisfy the query.". I have tried redirecting the VSS shadow storage to another drive and when that configuration is in place it shows the alternate storage location. So I take it that by returning no items found it is using the default configuration which is to be on the same drive.

I have also been through the following Microsoft MSDN article: http://msdn.microsoft.com/en-us/library/dd405549(v=vs.85).aspx

It essentially outlines the same points:

• Backup (volume snapshot) Integration Service is installed and running in the child VM. The service name is "Hyper-V Volume Shadow Copy Requestor".

  Windows 2000:  Backup Integration Service is not supported.

• The child VM must be in the running state.
• The Snapshot File Location for the VM is set to be the same volume in the host operating system as the VHD files for the VM.
• All volumes in the child VM are basic disks and there are no dynamic disks.
• All disks in the child VM must use a file system that supports snapshots (for example, NTFS).

Our machine meets all of these requirements.

I have also been told that you require 2GB of free disk space for this to work, of which all of our drives have.

Additionally we have the other node of the DAG, which is configured identically. The other server backups up successfully.

I have opened a case with Symantec about this but their only solution to the problem was to untick 'Exclude virtual machines that must be put in a saved state to backup up'. After I told the technician numerous times that I would not untick that option he fobbed me off by saying install Backup Exec Service Pack 4. Whilst it may be a good thing to install, I highly doubt that this is going to resolve my problem, and according to the tech I will have to update every Backup Exec client which will require a reboot, so it isn't something I can do quickly.

I am hoping that someone out there can help me with this problem.

If I have missed anything off this please let me know.

Regards,

Tim

[Ramprabhu]

Could you tell me the configuration of your virtual machine OS: Service Pack and built:

[tss_tim]

Hi Ramprabhu,

The VM is running Server 2008 R2 Enterprise SP1. It has 16GB and a dual core CPU @ 2.67GHz.

On a side note, I was playing around with the job on Monday and moved the Exchange sevrer to be the first in the resource order and the backup finished successfully. I am going to wait until the scheduled job runs on Monday, but testing has showing that the resource order may have resolved the issue.

Does anybody have any ideas as to why the order of the backup might affect if a server is put to saved state or not?

Tim

[Ramprabhu]

Hi Tim, 

Can you please check the following

The VM guest needs to have Integration Services installed, enabled, and running (COM System Application Service, Distributed Transaction Coordinator Service, and Volume Shadow Copy Service). Also review the VM settings in Hyper-V, the 'backup' option needs to be checked.

Please check this and let me know if it is helpful

Thanks,

Ram

[lmosla]

Make sure that there isnt any maintenance or utility jobs running during the job and that if there is any antivirus running on the servers the following processes have been excluded from the scan:

C:\Program Files\Symantec\Backup Exec\beremote.exe

C:\Program Files\Symantec\Backup Exec\beserver.exe

C:\Program Files\Symantec\Backup Exec\bengine.exe

C:\Program Files\Symantec\Backup Exec\benetns.exe

C:\Program Files\Symantec\Backup Exec\pvlsvr.exe

C:\Program Files\Symantec\Backup Exec\BkUpexec.exe

C:\Program Files\Symantec\Backup Exec\RAWS\beremote.exe  

and the Backup Exec temp directory C:\Temp\Backup Exec on the Exchange server 

[tss_tim]

Hi Imolsa,

I have checked our anti virus exclusions and all servers have an exlusion for the executable 'beremote.exe' and 'benetns.exe'

To ensure that it is not the antivirus that is possibly causing the problem I have added all the exclusions that you have listed, even the 'C:\temp\backup exec', even though the Exchange server doesn't have a temp folder.

I have checked the event logs on the Exchange server and there was no maintenance happening at that time, however I did realise that there was an Exchange agent backup starting at the same time. I have now schedueld the Hyper-V backup to start after the Exchange backup so they are not trying to use the VSS provider at the same time.

Tim

[tss_tim]

Hi Ram,

Thanks for your reply, I have checked those services and they are all running and the backup option is enabled. I have also managed to back this machine up during troubleshooting using the same selection list and backup job but not run on a schedule, so the settings for backup are most likely correct. 

As stated in the post below, this may be a contention issue with another backup, but further testing is required.

Tim

[stevebrat]

go into the vm and make sure that the shadow copies for each drive are actually pointing to itself and not another hard disk in the VM.  i had this same issue and that fixed it.

[tss_tim]

Hi Steve,

Thanks for the feedback, as noted in the original post, I have checked that the shadow copies for each drive are pointing to themselves, and I have checked again and they are still pointing to themselves.

Tim