Backup Exec 2010 R3 services stopping mid job

[Water_Street_So]

We are using Backup Exec 2010 R3, build 5204 64bit on a 2008 R2 server.  The way our backups normally work is they are backed up to disk first and then duplicated to tape to reduce down time.  The stuff we are backing up are a mix of physical servers and virtual servers running on vcenter 4.1, all VMs are 2008 R2, most physical are a mix of 2003 or 2003 R2 servers.  We also have the Exchange plugin which backs up our Exchange 2010 VM.

For a good 2-3 months, possibly even before we moved to R2, we've been having issues. Where once every week or every two weeks the Backup Exec services will stop on the server in the middle of our jobs and you'll see a string of failures.  A reboot will fix it again for another week or two, but it's guaranteed to occur again.

Looking at the most recent occurance this morning, the application log has just random informational things, most of which not from backup exec. As soon as the first job runs I see a warning of it 'failed to mount one or more virtual disk images.  Restores that use GRT may not be available from that backup set.'  Followed by a second for a different server and then an error stating 'Process beremote.exe requested memory dump but the maximum number of memory dumps has already been reached.'

Followed by a second error

Faulting application name: beremote.exe, version: 13.0.5204.0, time stamp: 0x4d976a80
Faulting module name: VirtApi.dll, version: 2.0.65.0, time stamp: 0x4d6d500f
Exception code: 0xc0000005
Fault offset: 0x000000000001f163
Faulting process id: 0x133c
Faulting application start time: 0x01cc47a9e22217ae
Faulting application path: C:\Program Files\Symantec\Backup Exec\beremote.exe
Faulting module path: C:\Program Files\Symantec\Backup Exec\VirtApi.dll
Report Id: 6f55e2ef-b71c-11e0-84ee-001c23d758c0

After which you start to see the job failures after two informational generic windows 'appcrash' notifications.

Seems to be the only issue with our BE deployment right now is the fact that it just likes to stop mid stream right now every so often.

[RahulG]

virtapi.dll is a signed dll, maybe you have some problems during the verify of the certificate.

check the eventlog for certificate related entries

1.) you have ipv6 active ( obviously with sbs) have you modified your hosts file according to
   http://seer.entsupport.symantec.com/docs/306689.htm
2.) i couldnt find it in the log, but disable the Advanced open file option when you backup Exchange,SQL, .... So you need to create a specfic job to create a backup of a vss aware application

3.) NAS: some NAS have problems with very large files, typically are this borders at 2 and 4 GB. Please verifiy it.

4.) try to use the same account for backup the exchange server which you use as system logon account.
   It is suggested to use only one account for the whole backup purpose.

5)Check antivirus exclusions

6) try updating the remote agent

[Water_Street_So]

I have verified the following was done prior before the most recent failure.

1. ipv6 is not active on the BE media server

2. Advanced open file option is disabled for all VMs including our SQL and Exchange servers, both of which are VMs.

3. No NAS, only a SAN for centralized storage with our VM information stored there, all of which are formated as VMFS.  The SAN is an Equallogic PS5000e SAN with the latest firmware.

4. Same account is being used.

5. No AV located on the BE media server, only on file servers.

6. All updates have been applied to remote agents and the BE media server

[Water_Street_So]

I should also add that the only servers that are physical are 1 domain controller, 1 file server, 1 web server and then the backup exec media server.  Everything else is virtualized in our environment.  The physical machines have the Advanced Open File Option enabled.  An VMs have it disabled, but GRT is enabled.

[Larry_Arndt]

I've been having the same problem since I upgraded to BUE 2010 R2.  I found documentation that said the problem was fixed in 2010 R3, so we upgraded.  The problem still exists.

  Every other day or so, the Backup Exec Job Engine Service fails, subsequently, all the jobs fail.  I have to reboot the server to get the service to come back online.

  More recently, I'm getting the following error in the event log (probably because it's failed so many times)

EventID 258: "Process bengine.exe requested memory dump but the maximum number of memory dumps has already been reached."

Also

EventID 1000: "Faulting application name: bengine.exe, version: 13.0.5204.109, time stamp: 0x4e18e5ec

Faulting module name: ndmpcomm.dll, version: 13.0.5204.109, time stamp: 0x4e18e5ca

Exception code: 0xc0000005

Fault offset: 0x0000000000025a89

Faulting process id: 0x948

Faulting application start time: 0x01cc5559dc0c1a6c

Faulting application path: C:\Program Files\Symantec\Backup Exec\bengine.exe

Faulting module path: C:\Program Files\Symantec\Backup Exec\ndmpcomm.dll

Report Id: c9252f56-c306-11e0-8cc9-00215ad4fb25"

[Jomy]

check where your job is getting failed.

is it on same file or same type of files.

warm regards

[Water_Street_So]

Jomy, it's not the same job that fails and it's not at a specific file that I've noticed.  It's not the service on the client side that is failing, it's a service on the BE server side that's failing and then causing all jobs to either fail or miss their window.

[sbora]

Water Street Solutions,

Remote agent is crashing in virtapi.dll whiich is part of Backup Exec VFF (virtual file filter) used for GRT backups. An updated version of these binaries will be released in an upcoming hotifx which may also help resolve your issue. I will recommend you to contact support to get pre release version of this fix and check if it resolves your issue. If it doesnt then we will have to collect crash dump and debugs from your setup to further investigate the issue.

[Carlos_Quiroga]

A first approach to try to solve your issue, would be to check the Job Log in detail, to see where exactly the services crashed.

Knowing the exact time of the failure, identify in the logs which resource was being backed up at that hour.

This may reveal wether your problem is with a particular virtual or real server, and with which one.

You may also then try to create an individual backup job for that specific server or resorce, to verify if the problem repeats itself.

Another troubleshooting tool to use can be to enable the debug logs, for some days.

On the Backup Exec Media Server, on the program directory \program files\symantec\backup exec, find a program named BEUtility.exe, open it, select Media Servers, right click on the name of the media server and choose Enable Debug Logs. This will generate two files on the Logs directory on the program folder mentioned above.

You can then run your backups normally, and when the issue happens again you will have a logged detail in two files named beremoteXX.log and bengineXX.log that you may submit to us for checking.

On each remote server also, go to start - run - services.msc, locate the Backup Exec Remote Agent Service and on its properties, stop the service, insert -debug on the Start Parameters textbox, and start the service again. This will add an additional log file in the \program files\symantec\raws directory.on the remote server named beremoteXX.log.

You can refer to the following technical article that describes enabling logs and collecting data:

How to enable or disable "debug logging" in Backup Exec for Windows Servers
http://www.symantec.com/docs/TECH23853

For virtual remote servers in particular, there are some additional steps for debug logging described in this other article:

How to enable debug logging for Backup Exec Agent for VMware Virtual Infrastructure (AVVI)
http://www.symantec.com/docs/TECH63926

The comments above are surely not the solution for your issue, but they comprise the necessary steps to start shedding some light on your problem.

Regards,