beremote.exe crashes on ntdll.dll or MSVCR80.dll

[nPc]

Hello,

We're running BE 2010 R3, currently up to date with hotfixes, on a Win2008 stand-alone server, backing up to disk. (then dup to tape).

Our current issue is that the beremote.exe service crashes on the Media Server, faulting either the ntdll.dll or the MSVCR80.dll.

This occurs when backing up a certain set of vCenter virtual machines, all running IIS; these are typically Win2003 (there are two Win2000 that will throw this error as well). And I do have one instance of a Win2008 VM crashing on the MSVCR80.dll (also running a web server, our intranet.)

The .vmdk files are being backed up through VMware vCenter.  Various different error codes appear (typically "lost communication with Remote Agent" or similar), but always the beremote.exe has crashed moments before, with Event ID 1000 in the Applications Log.  

Here are two typical Event Log records:

---

Faulting application name: beremote.exe, version: 13.0.5204.125, time stamp: 0x4f39758e
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000005
Fault offset: 0x00000000000532d0
Faulting process id: 0xdac
Faulting application start time: 0x01cd3ac6902ac170
Faulting application path: E:\Program Files\Symantec\Backup Exec\beremote.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 0f1115e7-a70b-11e1-836e-0015171e643a

And

Faulting application name: beremote.exe, version: 13.0.5204.125, time stamp: 0x4f39758e
Faulting module name: MSVCR80.dll, version: 8.0.50727.6195, time stamp: 0x4dcdd833
Exception code: 0xc0000005
Fault offset: 0x000000000001e2c9
Faulting process id: 0xe18
Faulting application start time: 0x01cd3eb0fc6daac4
Faulting application path: E:\Program Files\Symantec\Backup Exec\beremote.exe
Faulting module path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_88e41e092fab0294\MSVCR80.dll
Report Id: a1524a92-aaf7-11e1-836e-0015171e643a

---

The servers did not all have RAWS installed, most did, although I don't think that was an issue.  Just in case I have now updated all of the VMs in question to the current version of RAWS to eliminate that as a possibility. A similar crash occurred on a job last night, hours after I had finished installing RAWS on the hold-out servers and rebooting them.

Suspecting that GRT was an issue (as that has actually been causing e0009585 errors on "random" other machines in recent weeks), I have turned GRT off on the jobs in question.  Yet last night (GRT off) beremote.exe crashed again (the second error log above).

Alas, I have not had the SGmon debugger running during these past couple of days, but I do have debugger log info for a 5/19 crash of the same nature.

Has anyone else seen something similar?  I've scoured the KB to little avail. Similarly there is little available info on either of those .dll's and what they might be doing to cause a fault.

Any suggestions or directions appreciated.

Thanks.

 .

 Neil

[navintah]

Hello Neil:

I am sending you a seperate email mesage to send logs. This would help us in investigating the issue.

Thanks

Navin

[Sush---]

Also refer to these technote

http://www.symantec.com/docs/TECH189437 : Backup Exec 2012 remote agent service (BEREMOTE.EXE) terminates in Faulting module MSVCR100.DLL

Make sure to subscribe to it too.

Thanks,

-Sush...

[pkh]

The user is using BE 2010, not BE 2012

[Jirka]

Faulting application name: beremote.exe, version: 13.0.5204.125, time stamp: 0x4f39758e
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000005
Fault offset: 0x00000000000532d0
Faulting process id: 0x5e8
Faulting application start time: 0x01cd47bac6189764
Faulting application path: C:\Program Files\Symantec\Backup Exec\beremote.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: b194c53f-b4d1-11e1-90a7-000533641566

This occurs when backing up VM

vCenter 5.0.0 on Windows 2008 R2.
ESXi 5.0.0.
GRT - enabled
Application GRT - disabled

Have you any answer from Navin or other users?

Thanks,

Jirka

[nPc]

In fact yes. Navin connected me with engineers working on this, apparenlty there's an "orphan" fix for this issue.  I haven't been able to test it yet because my error typically (although not always) occurs over the weekend and I'm waiting on confirmation of a question relating to implementation of the fix.  

 .

 Neil

[Meindert]

Hi gentles,

I'm running into exactly the same problem, hitherto I've only had crashes on ntdll.dll but same symptoms. I'm trying to backup VMs through a Policy. Funny thing is, once I manually restart the backup-exec service on the backup server and retry one of the jobs, it runs. Could it be that it's because several jobs try to start at the same time? We have it configured so that only two can run at any given time, but there's about 20 jobs in the policy. 

I'd be happy to send any logs or whatever if it helps. Is there any workaround that I can apply for now?

Addendum: I've narrowed it down to one specific webserver. VMware seems to make the snapshot, but after that something fails. 

[Jirka]

Hello,

Beremote.exe now faulting every day and we haven't any actual backup. Faulting beremote.exe causes corruption many other running jobs.

In debug log are thousands of this lines:

[3096] 06/19/12 23:35:02 [fsys\shared]        - vddkWrapper::WriteDiskExtents() Buffer is not empty...
[3096] 06/19/12 23:35:02 [fsys\shared]        - vddkWrapper::WriteDiskExtents() Buffer is not empty...
[3096] 06/19/12 23:35:02 [fsys\shared]        - vddkWrapper::WriteDiskExtents() Buffer is not empty...

Debug log end wit this information:

[4924] 06/19/12 23:44:16 [fsys\shared]        - GetNextDiskExtent(): End of disk

I thing about upgrade to BE 2012 but when I reading this thread:
https://www-secure.symantec.com/connect/forums/backup-exec-2012-service-pack-1a-released 
I'm afraid to do it. This is horrible story!

Jirka

[JohnJohn]

Hi @ll,

We have the same problem as you, have you since found a solution?

[Meindert]

Nothing yet as far as I know. Workaround was to just backup the data and not the VM, but I'm not happy with it. Anyone know if Symantec has made any progress? I haven't been contacted for any logs, so I guess they have enough info to work on a solution.

[nPc]

Hey folks,

I was put in touch with an engineer who offered a vddkWrapper.dll 'orphan hotfix' to replace the stock .dll on the servers that were triggering the crashes.  This seemed to work for a few weeks after I installed it on 8 machines.  Then we had a crash of a similar nature on one of the machines with the wrapper faulting the ndmpsrvr.dll on June 22, then a weekend of quiet, then 2 more ntdll.dll crashes on servers that had received the wrapper.dll.

Since we'd already been experimenting with vRanger, we decided to use that to back up these 8 key machines (2 of which are Win2k VMs...part of the problem, I am sure) and just use BE to move the vRanger repository to tape.  This hybrid method has eliminated the problem for us. :(

I will say that the vddkWrapper.dll did seem to work for Win2003/2008 VMs for us, overall, except the one time a Win2003 server went down with the ntdll.dll fault at the tail end of June.  Until then it seemed that only the two legacy Win2k machines were the holdout problems, which is understandable as they are no longer supported by MS.

The 'orphan' moniker means that it won't be rolled into a regular hotfix in the future, FYI.

Not the best news I realize, but thought the update would be useful info.

Best of luck.

 .

 Neil

[Jirka]

Hi,

I try use vddkWrapper.dll 'orphan hotfix but beremote.exe fault again frequently.. On this Backup Exec 2012 Revision 1798 Hotfix 180964 i found reason for this bug. On this Monday we upgrade BE to 2012 and apply last updates SP1a, 189571, 180964. After first nightly backup virtual machines beremote.exe fault again.

Faulting application name: beremote.exe, version: 14.0.1798.0, time stamp: 0x4f1e02d7
Faulting module name: MSVCR100.dll, version: 10.0.30319.1, time stamp: 0x4ba220dc
Exception code: 0xc0000005
Fault offset: 0x00000000000347de
Faulting process id: 0x9bc
Faulting application start time: 0x01cd748519ea6789
Faulting application path: C:\Program Files\Symantec\Backup Exec\beremote.exe
Faulting module path: C:\Windows\system32\MSVCR100.dll
Report Id: b5455b46-e0fa-11e1-bcc3-000533641566

Jirka

[Nabil_polska]

A hotfix is now available for this issue in the current version(s) of the product(s) mentioned in this article. Refer to the Hotfix link under Related Documents at the end of this article to obtain the hotfix needed to resolve the issue.

Backup Exec 2012 Revision 1798 Hotfix 180964

[rvanbaalen]

Glad to see a fix for 2012 but I am at 2010R3.  Like others, I am having the same problem.   

[nPc]

rvanbaalen & other 2010R3 folks, try this: 

Backup Exec 2010 R3 Hotfix 195395 - http://www.symantec.com/docs/TECH195395

[rvanbaalen]

I have this HF installed; http://www.symantec.com/business/support/index?page=content&id=TECH200892 The Backup Exec Remote Agent for Windows Systems service (beremote.exe) on a media server to intermittently crash in ntdll.dll with Hotfix 195395 installed and backing up a VMware guest machine. I would get rid of the HF but it can not be uninstalled

[Colin_Weaver]

For customers still experiencing beremote crashes after applying Hotfix 195395 for BE 2010 R3 or Hotfix 199866 for 2012, we are aware of at least one cause of beremote.exe stability problems relating to VMware backups that is not resolved. if you contact support formally (log a case) we have a new orphan and options to adjust registry settings which should help. Do not uninstall the Hotfixes mentioned above as they need to be installed to use the orphan and/or registry keys

[rvanbaalen]

Thanks Colin. One of our support engineers has opened up case number 03342405 . Just for the heck of it, are you able to elaborate on what could cause the stability problems? We were running wonderfully at 8pm Sunday and all of a sudden everything crashed just 17 hours later. I can find no changes/situations that occurred that window. I would sleep better if I knew what triggered the intermittent problem :)