Highlighted

BESERVER.EXE crashing randomly since upgrading to BE2014

Hello.

 

I have upgraded three different backup servers to BE2014 and all of them have issues with the BESERVER.EXE service crashing at random times.  I have attempted to repair the BE2014 install, as well remove .NET 4 and reinstall, but it continues to crash.  I am getting the following errors in event viewer related to the crashes:

In the Application Log

 

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 8/3/2014 10:00:52 PM
Event time (UTC): 8/4/2014 4:00:52 AM
Event ID: 75c41b8b3318444d965fa72b2fb35546
Event sequence: 3
Event occurrence: 1
Event detail code: 0
 
Application information:
    Application domain: 8f5b00dc69a64bebb5fffc49daba4120-6524-130515984474946815
    Trust level: Full
    Application Virtual Path: /CRF
    Application Path: C:\ProgramData\Symantec\CRF\8f5b00dc69a64bebb5fffc49daba4120\
    Machine name: FLX00011
 
Process information:
    Process ID: 6472
    Process name: beserver.exe
    Account name:
 
Exception information:
    Exception type: ThreadAbortException
    Exception message: Thread was being aborted.
   at System.RuntimeMethodHandle._InvokeConstructor(IRuntimeMethodInfo method, Object[] args, SignatureStruct& signature, RuntimeType declaringType)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
   at System.Configuration.TypeUtil.InvokeCtorWithReflectionPermission(ConstructorInfo ctor)
   at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
   at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithFullTrust(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
   at System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
   at System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader, String filename, Int32 line)
   at System.Configuration.BaseConfigurationRecord.CreateSectionDefault(String configKey, Boolean getRuntimeObject, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)
   at System.Web.Configuration.RuntimeConfig.GetSectionObject(String sectionName)
   at System.Web.Configuration.RuntimeConfig.GetSection(String sectionName, Type type, ResultsIndex index)
   at System.Web.Configuration.RuntimeConfig.get_RoleManager()
   at System.Web.Security.RoleManagerModule.Init(HttpApplication app)
   at System.Web.HttpApplication.InitModulesCommon()
   at System.Web.HttpApplication.InitInternal(HttpContext context, HttpApplicationState state, MethodInfo[] handlers)
   at System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context)
   at System.Web.HttpApplicationFactory.GetApplicationInstance(HttpContext context)
   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)

 
 
Request information:
    Request URL: http://127.0.0.1/CRF/CRF.aspx?DataFile=C:\Users\***\AppData\Local\Temp\8f5b00dc69a64bebb5fffc49daba4120\Data.xml&RDLFile=C:\Users\***AppData\Local\Temp\8f5b00dc69a64bebb5fffc49daba4120\RDL.rdl
    Request path: /CRF/CRF.aspx
    User host address: 127.0.0.1
    User:  
    Is authenticated: False
    Authentication Type:  
    Thread account name:
 
Thread information:
    Thread ID: 45
    Thread account name:
    Is impersonating: False
    Stack trace:    at System.RuntimeMethodHandle._InvokeConstructor(IRuntimeMethodInfo method, Object[] args, SignatureStruct& signature, RuntimeType declaringType)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
   at System.Configuration.TypeUtil.InvokeCtorWithReflectionPermission(ConstructorInfo ctor)
   at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
   at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithFullTrust(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
   at System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
   at System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader, String filename, Int32 line)
   at System.Configuration.BaseConfigurationRecord.CreateSectionDefault(String configKey, Boolean getRuntimeObject, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)
   at System.Web.Configuration.RuntimeConfig.GetSectionObject(String sectionName)
   at System.Web.Configuration.RuntimeConfig.GetSection(String sectionName, Type type, ResultsIndex index)
   at System.Web.Configuration.RuntimeConfig.get_RoleManager()
   at System.Web.Security.RoleManagerModule.Init(HttpApplication app)
   at System.Web.HttpApplication.InitModulesCommon()
   at System.Web.HttpApplication.InitInternal(HttpContext context, HttpApplicationState state, MethodInfo[] handlers)
   at System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context)
   at System.Web.HttpApplicationFactory.GetApplicationInstance(HttpContext context)
   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
 
 
Custom event details:

 

An unhandled exception occurred and the process was terminated.

Application ID: 8f5b00dc69a64bebb5fffc49daba4120

Process ID: 6472

Exception: System.OutOfMemoryException

Message: Exception of type 'System.OutOfMemoryException' was thrown.

StackTrace:    at System.Web.DirectoryMonitor.FireNotifications()
   at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

 

Application: beserver.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
Stack:
   at System.Web.DirectoryMonitor.FireNotifications()
   at System.Web.Util.WorkItem.CallCallbackWithAssert(System.Web.Util.WorkItemCallback)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

 

A process crash has been detected.

Faulting application: beserver.exe 14.1.1786.0
Faulting module: kernel32.dll 6.0.6002.19034
Fault offset            0x1674d
Exception code         0xe0434352.

A process crash has been detected.

Faulting application: beserver.exe 14.1.1786.0
Faulting module: kernel32.dll 6.0.6002.19034
Fault offset            0x1674d
Exception code         0xe0434352.

 

From the System Log:

Log Name:      System
Source:        Service Control Manager
Date:          8/3/2014 10:11:31 PM
Event ID:      7011
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:     
Description:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the BackupExecRPCService service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908D1-A6D7-4695-8E1E-26931D2012F4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7011</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-08-04T04:11:31.000Z" />
    <EventRecordID>1312537</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer></Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">30000</Data>
    <Data Name="param2">BackupExecRPCService</Data>
  </EventData>
</Event>

 

Log Name:      System
Source:        Service Control Manager
Date:          8/4/2014 12:14:36 AM
Event ID:      7034
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:     
Description:
The Backup Exec Server service terminated unexpectedly.  It has done this 2 time(s).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908D1-A6D7-4695-8E1E-26931D2012F4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7034</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-08-04T06:14:36.000Z" />
    <EventRecordID>1312618</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer></Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">Backup Exec Server</Data>
    <Data Name="param2">2</Data>
  </EventData>
</Event>

 

Any help would be greatly appreciated.

13 Replies
Highlighted

Is BE 2014 updated with HF

Is BE 2014 updated with HF 218257 ?

Was the server rebooted post the upgrade & the repair install ?

 

Highlighted

...open up BEutility.exe and

...open up BEutility.exe and run a repair of the BEDB.

Thanks!

Highlighted

Would this HF not be applied

Would this HF not be applied when using Live Update?  I have run Live Update and no updates have been found.

 

These servers have been rebooted post repair and upgrade.

Highlighted

I have run a repair on all

I have run a repair on all three server.  Unfortunately, the crashing usually only occurs at night, so I will have to wait and see if this resolves the issue.  Thanks for your input.

Highlighted

I have seen cases where

I have seen cases where LiveUpdate incorrectly does not inform of any update. Best way to confirm would be check "Installed Updates" from the UI itself.

Highlighted

218257 is installed.  Thanks

218257 is installed.  Thanks for pointing me to the this.  It is good information to have.

Highlighted

This did not resolve the

This did not resolve the issue.

Highlighted

Would recommend to log a

Would recommend to log a formal support case so that the crash dumps can be reviewed and this issue investigated further.

Highlighted

Dear colleague, You are not

Dear colleague,

You are not alone with this problem.

Last week we updated from BE2010R3 to BE2014, fortunately only on one server.

Since then, there is a problem with the memory consumption of the BE RPC server. It keeps increasing till all RAM is consumed, and then crashes, especially when several long jobs are running simultaneously.

But even when no jobs run, memory use of RPC server keeps increasing.

Did you manage to solve your problems? If so, can I know how?

I opened a formal case yesterday, which got escalated today, after a repair installation was done with no effect. I was also asked to exclude BE from the antivirus scanning.

I will inform you if a solution is presented!

Where are you located, if i may ask? I am in Brussels Belgium!

Best regards, and good luck!

Luc Vandromme

Highlighted

If you are running Mcafee or

If you are running Mcafee or please take a look at the following:   Basically Exclude the CRF folder from scanning.

 

 

http://www.symantec.com/business/support/index?page=content&id=TECH223209

 

https://www-secure.symantec.com/connect/forums/v-79-57344-33967-programdatasymanteccrf

Highlighted

I will give this a try.  I

I will give this a try.  I work with McAfee and just wanted to give a heads up that the information in article is incorrect.  The exception path should be "C:\ProgramData\Symantec\CRF\".  If you do not include the backslash at the end, McAfee thinks you are excluding the filename CRF and not the folder.

 

Thanks for the info.

Highlighted

Hello Theo, I opened a case

Hello Theo,

I opened a case with symantec and got some good and helpfull support. You should also exclude the exe.files from the BE services from scanning by MacAfee.

Last night, i executed all my jobs simultaneously, and no crash happened. The memory use was also within limits, and there were not many warnings in the event logs.

I hope this might help you!

Good luck!

 

Best regards,

 

Lucky Luke

Highlighted

Hello, This document

Hello,

This document concerning exclusions from MAcAfee antivirusscanning might help:

http://www.symantec.com/business/support/index?page=content&id=TECH74529&profileURL=https%3A%2F%2Fsy...

 

Best regards,

 

LuckyLuke