β08-04-2014 04:14 PM
Hello.
I have upgraded three different backup servers to BE2014 and all of them have issues with the BESERVER.EXE service crashing at random times. I have attempted to repair the BE2014 install, as well remove .NET 4 and reinstall, but it continues to crash. I am getting the following errors in event viewer related to the crashes:
In the Application Log
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 8/3/2014 10:00:52 PM
Event time (UTC): 8/4/2014 4:00:52 AM
Event ID: 75c41b8b3318444d965fa72b2fb35546
Event sequence: 3
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: 8f5b00dc69a64bebb5fffc49daba4120-6524-130515984474946815
Trust level: Full
Application Virtual Path: /CRF
Application Path: C:\ProgramData\Symantec\CRF\8f5b00dc69a64bebb5fffc49daba4120\
Machine name: FLX00011
Process information:
Process ID: 6472
Process name: beserver.exe
Account name:
Exception information:
Exception type: ThreadAbortException
Exception message: Thread was being aborted.
at System.RuntimeMethodHandle._InvokeConstructor(IRuntimeMethodInfo method, Object[] args, SignatureStruct& signature, RuntimeType declaringType)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
at System.Configuration.TypeUtil.InvokeCtorWithReflectionPermission(ConstructorInfo ctor)
at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithFullTrust(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
at System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
at System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader, String filename, Int32 line)
at System.Configuration.BaseConfigurationRecord.CreateSectionDefault(String configKey, Boolean getRuntimeObject, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)
at System.Web.Configuration.RuntimeConfig.GetSectionObject(String sectionName)
at System.Web.Configuration.RuntimeConfig.GetSection(String sectionName, Type type, ResultsIndex index)
at System.Web.Configuration.RuntimeConfig.get_RoleManager()
at System.Web.Security.RoleManagerModule.Init(HttpApplication app)
at System.Web.HttpApplication.InitModulesCommon()
at System.Web.HttpApplication.InitInternal(HttpContext context, HttpApplicationState state, MethodInfo[] handlers)
at System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context)
at System.Web.HttpApplicationFactory.GetApplicationInstance(HttpContext context)
at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
Request information:
Request URL: http://127.0.0.1/CRF/CRF.aspx?DataFile=C:\Users\***\AppData\Local\Temp\8f5b00dc69a64bebb5fffc49daba4120\Data.xml&RDLFile=C:\Users\***AppData\Local\Temp\8f5b00dc69a64bebb5fffc49daba4120\RDL.rdl
Request path: /CRF/CRF.aspx
User host address: 127.0.0.1
User:
Is authenticated: False
Authentication Type:
Thread account name:
Thread information:
Thread ID: 45
Thread account name:
Is impersonating: False
Stack trace: at System.RuntimeMethodHandle._InvokeConstructor(IRuntimeMethodInfo method, Object[] args, SignatureStruct& signature, RuntimeType declaringType)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
at System.Configuration.TypeUtil.InvokeCtorWithReflectionPermission(ConstructorInfo ctor)
at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
at System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithFullTrust(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
at System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader)
at System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader, String filename, Int32 line)
at System.Configuration.BaseConfigurationRecord.CreateSectionDefault(String configKey, Boolean getRuntimeObject, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)
at System.Web.Configuration.RuntimeConfig.GetSectionObject(String sectionName)
at System.Web.Configuration.RuntimeConfig.GetSection(String sectionName, Type type, ResultsIndex index)
at System.Web.Configuration.RuntimeConfig.get_RoleManager()
at System.Web.Security.RoleManagerModule.Init(HttpApplication app)
at System.Web.HttpApplication.InitModulesCommon()
at System.Web.HttpApplication.InitInternal(HttpContext context, HttpApplicationState state, MethodInfo[] handlers)
at System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context)
at System.Web.HttpApplicationFactory.GetApplicationInstance(HttpContext context)
at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
Custom event details:
An unhandled exception occurred and the process was terminated.
Application ID: 8f5b00dc69a64bebb5fffc49daba4120
Process ID: 6472
Exception: System.OutOfMemoryException
Message: Exception of type 'System.OutOfMemoryException' was thrown.
StackTrace: at System.Web.DirectoryMonitor.FireNotifications()
at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
Application: beserver.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
Stack:
at System.Web.DirectoryMonitor.FireNotifications()
at System.Web.Util.WorkItem.CallCallbackWithAssert(System.Web.Util.WorkItemCallback)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
A process crash has been detected.
Faulting application: beserver.exe 14.1.1786.0
Faulting module: kernel32.dll 6.0.6002.19034
Fault offset 0x1674d
Exception code 0xe0434352.
A process crash has been detected.
Faulting application: beserver.exe 14.1.1786.0
Faulting module: kernel32.dll 6.0.6002.19034
Fault offset 0x1674d
Exception code 0xe0434352.
From the System Log:
Log Name: System
Source: Service Control Manager
Date: 8/3/2014 10:11:31 PM
Event ID: 7011
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer:
Description:
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the BackupExecRPCService service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Service Control Manager" Guid="{555908D1-A6D7-4695-8E1E-26931D2012F4}" EventSourceName="Service Control Manager" />
<EventID Qualifiers="49152">7011</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-08-04T04:11:31.000Z" />
<EventRecordID>1312537</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>System</Channel>
<Computer></Computer>
<Security />
</System>
<EventData>
<Data Name="param1">30000</Data>
<Data Name="param2">BackupExecRPCService</Data>
</EventData>
</Event>
Log Name: System
Source: Service Control Manager
Date: 8/4/2014 12:14:36 AM
Event ID: 7034
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer:
Description:
The Backup Exec Server service terminated unexpectedly. It has done this 2 time(s).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Service Control Manager" Guid="{555908D1-A6D7-4695-8E1E-26931D2012F4}" EventSourceName="Service Control Manager" />
<EventID Qualifiers="49152">7034</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-08-04T06:14:36.000Z" />
<EventRecordID>1312618</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>System</Channel>
<Computer></Computer>
<Security />
</System>
<EventData>
<Data Name="param1">Backup Exec Server</Data>
<Data Name="param2">2</Data>
</EventData>
</Event>
Any help would be greatly appreciated.
β08-04-2014 10:00 PM
Is BE 2014 updated with HF 218257 ?
Was the server rebooted post the upgrade & the repair install ?
β08-05-2014 12:23 AM
...open up BEutility.exe and run a repair of the BEDB.
Thanks!
β08-05-2014 07:48 AM
Would this HF not be applied when using Live Update? I have run Live Update and no updates have been found.
These servers have been rebooted post repair and upgrade.
β08-05-2014 08:03 AM
I have run a repair on all three server. Unfortunately, the crashing usually only occurs at night, so I will have to wait and see if this resolves the issue. Thanks for your input.
β08-05-2014 09:07 AM
I have seen cases where LiveUpdate incorrectly does not inform of any update. Best way to confirm would be check "Installed Updates" from the UI itself.
β08-05-2014 09:35 AM
218257 is installed. Thanks for pointing me to the this. It is good information to have.
β08-07-2014 07:04 AM
This did not resolve the issue.
β08-07-2014 09:23 AM
Would recommend to log a formal support case so that the crash dumps can be reviewed and this issue investigated further.
β08-27-2014 07:49 AM
Dear colleague,
You are not alone with this problem.
Last week we updated from BE2010R3 to BE2014, fortunately only on one server.
Since then, there is a problem with the memory consumption of the BE RPC server. It keeps increasing till all RAM is consumed, and then crashes, especially when several long jobs are running simultaneously.
But even when no jobs run, memory use of RPC server keeps increasing.
Did you manage to solve your problems? If so, can I know how?
I opened a formal case yesterday, which got escalated today, after a repair installation was done with no effect. I was also asked to exclude BE from the antivirus scanning.
I will inform you if a solution is presented!
Where are you located, if i may ask? I am in Brussels Belgium!
Best regards, and good luck!
Luc Vandromme
β08-27-2014 02:41 PM
If you are running Mcafee or please take a look at the following: Basically Exclude the CRF folder from scanning.
http://www.symantec.com/business/support/index?page=content&id=TECH223209
https://www-secure.symantec.com/connect/forums/v-79-57344-33967-programdatasymanteccrf
β08-28-2014 08:28 AM
I will give this a try. I work with McAfee and just wanted to give a heads up that the information in article is incorrect. The exception path should be "C:\ProgramData\Symantec\CRF\". If you do not include the backslash at the end, McAfee thinks you are excluding the filename CRF and not the folder.
Thanks for the info.
β08-28-2014 11:51 PM
Hello Theo,
I opened a case with symantec and got some good and helpfull support. You should also exclude the exe.files from the BE services from scanning by MacAfee.
Last night, i executed all my jobs simultaneously, and no crash happened. The memory use was also within limits, and there were not many warnings in the event logs.
I hope this might help you!
Good luck!
Best regards,
Lucky Luke
β08-29-2014 06:23 AM
Hello,
This document concerning exclusions from MAcAfee antivirusscanning might help:
Best regards,
LuckyLuke