Media Server Rebooting During Backups

[Justin_Bell]

I have a Dell PowerEdge 1850 that keeps spontaneously rebooting during backup jobs. I receive a BSOD, but I haven't caught what it actually says because the problem is so unpredictable.

In the Event Log I get an Event ID 1003 with the following error code:

"Error Code 1000007e, parameter1 C00000005, parameter2 badb6421, parameter3 b5da85cc, parameter4 b5da82c8"

I can't put my finger on any particular time that this is happening, or what is unique about the jobs that cause this. I am not having reboot problems when I backup the Exchange Server. I am not having problems backing up the Accounting Server. I am having problems when I am backing up the file server, and files on the media server. I do have AOFO installed. I am backing up the System State as well.

We have hotfix2 installed on the media server and I've pushed new installs to all the remote servers.

Thanks in advance for any help!

[Justin_Bell]

Just wanted to post an update:

I've run a debugger against the Memory dumps and it looks like Symevent.sys is the problem. The output from the debugger follows"

Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: badb6421, The address that the exception occurred at
Arg3: b5da85cc, Exception Record Address
Arg4: b5da82c8, Context Record Address

Debugging Details:
------------------

DBGHELP: C:\WINDOWS\Debug\ntfs.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\symbols\sys\ntfs.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\sys\ntfs.pdb - file not found
DBGHELP: Ntfs - public symbols
c:\windows\symbols\ntfs.pdb\F237A6C4F15749599A2BB43878FC27CA2\ntfs.pdb
DBGHELP: C:\WINDOWS\Debug\fltMgr.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\symbols\sys\fltMgr.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\sys\fltMgr.pdb - file not found
DBGHELP: fltMgr - public symbols
c:\windows\symbols\fltMgr.pdb\52D998F48F574FD9AAEB7BA760E3A4741\fltMgr.pdb
DBGHELP: C:\WINDOWS\Debug\SymEvent.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\symbols\SYS\SymEvent.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\SYS\SymEvent.pdb - file not found
SYMSRV: c:\windows\symbols\SymEvent.pdb\B1C2B1328C684DFEA71B8F468C37CF041\SymEvent.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/SymEvent.pdb/B1C2B1328C684DFEA71B8F468C37CF041/SymEvent.pdb not found
DBGHELP: C:\bld_area\SymEvent_r11\Symevent\src\bin.iruk\SymEvent.pdb - file not found
*** ERROR: Symbol file could not be found. Defaulted to export symbols for SYMEVENT.SYS -
DBGHELP: SYMEVENT - export symbols
DBGHELP: C:\WINDOWS\Debug\srv.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\symbols\sys\srv.pdb - file not found
DBGHELP: C:\WINDOWS\Debug\sys\srv.pdb - file not found
DBGHELP: srv - public symbols
c:\windows\symbols\srv.pdb\5966D99E78A645A1A0690E418CACCF032\srv.pdb
DBGHELP: c:\windows\symbols\ntkrpamp.exe\42435B14253000\ntkrpamp.exe - OK
DBGHELP: c:\windows\symbols\Ntfs.sys\42435DFD95000\Ntfs.sys - OK

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

FAULTING_IP:
Ntfs!NtfsAcquireResourceExclusive+8
badb6421 6681380207 cmp word ptr ,702h

EXCEPTION_RECORD: b5da85cc -- (.exr ffffffffb5da85cc)
ExceptionAddress: badb6421 (Ntfs!NtfsAcquireResourceExclusive+0x00000008)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter: 00000000
Parameter: 00000000
Attempt to read from address 00000000

CONTEXT: b5da82c8 -- (.cxr ffffffffb5da82c8)
eax=00000000 ebx=88f8cde0 ecx=0000da7e edx=8a23c020 esi=88f8cf94 edi=00000000
eip=badb6421 esp=b5da8694 ebp=b5da8694 iopl=0 nv up ei ng nz ac pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010296
Ntfs!NtfsAcquireResourceExclusive+0x8:
badb6421 6681380207 cmp word ptr ,702h ds:0023:00000000=????
Resetting default scope

PROCESS_NAME: System

CURRENT_IRQL: 0

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

READ_ADDRESS: 00000000

BUGCHECK_STR: 0x7E

DEFAULT_BUCKET_ID: NULL_DEREFERENCE

LAST_CONTROL_TRANSFER: from badfbf85 to badb6421

STACK_TEXT:
b5da8694 badfbf85 88f8cc38 00000000 00000001 Ntfs!NtfsAcquireResourceExclusive+0x8
b5da8798 8081dce5 8a23c020 88f8cde0 88f8cde0 Ntfs!NtfsFsdCreate+0x362
b5da87ac bae7c51e 8a8860c0 00000000 8a9c4490 nt!IofCallDriver+0x45
b5da87dc 8081dce5 8a745ac8 88f8cde0 b5da8844 fltMgr!FltpCreate+0x1aa
b5da87f0 b8f5022b b5da8844 8a040200 00000000 nt!IofCallDriver+0x45
WARNING: Stack unwind information not available. Following frames may be wrong.
b5da8874 bae6eb43 00000000 88f8cde0 88f8cfdc SYMEVENT!SYMEvent_GetVMDataPtr+0x886b
b5da8898 bae7c5af b5da88b8 8a7458a0 00000000 fltMgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x20b
b5da88d4 8081dce5 8a7458a0 88f8cde0 88f8cde0 fltMgr!FltpCreate+0x23b
b5da88e8 808f8255 86358c90 879aff20 00000000 nt!IofCallDriver+0x45
b5da89d0 808f869c 8a7458a0 00000000 8631e138 nt!IopParseDevice+0xa35
b5da8a08 8093666b 86358c90 00000000 8631e138 nt!IopParseFile+0x46
b5da8a88 80932de6 000038bc b5da8ac8 00000040 nt!ObpLookupObjectName+0x11f
b5da8adc 808ea211 00000000 00000000 77386800 nt!ObOpenObjectByName+0xea
b5da8b58 808eb4ab b5da8cc8 02000080 b5da8c90 nt!IopCreateFile+0x447
b5da8bb4 b699fefd b5da8cc8 02000080 b5da8c90 nt!IoCreateFile+0xa3
b5da8c24 b69a0f29 88287380 b5da8cc8 02000080 srv!SrvIoCreateFile+0x36d
b5da8cf0 b69a15e4 8794f1d8 e4423730 02000080 srv!SrvNtCreateFile+0x5cc
b5da8d78 b6984bc6 88287388 8a75b6e0 b699a451 srv!SrvSmbNtCreateAndX+0x15c
b5da8d84 b699a451 00000000 89abb880 00000000 srv!SrvProcessSmb+0xb7
b5da8dac 80948bb2 0075b6e0 00000000 00000000 srv!WorkerThread+0x138
b5da8ddc 8088d4d2 b699a394 8a75b6e0 00000000 nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16


FOLLOWUP_IP:
SYMEVENT!SYMEvent_GetVMDataPtr+886b
b8f5022b 894618 mov dword ptr ,eax

SYMBOL_STACK_INDEX: 5

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: SYMEVENT

IMAGE_NAME: SYMEVENT.SYS

DEBUG_FLR_IMAGE_TIMESTAMP: 4272844c

SYMBOL_NAME: SYMEVENT!SYMEvent_GetVMDataPtr+886b

STACK_COMMAND: .cxr 0xffffffffb5da82c8 ; kb

FAILURE_BUCKET_ID: 0x7E_SYMEVENT!SYMEvent_GetVMDataPtr+886b

BUCKET_ID: 0x7E_SYMEVENT!SYMEvent_GetVMDataPtr+886b

Followup: MachineOwner

[Philip_Pearson]

Are you using symantec tape drivers or proprietry

[Justin_Bell]

I am using 100% Symantec drivers for the tape drive and library (it's a Dell PV132).

I've also downloaded and installed the newest drivers off the Symantec Support website.

[Gauri_Ketkar]

Hi,

have you tried a backup without AOFO activated for that job ?


Update us on the same and revert for any further Query



Thank you
Gauri

[Justin_Bell]

Yes, I've tried it with AOFO enabled and disabled. The rebooting is seemingly random in that regard. The server hasn't rebooted for 3 days now...

Is there a known issue with the AOFO?

Thanks,
Justin

[Philip_Pearson]

There is a know issue with 11d, its got more bugs than a hollow log.

[Justin_Bell]

Okay, it's still happening. Last night (Monday the 6th of February) it rebooted 12 times. Each time I get the same exact error and each time it says an inventory failed. Since BE makes an inventory job if the server reboots I have 12 inventories. But, before the inventory can finish the server reboots, so the job fails and the job log gives an error that the job log is not complete and the job log for this job may not be generated.

There are no warnings in the error log prior to the reboot and just your typical "the server has shutdown unexpectedly" errors afterwards.