Backup Exec 2010 R2 Job engine keeps crashing out

[Greg_Sayer]

hi all,

we are experincing a issue with BE 2010 r2,

at random intervals, mostly before the nightly backup schedules run, the job engine keeps faulting and which then creates a comms stalled error, and the jobs then fail

setup: 2 servers, one with Central administrator server roles, the other a managed media server, its this media server that has the faulting BEngine

windows logs report the following:
Log Name: Application
Source: Application Error
Date: 12/11/2010 09:05:49
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: XXXX
Description:
Faulting application name: bengine.exe, version: 13.0.4164.103, time stamp: 0x4c6e8ddb
Faulting module name: DeviceIo.dll, version: 0.0.5071.104, time stamp: 0x4c73cd7b
Exception code: 0xc0000409
Fault offset: 0x000000000002047d
Faulting process id: 0x1790
Faulting application start time: 0x01cb823b0b77a113
Faulting application path: C:\Program Files\Symantec\Backup Exec\bengine.exe
Faulting module path: C:\Program Files\Symantec\Backup Exec\DeviceIo.dll
Report Id: 0a5be3fb-ee3c-11df-818c-0022196a1b33
Event Xml:

1000
2
100
0x80000000000000

26266228
Application
XXXX

bengine.exe
13.0.4164.103
4c6e8ddb
DeviceIo.dll
0.0.5071.104
4c73cd7b
c0000409
000000000002047d
1790
01cb823b0b77a113
C:\Program Files\Symantec\Backup Exec\bengine.exe
C:\Program Files\Symantec\Backup Exec\DeviceIo.dll
0a5be3fb-ee3c-11df-818c-0022196a1b33

[RahulG]

Because the DeviceIo.dll is reporting in the crash then it could be the disk configuration itself that is possibly causing the issue. Make sure the Drivers are update to date and also run live update and install all the patches available

[Greg_Sayer]

cheers, we tried that last night, but still getting the issue :(

[PaulF]

We have the same problem backing up to or restoring from our 16TB deduplication folder.  I suspect a problem with the PureDisk back end (as this appears to be handled by Backup Exec as a tape changer).

We can run catalogue jobs though.

Extract from debug log:

...

[6060] 01/08/11 12:15:17 [server]             - DeviceManager: returned result to caller 0x00000000

[8112] 01/08/11 12:15:17 [server]             - Job::Mount: acquired device

[6060] 01/08/11 12:15:17 [server]             - DeviceManager: going to sleep for 900000 msecs

[8112] 01/08/11 12:15:17 [server]             - Job::Mount: mounting tape

[6508] 01/08/11 12:15:21 [server]             - Updating status for: 'Restore 00009' (0x0 0x9)

[6508] 01/08/11 12:15:21 [server]             - Status for: 'Restore 00009' updated

[8112] 01/08/11 12:16:24 [server]             - Updating session {a1711539-87e7-403b-8b59-9463b8828e1e} with drive Deduplication Storage Folder 0001:8 {2fbab752-2ab4-4828-8a6e-0a3ab7daaf2d}

[8112] 01/08/11 12:16:24 [server]             - Job::Mount: media mounted      

***CRASH.

[pkh]

You should start a new discussion so that your problem gets the attention that it deserves.

[PaulF]

CORRECTION - cataloguing media causes the same problem - we can run *inventory* jobs, but that's it.

[PaulF]

More debug output from a catalogue job on random media in the dedupe folder:

PVLSVR:   [09/01/11 11:50:24] [7972]     AdammSession::MountMedia()

            Session = {BCD26890-53B5-4CD4-BF1B-A8B21D02CC84}

            Library = {057FF383-A45D-4997-BA08-870BB7BCAB87}, "Deduplication Storage Folder 0001"

            Slot = 0000

            Drive = {E36B9856-9568-4700-989C-3F1A83DFEC6F}, "Deduplication Storage Folder 0001:1"

            Media = {CCD306A5-3913-4645-8860-3FD80211521B}, "OST00000001-49E4CD0EA7365614"

            Side = 0001

            SUCCESS!

BENGINE:  [09/01/11 11:50:24] [4064]     [server]             - Updating session {bcd26890-53b5-4cd4-bf1b-a8b21d02cc84} with drive Deduplication Storage Folder 0001:1 {e36b9856-9568-4700-989c-3f1a83dfec6f}

BENGINE:  [09/01/11 11:50:24] [4064]     [server]             - Job::Mount: media mounted

BECAT:    [09/01/11 11:50:24] [5648]     ReleaseConnection ...

BECAT:    [09/01/11 11:50:24] [5648]     Reset connection ...

BECAT:    [09/01/11 11:50:24] [5648]     ReleaseConnection ...

BECAT:    [09/01/11 11:50:24] [5648]     Reset connection ...

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Info] VSN received: type = MEDIA_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Info] VSN received: type = DRIVE_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Info] VSN received: type = CHANGER_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Info] VSN received: type = CATALOG_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042]     [Debug] No handler registered for this VSN - ignoring

BESERVER: [09/01/11 11:50:25] [4112]     05 AdammAdminBO::Execute() - ADAMM_ADMIN_MEDIA_SUMMARY.

PVLSVR:   [09/01/11 11:50:25] [7972]     PvlEntityDatabase::QueryEntity()

            ParentEntityType = NULL_ENTITY_TYPE

            ParentEntityGuid = {00000000-0000-0000-0000-000000000000}

            EntityType = 0031

            EntityGuid = {00000000-0000-0000-0000-000000000000}

            EntityName = "

            EntityFlags = 0x00000000

BESERVER: [09/01/11 11:50:25] [4112]     05 AdammAdminBO::Execute() - hr = 0x0

<---- and here is our crash again.  This NULL_ENTITY_TYPE is also returned on other jobs.

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Info] VSN received: type = BEM_DRIVE

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Debug] VSN is a MODIFY

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Info] VSN received: type = BEM_LIBRARY

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Debug] VSN is a MODIFY

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Info] VSN received: type = BEM_MEDIA

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Debug] VSN is a MODIFY

BKUPEXEC: [09/01/11 11:50:25] [0042]     [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:25] [0007]     [Debug] Preview pane received an object change event

[BE_ADMIN2010]

Hi there,

I have been dealing with that issue over the past week or so, I have managed to get my backups working again thankfully.

It seems to be  a problematic hotfix for windows 2008 R2, if you have windows update KB979444. Found here:

http://support.microsoft.com/kb/979444

If that is installed, uninstall it and give that a try:)