Crash when trying to do anything with deduplicatio...

PaulF · ‎01-10-2011

We have a 16 core Dell server, 24GB RAM, 20TB Dell PowerVault disk array running Server 2008R2 Std. with Backup Exec 2010R2 with all patches.

Windows UAC is disabled.

Windows DEP is disabled.

I have a deduplication folder set up on the array which is dedicated to this purpose. Needless to say the array is formatted NTFS as is the boot drive.

I have about 20 servers backed up to the deduplication folder.

Everything was running ok until before xmas. After returning to work after xmas the server was complaining it wanted a reboot after automatically updating (bexec). Now when I run any jobs that involve mounting media (backup, restore, catalogue of any dedupe media), the backup exec engine service crashes:

Log Name: Application

Source: Application Error

Date: 10/01/2011 17:15:36

Event ID: 1000

Task Category: (100)

Level: Error

Keywords: Classic

User: N/A

Computer: osiris.f.q.d.n

Description:

Faulting application name: bengine.exe, version: 13.0.4164.111, time stamp: 0x4d0cc94b

Faulting module name: DeviceIo.dll, version: 0.0.5071.110, time stamp: 0x4cfea7e2

Exception code: 0xc0000409

Fault offset: 0x000000000002069d

Faulting process id: 0x1408

Faulting application start time: 0x01cbb0e5a4f2ebed

Faulting application path: C:\Program Files\Symantec\Backup Exec\bengine.exe

Faulting module path: C:\Program Files\Symantec\Backup Exec\DeviceIo.dll

Report Id: 3c99baa2-1cdd-11e0-b9c3-b8ac6f15d63c

Event Xml:

<Channel>Application</Channel>

<Computer>osiris.f.q.d.n</Computer>

</System>

<Data>bengine.exe</Data>

<Data>DeviceIo.dll</Data>

<Data>C:\Program Files\Symantec\Backup Exec\bengine.exe</Data>

<Data>C:\Program Files\Symantec\Backup Exec\DeviceIo.dll</Data>

</EventData>

</Event>

Ok, it's a deviceio error which suggests a problem with the backup media, which in this case is a deduplication folder. All drives are ok, disks have been checked (however I did note on the first disk check that Windows reported minor errors were repaired).

I enabled debugging (using -debug in service parameters) - I get this in the logs:

6060] 01/08/11 12:15:17 [server] - DeviceManager: returned result to caller 0x00000000

[8112] 01/08/11 12:15:17 [server] - Job::Mount: acquired device

[6060] 01/08/11 12:15:17 [server] - DeviceManager: going to sleep for 900000 msecs

[8112] 01/08/11 12:15:17 [server] - Job::Mount: mounting tape

[6508] 01/08/11 12:15:21 [server] - Updating status for: 'Restore 00009' (0x0 0x9)

[6508] 01/08/11 12:15:21 [server] - Status for: 'Restore 00009' updated

[8112] 01/08/11 12:16:24 [server] - Updating session {a1711539-87e7-403b-8b59-9463b8828e1e} with drive Deduplication Storage Folder 0001:8 {2fbab752-2ab4-4828-8a6e-0a3ab7daaf2d}

[8112] 01/08/11 12:16:24 [server] - Job::Mount: media mounted

***CRASH.

With SGMon:

PVLSVR: [09/01/11 11:50:24] [7972] AdammSession::MountMedia()

Session = {BCD26890-53B5-4CD4-BF1B-A8B21D02CC84}

Library = {057FF383-A45D-4997-BA08-870BB7BCAB87}, "Deduplication Storage Folder 0001"

Slot = 0000

Drive = {E36B9856-9568-4700-989C-3F1A83DFEC6F}, "Deduplication Storage Folder 0001:1"

Media = {CCD306A5-3913-4645-8860-3FD80211521B}, "OST00000001-49E4CD0EA7365614"

Side = 0001

SUCCESS!

BENGINE: [09/01/11 11:50:24] [4064] [server] - Updating session {bcd26890-53b5-4cd4-bf1b-a8b21d02cc84} with drive Deduplication Storage Folder 0001:1 {e36b9856-9568-4700-989c-3f1a83dfec6f}

BENGINE: [09/01/11 11:50:24] [4064] [server] - Job::Mount: media mounted

BECAT: [09/01/11 11:50:24] [5648] ReleaseConnection ...

BECAT: [09/01/11 11:50:24] [5648] Reset connection ...

BECAT: [09/01/11 11:50:24] [5648] ReleaseConnection ...

BECAT: [09/01/11 11:50:24] [5648] Reset connection ...

BKUPEXEC: [09/01/11 11:50:24] [0042] [Info] VSN received: type = MEDIA_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042] [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:24] [0042] [Info] VSN received: type = DRIVE_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042] [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:24] [0042] [Info] VSN received: type = CHANGER_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042] [Debug] VSN added to queues

BKUPEXEC: [09/01/11 11:50:24] [0042] [Info] VSN received: type = CATALOG_UPDATE

BKUPEXEC: [09/01/11 11:50:24] [0042] [Debug] No handler registered for this VSN - ignoring

BESERVER: [09/01/11 11:50:25] [4112] 05 AdammAdminBO::Execute() - ADAMM_ADMIN_MEDIA_SUMMARY.

PVLSVR: [09/01/11 11:50:25] [7972] PvlEntityDatabase::QueryEntity()

ParentEntityType = NULL_ENTITY_TYPE

ParentEntityGuid = {00000000-0000-0000-0000-000000000000}

EntityType = 0031

EntityGuid = {00000000-0000-0000-0000-000000000000}

EntityName = "

EntityFlags = 0x00000000

BESERVER: [09/01/11 11:50:25] [4112] 05 AdammAdminBO::Execute() - hr = 0x0

^---- and here is our crash again. Always after AdammAdminBO and 'ADAMM_ADMIN_MEDIA_SUMMARY'

I had several months of backups in this store which was running at over 12TB in size. I only hesitated in wiping it and starting again due to one of our students needed some data restoring, however I have been unable to restore this.

I had a look at dbutil.exe which can list fingerprints from what I can tell, but it _always_ crashes with dbutil -n

Problem signature:

Problem Event Name: APPCRASH

Application Name: Dbutil.exe

Application Version: 6.6.0.38791

Application Timestamp: 4d0b0793

Fault Module Name: pthreadVC2.dll

Fault Module Version: 2.9.0.0

Fault Module Timestamp: 4c28caa8

Exception Code: c0000005

Exception Offset: 00000000000045d8

OS Version: 6.1.7600.2.0.0.272.7

Locale ID: 2057

Additional Information 1: 8a26

Additional Information 2: 8a26e23e382a80f4043654a2c9fb737e

Additional Information 3: 336d

Additional Information 4: 336d8a278a592baebfce4696ff2b3dd3

Having had enough, I deleted the deduplication folder from Backup Exec (but did not delete the physical folder from the disk), rebooted then created a new folder (using a new disk folder to avoid known problems). The folder has the same name in Backup Exec however.

So I have a fresh deduplication folder, totally empty, I try to run a backup job and I get the exact same crash. I have deleted it again and recreated it, same problem. The disks have been checked for errors, none found.

I suspect there must be some old data lying around causing this problem - anyone have any suggestions?

*edit - I do notice all my old media is still listed under the Media tab, which I suspect might be part of the problem.

teiva-boy · ‎01-10-2011

Normally I'm pretty helpful when it comes to dedupe, but I would suggest in this case, to log a case with support.

Save yourself the trouble, and register with mysupport.symantec.com and log the initial case, so you can call in with a support number. Log the case as a level 2 or 1 severity.

PaulF · ‎01-10-2011

More info - in a newly created deduplication folder, I get the same crash.

If I rename the folder (to say, 0002 on the end rather than 0001) then I can run a backup without a crash. Note this is the cosmetic name within backup exec, not the folder name on the disk array.

Lars_J · ‎01-17-2011

I'm experiencing exactly the same problem - Job Engine has began to crash each time a Deduplication job runs and this behaviour showed up after 25th December.

My first thought was that a windows update was automatically installed around that time, because no other changes were made on the server. I've removed the dedup feature, uninstalled all BE hotfixes and re-applied them without success. So my question is if you have resolved this issue at your side yet?

VOX

Crash when trying to do anything with deduplication media