Nightly database maintenance locking up SQL/BE, help!

[Brad_Hall]

Hello all, I have a BE11D install with all the latest patches on our Win2k3 SP2 File Server. Starting last week (suspiciously right around the same time this month's Windows Updates became available) our nightly backups started failing due to BE and it's SQL instance locking up and timing out repeatedly. The only way to get things working again is to restart BE's MSSQL service along with all other BE services. Just minutes after the 4AM database maintenance starts, I get these events in the system log that repeat endlessly until I restart the services:

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/20/2007
Time:        9:59:47 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Database Exception Context:AutoClearAlerts: sp ClearAutoAlert Error:-536837662: :: -2147217871:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80040e31 Timeout expired

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/20/2007
Time:        10:01:16 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Database Exception Context:{CALL dbo.[AutoClearAlert](?,?,?)} Error:-536837662: :: -2147217871:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80040e31 Timeout expired

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/20/2007
Time:        10:01:16 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Current DbTimeoutValue=30

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/20/2007
Time:        10:01:16 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Retry one more time

No clue why it's having this problem now. I've run BEUtility and ran a consistency check on the database and it passed that test fine. I rebuilt the indices as well as repaired the database and nothing abnormal was reported. Anyone have any suggestions as to what to try next? The only way I can get our nightly backups to complete now is to disable the database maintenance in the options menu.

[harmstrong]

Hi Brad,

Do you use the Desktop Laptop Option ,  check the windows services look for Backup Exec DLO administration service and Backup Exec Maintenance service.  If is install and you do not use go into the options and remove the DLO option .   Also , you can test moving the time of the backup exec maintenance .

Regards,

HArmstrong

Symantec

[Brad_Hall]

Hello HArmstrong,

No, we do not use the DLO option, those services are not present. I've tried moving the maintenance time up to before the nightly backups start as well as early in the morning after they've finished, and regardless of time it still seems to lock things up. The odd thing is BE reports that maintenance has completed in it's Alert log, but the services are all locked up after that. Thanks,

Brad

[effedup]

I am also experiencing very similar symptoms. I have not yet found a solution. If you find anything that yeilds positive results, would you post it here? I will do the same.

[CM]

We are having the same problem also, cannot believe there is no fix for this!

[Brad_Hall]

Somehow I'm not suprised.. still having this issue if anyone knows what's going on over here. It seems like it all started shortly after we installed the KB 928365 update for .NET framework 2.0. If you'll do a quick google search on "kb 928365" you'll see several others having problems with broken apps after this update as well, though I didn't see anyone having problems specifically with BE. Uninstalling said update didn't help matters, however.

[Lou_Buffetta]

Hi Brad

It looks like the problem is centered around "AutoClearAlert." Thanks for the good documentation.

-- From within Backup Exec go to the ALERTS page

-- In the left column (Task Bar) under "Alert Tasks" click on "Configure alert categories"

-- Scroll down the "Alert Categories" column to "General Information"

  -- This is the alert that records the Event ID 57345 in the Event Viewer

-- Is it set to Automatically clear after?

  -- If this is already selected - unselect it

  -- If this is not selected - select it and set it to something like 1 hour

-- OR -- you can simply uncheck the box for "Enable alerts for this category"

Another thing to check is make sure the volume where Backup Exec is installed has not run out of disk space.

Check out this TechNote. If you have database maintenance performaing a Consistency Check - deselect it. You will find this option in the same place indicated in the TechNote

    Tools menu> Options> Database Maintenance

Event ID 57348: Database Maintenance Failure, Maintenance has failed , Dump Database Failed.

http://seer.entsupport.symantec.com/docs/279751.htm

[Brad_Hall]

Hey Lou,

Alright, it looks like it actually finished its maintenance without locking up completely, however it did still time out a few times. I did what you suggested as far as turning off General Information alerts. However on the database maintenance options menu, the consistency check option has never been enabled. I unchecked the other 2 options, 'save contents of database to the BE data directory' and 'optimize database size'. This seems to have made it work.. but here's the event log errors I'm still seeing:

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/26/2007
Time:        11:50:45 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Database Exception Context:{? = CALL DBO.AGEJOBHISTORY(?,?,?)} Error:-536837662: :: -2147217871:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80040e31 Timeout expired

Event Type:    Information
Event Source:    MSSQL$BKUPEXEC
Event Category:    (2)
Event ID:    5144
Date:        7/26/2007
Time:        11:51:16 AM
User:        BEBETTER\backupexec
Computer:    SVCHAFILE01
Description:
Autogrow of file 'tempdev' in database 'tempdb' was cancelled by user or timed out after 141 milliseconds.  Use ALTER DATABASE to set a smaller FILEGROWTH value for this file or to explicitly set a new file size.

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/26/2007
Time:        11:51:38 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Database Exception Context:{? = CALL DBO.AGEJOBHISTORY(?,?,?)} Error:-536837662: :: -2147217871:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80040e31 Timeout expired

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/26/2007
Time:        11:51:38 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Database Exception Context:JobHistoryBO::DoAging Error:-536837662: :: -2147217871:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80040e31 Timeout expired

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/26/2007
Time:        11:53:02 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Database Exception Context:{CALL dbo.[AutoClearAlert](?,?,?)} Error:-536837662: :: -2147217871:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80040e31 Timeout expired

Event Type:    Information
Event Source:    Backup Exec
Event Category:    (65535)
Event ID:    57345
Date:        7/26/2007
Time:        11:53:32 AM
User:        N/A
Computer:    SVCHAFILE01
Description:
Database Exception Context:AutoClearAlerts: sp ClearAutoAlert Error:-536837662: :: -2147217871:Timeout expired :: DB Error Set 0: native=0xffffffff source=Microsoft OLE DB Provider for SQL Server hr=0x80040e31 Timeout expired

After it gave me these error events 6 or 7 times, it finished up DB maintenance and things seem to be fine. I'm rather disappointed that I had to uncheck those 2 options to get it to function however, it seems like the maintenance isn't really doing anything except purging old logs now. Any other suggestions? Thanks for the help!