SQL 2008 R2 backup keeps failing with error code :e000ff19

[coolhunk824u]

We have a Windows 2008 R2 environment , the Backup Exec 2010 R2 server is installed on a separate server. We are using HP 1/8 G2 Autoloader tape library. The backup jobs are running fine for AD - system state ; Exchange 2010 SP1 DAG and system state.

When I choose to select the passive SQL node for system state, it works fine as well.

But it wont let me choose the active node. And when I select the Cluster database name, the backup job fails with the error code :e000ff19.

Below is the description of th error :

Job History for Backup 00033

--------------------------------------------------------------------------------

Expand All Collapse All

Backup 00033
Job name            : Backup 00033Job type            : BackupJob status          : FailedJob log             : C:\Program Files\Symantec\Backup Exec\Data\BEX_xxxxFB_00030.xmlServer name         : xxxxFBSelection list name : Backup 00033
Device name         : Backup-to-Disk Folder 0001Target name         : Backup-to-Disk Folder 0001Media set name      : Keep Data for 8 Weeks
Error category    : Resource ErrorsError             : e000ff19 - A communications failure has occurred with a SQL Server resource.
For additional information regarding this error refer to link V-79-57344-65305
 

Job Summary Information
Byte count          : 0 bytesJob rate            : 0.00 MB/Min (Byte count of all backup sets divided by Elapsed time for all backup sets)
Files               : 0Directories         : 0Skipped files       : 0Corrupt files       : 0Files in use        : 0
Original start time : Thursday, October 27, 2011 5:22:35 PMJob started         : Thursday, October 27, 2011 5:22:39 PMJob ended           : Thursday, October 27, 2011 5:24:43 PMElapsed time        : 00:02:04
 

Set Detail Information - Backup \\xxxxCLDB.local.com\Microsoft SQL Server
Set type               : BackupSet status             : CompletedSet description        :
Resource name          : \\xxxxCLDB.local.com\Microsoft SQL ServerLogon account          : Domain AdministratorEncryption used        : None
Error                  : e000ff19 - A communications failure has occurred with a SQL Server resource.

Agent used                     : YesAdvanced Open File Option used : Microsoft Volume Shadow Copy Service (VSS)
 

Byte count             : 0 bytesRate                   : 0.00 MB/Min
Files                  : 0Directories            : 0Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Thursday, October 27, 2011 5:22:40 PMEnd time               : Thursday, October 27, 2011 5:24:43 PM

 
Set Detail Information - Backup \\xxxxCLDB.local.com\P:
Set type               : BackupSet status             : CompletedSet description        :
Resource name          : \\xxxxCLDB.local.com\P:Logon account          : Domain AdministratorEncryption used        : None
Error                  : e000ff19 - A communications failure has occurred with a SQL Server resource.

Agent used                     : YesAdvanced Open File Option used : Yes
 

Byte count             : 0 bytesRate                   : 0.00 MB/Min
Files                  : 0Directories            : 0Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Thursday, October 27, 2011 5:24:42 PMEnd time               : Thursday, October 27, 2011 5:24:43 PM

 
Set Detail Information - Backup \\xxxxCLDB.local.com\S:
Set type               : BackupSet status             : CompletedSet description        :
Resource name          : \\xxxxCLDB.local.com\S:Logon account          : Domain AdministratorEncryption used        : None
Error                  : e000ff19 - A communications failure has occurred with a SQL Server resource.

Agent used                     : YesAdvanced Open File Option used : Yes
 

Byte count             : 0 bytesRate                   : 0.00 MB/Min
Files                  : 0Directories            : 0Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Thursday, October 27, 2011 5:24:43 PMEnd time               : Thursday, October 27, 2011 5:24:43 PM

 
Set Detail Information - Backup \\xxxxCLDB.local.com\L:
Set type               : BackupSet status             : CompletedSet description        :
Resource name          : \\xxxxCLDB.local.com\L:Logon account          : Domain AdministratorEncryption used        : None
Error                  : e000ff19 - A communications failure has occurred with a SQL Server resource.

Agent used                     : YesAdvanced Open File Option used : Yes
 

Byte count             : 0 bytesRate                   : 0.00 MB/Min
Files                  : 0Directories            : 0Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Thursday, October 27, 2011 5:24:43 PMEnd time               : Thursday, October 27, 2011 5:24:43 PM

 -------

I have tried various options like turning off AOFO, but it wont that way either. Further I have noticed that as soon as it has completed backing up the LOGs partition of SQL , the backup fails.

Any suggestions are welcome.

[newsolutionBE]

Hi

Please check the link below & ignore first point as in your case it is SQL2008 r2

but if you are getting VDI events you can give 2 & 3 point a try

http://www.symantec.com/docs/TECH124776

Thanks

[VJware]

It is best recommended to disable AOFO. Try the following using the SQL Management Studio.

- Connect to the SQL instance & access Properties - Connections.

- Change the 'Maximum number of concurrent connection' to a different value & click OK

- Go back to Connections & revert the value of 'Maximum number of concurrent connection' to the previous setting.

- Click OK & re-run the backup.

[coolhunk824u]

Both the solutions dont seem to  work for me.

I have question though, how would the backup work if AOFO is disabled ? What do you exactly mean by disabling it- i.e. to disable the option when creating a backup job, or not to select the AOFO option while installing the agent?

What I have done as of now is while installing the agent on the SQL servers, I chose the AOFO option along with the Backup Exec remote agent. And while doing the backup job, I have the AOFO option selected as well. But I still notice, the "microsoft software shadow copy provider" and " Volume Shadow copy" services are not started or running or set to automatic by default. Not sure if they remaining manual is enough, as they would be started if and when required. I tried keeping them to automatic and started , but that wont work either.

The problem I face now is when i select the passive node , it lets me chose what to backup. but when i select the active node or the database cluster name, in the selection pane it says "NONE", i.e. it wont let me choose anything. i think i will need to restart both database servers and backup server and then it might go back to normal. just not sure what is causing the issue.

[newsolutionBE]

Hi

There is no more need to install AOFO option on client machine win2003 & above

Regadring not using AOFO is in your job configuration uncheck the AOFO option if selected for the job

Also I presume you are backing the cluster from virutal node & not from seprate acitve & passive node

Also if you could upload the selection list snap & job log too wil be perfect

Thanks

[coolhunk824u]

I unistalled the agents. Reinstalled without the AOFO option. And then in the Backup job that i created, i unchecked the AOFO option. The backup still fails. and I see that after the backup fails, it seems the agent service is stopping on the remote database server. I will forward you the snapshots right away, as well as i was wondering if the debug log would be helpful. Attaching the same:

[newsolutionBE]

Hi

For Sql server if you use specifically SA account does the backup work you can also do test credential in resource credential tab

& if you want to debug then do the following

On sql server open registry & then go to hklocal machine-softwares-symantec-backupexec-engine-logging--on right hand side make create debug log key as 1 & then run the backup from media server & then once job is failed please upload the job log & beremote log present on C-program files-symantec-backupexec-raws--logs folder

Thanks

[coolhunk824u]

please find it attached here

[coolhunk824u]

This time , i tried changing the order of backup. It seems as long as the backup is going on for the other partitions , ie.. other than the data and the MS SQL server it is working but as soon as it goes on either of these two, it fails.i am attaching here the backup job failure log.

[newsolutionBE]

Hi

This is the error in the log

6508] 10/28/11 17:35:07 [fsys\sql2]          - Returning Database DMSTESTDBNV - Log backup supported
[6508] 10/28/11 17:35:07 [fsys\sql2]          - SQL2_StartCommand = DBCC CHECKDB([DMSTESTDBNV]) WITH PHYSICAL_ONLY
[6508] 10/28/11 17:35:09 [fsys\sql2]          - Returning Database IT_DMSK - Log backup supported
[6508] 10/28/11 17:35:09 [fsys\sql2]          - SQL2_StartCommand = DBCC CHECKDB([IT_DMSK]) WITH PHYSICAL_ONLY
[6508] 10/28/11 17:36:34 [fsys\sql2]          - SQL Error GUID: 1f3a000
[6508] 10/28/11 17:36:34 [fsys\sql2]          - SQL Error Description: Unspecified error

It seems it is getting failed on DBCC of your sql 2008 so please try manual DBCC of sql2008 database using studio management also see if that backup using sql studio if working fine or not

[coolhunk824u]

both seem to be working fine manually ! what next ?

[newsolutionBE]

Hi

Do you see any error in the application tab of sql server at the point of failure of backup job

[coolhunk824u]

application tab of sql server? no idea... but this is what i see in event viewer:

Log Name:      Application
Source:        Application Error
Date:          10/28/2011 8:07:37 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      GDSDDB01.RYDM.SA
Description:
Faulting application name: beremote.exe, version: 13.0.4164.109, time stamp: 0x4cffac43
Faulting module name: bedssql2.dll, version: 13.0.4164.108, time stamp: 0x4cfebc3b
Exception code: 0xc0000005
Fault offset: 0x000000000004b691
Faulting process id: 0x460
Faulting application start time: 0x01cc959397830ab7
Faulting application path: C:\Program Files\Symantec\Backup Exec\RAWS\beremote.exe
Faulting module path: C:\Program Files\Symantec\Backup Exec\RAWS\bedssql2.dll
Report Id: 55cc1c98-0187-11e1-9130-68b5996fda12
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-10-28T17:07:37.000000000Z" />
    <EventRecordID>8920</EventRecordID>
    <Channel>Application</Channel>
    <Computer>GDSDDB01.RYDM.SA</Computer>
    <Security />
  </System>
  <EventData>
    <Data>beremote.exe</Data>
    <Data>13.0.4164.109</Data>
    <Data>4cffac43</Data>
    <Data>bedssql2.dll</Data>
    <Data>13.0.4164.108</Data>
    <Data>4cfebc3b</Data>
    <Data>c0000005</Data>
    <Data>000000000004b691</Data>
    <Data>460</Data>
    <Data>01cc959397830ab7</Data>
    <Data>C:\Program Files\Symantec\Backup Exec\RAWS\beremote.exe</Data>
    <Data>C:\Program Files\Symantec\Backup Exec\RAWS\bedssql2.dll</Data>
    <Data>55cc1c98-0187-11e1-9130-68b5996fda12</Data>
  </EventData>
</Event>

[pkh]

Your job is failing on DBCC CHECKDB([IT_DMSK]) WITH PHYSICAL_ONLY which implies that your database IT_DMSK is damaged.  Exclude this database from your backup and I think you job will go throught successfully.  In the meantime, you need to consult your SQL expert on how to repair the IT_DMSK database.

[coolhunk824u]

I am not really sure here what the consistency checker does. When i dont select the option from the backup job to do a physical consistency check, the backup is successful.

If its failing, is it really bad regarding the database ?

[newsolutionBE]

Hi

Please check running DBCC from sql studio or consult your SQL admin regarding the probelm faced with DBCC so that he can analyze the issue for you

Thanks

[pkh]

The consistency check makes sure that the database is structurally correct.  If it failed the consistency check, then it is bad.   If you do not select the consistency check, it means that BE will backup the database regardless of whether it is structurally sound or not.  There is no point backing up a bad database.  You might not be able to restore it.  You should resolve this consistency check failure as soon as possible.