BE12.5, Server 2003 w/ RD1000, e0008821 and Verify Failures

[SRupp]

A few weeks ago, my BE12.5 nightly backup began failing.

This is a Windows 2003 Server. The backup was previously working without issue backing up to RD1000 cartridges. I have temporarily connected a 500GB USB drive for troubleshooting to the system.

The Job History States:

Error category    : Job Errors
Error: e0008821 - Job was recovered as a result of Backup Exec RPC service starting.  No user action is required.
For additional information regarding this error refer to link V-79-57344-34849

I reviewed the information at http://www.symantec.com/business/support/knowledge_base_sli.jsp?umi=EvtID=V-79-57344-34849

We can rule out:

1. It worked previously so slipstream would likely not be a valid possibility. 
2. The Job Log is not too big.
3. There is no Domino Server.
4. RSM is not running
5. Automatic Cancellation is enabled but the backup never exceeds the 8 hour window.
6. There is no tape drive, so updating tape device drivers is not applicable.

I went ahead and did basic troubleshooting on the issue based on other issues I have seen on other servers. Started with the typical Reboot and Restart Services. I Unistalled/Reinstalled 12.5 and applied all Symantec Updates. (Perhaps overkill at the outset but I figured I'd get it out of the way early.) Rebuilt Backup Selections, Job and Devices. The RD1000 is configured as a Removable Backup-to-Disk Folder. My USB HD is set as a standard Backup-to-disk folder. I applied all MS Updates to Windows. I have formatted the RD1000 cartridges and have tried many different, known-good cartridges.

When I babysit the backup job, the RD1000 completes the backup, begins the verify and then fails. It does not matter if I disable the "Verify" option on the job, it still fails. I turned "verify" back on.

I retarged the job to use the USB drive and got a successful backup. I then switched back to the RD1000 and the issue returned immediately.

I have Googled the daylights out of this and reviewed countless forums and have not found any solid leads to resolve this. I'd like to actually find an answer so others like myself will actually find a solution to this generic and frustrating issue.

Here is a full job log...

--------------------------------------------------------------------------------
Job History for FS-DC1 Full Backup
--------------------------------------------------------------------------------

FS-DC1 Full Backup
Job name            : FS-DC1 Full BackupJob type            : BackupJob status          : FailedJob log             : C:\Program Files\Symantec\Backup Exec\Data\BEX_FS-DC1_00015.xmlServer name         : FS-DC1Selection list name : Backup
Device name         : RD1000Target name         : RD1000Media set name      : Media
All Media UsedB2D000023B2D000022
Error category    : Job ErrorsError             : e0008821 - Job was recovered as a result of Backup Exec RPC service starting.  No user action is required.
For additional information regarding this error refer to link V-79-57344-34849
 
Job Summary Information
Byte count          : 9,874,650,214 bytesJob rate            : 0.00 MB/Min (Byte count of all backup sets divided by Elapsed time for all backup sets)
Files               : 24,075Directories         : 3,671Skipped files       : 0Corrupt files       : 0Files in use        : 0
Original start time : Monday, January 25, 2010 11:00:00 PMJob started         : Monday, January 25, 2010 11:00:01 PMJob ended           : Monday, January 25, 2010 11:00:01 PMElapsed time        : 00:00:00
 
Set Detail Information - Backup \\FS-DC1\Utility Partition
Set type               : BackupSet status             : RunningSet description        :
Resource name          : \\FS-DC1\Utility PartitionLogon account          : System Logon AccountEncryption used        : None
Agent used                     : YesAdvanced Open File Option used : Yes
 
Start time             : Monday, January 25, 2010 11:00:01 PM

Set Detail Information - Backup \\FS-DC1\Shadow?Copy?Components
Set type               : BackupSet status             : RunningSet description        :
Resource name          : \\FS-DC1\Shadow?Copy?ComponentsLogon account          : System Logon AccountEncryption used        : None
Agent used                     : YesAdvanced Open File Option used : Yes
 
Start time             : Monday, January 25, 2010 11:00:01 PM

Set Detail Information - Backup \\FS-DC1\System?State
Set type               : BackupSet status             : RunningSet description        :
Resource name          : \\FS-DC1\System?StateLogon account          : System Logon AccountEncryption used        : None
Agent used                     : YesAdvanced Open File Option used : Yes

Start time             : Monday, January 25, 2010 11:00:01 PM

Set Detail Information - Backup \\FS-DC1\C:
Set type               : BackupSet status             : CompletedSet description        : FS-DC1 Full Backup
Resource name          : \\FS-DC1\C:Logon account          : System Logon AccountEncryption used        : None
Agent used                     : YesAdvanced Open File Option used : Microsoft Volume Shadow Copy Service (VSS)

Byte count             : 9,874,650,214 bytesRate                   : 169.00 MB/Min
Files                  : 24,075Directories            : 3,671Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Monday, January 25, 2010 11:00:10 PMEnd time               : Monday, January 25, 2010 11:55:38 PM
Media used             : B2D000022
 
Set Detail Information - Backup \\FS-DC1\F:
Set type               : BackupSet status             : RunningSet description        : FS-DC1 Full Backup
Resource name          : \\FS-DC1\F:Logon account          : System Logon AccountEncryption used        : None
Agent used                     : YesAdvanced Open File Option used : Microsoft Volume Shadow Copy Service (VSS)

Start time             : Monday, January 25, 2010 11:55:41 PM
Media used             : B2D000022Media used             : B2D000023
--------------------------------------------------------------------------------

Any takers for assistance?

Regards,
Stephen

[vinanteg]

Do you have any pre/post commands configured for the job?

Is there any evidence in Windows Event Viewer of some service being suddenly stopped?

Regards,

[CarolineKiel]

Hi Stephen,
as stated in TechNote 305043 unter http://seer.entsupport.symantec.com/docs/305043.htm you do perhaps need a firmware upgrade for your RD1000. It seems to be a known problem...

Hope this helps,
Oliver

[SRupp]

Thank you for the response Vinanteg, to answer your question I do not have any pre/post commands configured. I have spot checked my event viewer looking for any entries out of he ordinary but nothing presented itself. I will double check again tonight after I try it again.

Thank you also Oliver, I will review the firmware information and see if that is my culprit.

I will report back!

Best Regards,
Stephen