Backup Job Disappears, Reappears with error "e0008821 - Job was recovered as a result of Backup Exec RPC service starting. No user action is required."

[kidtrebor]

Windows Server 2008 R2 64-bit

Backup Exec 2010 Media Server 13.0 Rev. 2896

Here's what happened:

• We have 3 backup to disk jobs configured on our central administration server
• Yesterday only 1 of them returned an alert message overnight that it completed
• I checked whats going on on the server > Job Monitor tab and sure enough only 1 of the 3 could be seen in the job history (all 3 jobs issued a result the day before that)
• Surprisingly, in the Current Job list only 1 of them was listed as being scheduled to run tonight; other 2 were missing, as if they weren't scheduled
• Checked the Job Setup tab and all 3 were listed; tried to double click on the 2 missing jobs but could only view the properties in read-only mode, as if they were currently running (3rd job's properties could be opened fine)
• No unusual events in Windows event viewer
• Restarted Backup Exec services and launched the application again - missing jobs appeared in the Current jobs list again, correctly sceduled for tonight
• They also appeared in the Job History list with status failed, Elapsed Time 0:00:00 and error code:

e0008821 - Job was recovered as a result of Backup Exec RPC service starting.  No user action is required.

I cant see any events or clues as to what happened, could somone point me in the right direction?

Regards,

Robert

[CraigV]

Hi Robert,

Check the TN below:

http://www.symantec.com/business/support/index?page=content&id=TECH49428

I'd also recommend upgrading to BE 2010 R3 with SP3 as this fixed a number of issues with the previous 2 versions.

You can use your current license keys too...the ISO and SP3 are on the links below. Once upgraded with SP3, you'd run LiveUpdate to see if there are any subsequent patches, and then push-install the updates to any remote servers you might have. Just grab copies of the Data/Catalogs folders before doing the upgrade, if you so choose:

http://trialware.norton.com/files/fc/Backup_Exec_2010_13.0.5204_MultiPlatforms_Multilingual_DVD.iso

http://www.symantec.com/docs/TECH208601

THanks!

[kidtrebor]

 1.  Check the Application Event log. If there is an error on RSM, disable the service from Start-Run-Services.msc and try the backup again.

Sorry, what is meant by RSM?
 

 2. This error may sometimes occur if the Automatic Cancellation feature has been enabled for a short duration. To change this duration refer to the Related Documents section. 

Automatic Cancellation not enabled
 

 3. Install the latest device drivers for the tape device:
 

http://www.symantec.com/business/support/index?page=content&id=TECH6098

Pending exhaution of all other options!!

4. If it appears to happen after the backup job is finished, check to see if email notifications are enabled and if the job logs are being sent with the emails. If so, the error above can occur because job log is larger then the maximum attachment size or the job log is not being processed properly.

It appears to be happening while job is running, so I suppose we can discount this possibility, right?

5. Run LiveUpdate and make sure there are no pending hotfixes. Update the Remote Agent if any updates are found and downloaded.

Yes, there are pending updates but I would like to exhaust other troubleshooting options since this would take weeks to get approval

6. Schedule the job to run at another time to rule out possible third party application interference. Any applications that utilize Volume Shadow Copy will cause a conflict with Backup Exec and may have to be temporarily disabled during the backup window, or removed from the server.

I don't think this is a factor since the issue is happening very infrequently

7. Perform a repair installation of Backup Exec.

Again, I would prefer this to be a last resort since it will take a long time to approve

8. Disable or reschedule any tasks responsible for rebooting the server during the backup window.

No such tasks exist

9. Packets being generated from that server may be overloading the NIC, causing the driver failure and the Backup Exec services to crash.

Disable Large Send Offload in the NIC configuration, update or reinstall the NIC driver, update the NIC firmware, or replace the NIC.

Is there any way to further narrow down if this is a factor or not?  Replacing the NIC is not something we could do without knowing it was at fault

10. If a Windows 2003 computer is locked or is configured with a password-protected screen saver AND a shutdown operation is initiated automatically by a program that uses the InitiateSystemShutdownEx function with the force flag, look for the following Event ID and if it is found, apply the Microsoft hotfix below:

OS is 2008R2 so this one isn't relevant

11. Add the IP address of the media server in the remote agent publishing tab to rule out mismatched DNS name resolution. This issue is very common in a NLB environment that uses a multiple IP's or/and multiple network adapters.

127.0.0.1 is present

12. This error can occur if the DCOM security permissions for a specific application is set to customize. Verify Event ID 10016 has occurred in the Event Viewer\System Events log.

Last occurrence is too long ago to be a match

[kidtrebor]

Hi Craig,

One other thing: to install SP3 it looks like the minimum requirement is "Backup Exec 2010 revision 5204" but our current installed revision number is 2896 (taken from Help > About; Media Server version 13.0 Rev. 2896).

Do we have to run Live Update first?  What's the correct update path?  I can see the currently installed updates are:

• Service Pack 1
• Hotfix 358409
• Hotfix 354875
• Hotfix 138449
• Hotfix 354913
• Hotfix 141686
• Hotfix 153396
• Hotfix 162860

Regards,

Robert

[Sush---]

Hello Robert,

 Upgrade to BE 2010 R3 which is the latest version. Your existing licence keys of BE 2010 R1 will work for BE 2010 R3 too so there is no need to buy new license keys.  You can download BE 2010 R3 from either the fileconnect site or

http://trialware.norton.com/files/fc/Backup_Exec_2010_13.0.5204_MultiPlatforms_Multilingual_DVD.iso 

So for R3 the latest SP is SP4 which is released few days back and still not on Live Updates. You can download and install SP4 from following link:

http://www.symantec.com/docs/TECH208828 :Backup Exec 2010 R3 revision 5204 Service Pack 4

Regards,

-Sush...

[CraigV]

...and to add to the above, if you have remote servers, you have to push-install the RAWS agent to these servers after upgrading to BE 2010 R3 with SP4...otherwise you're going to get exceptions and/or failures in your backup jobs.

Thanks!

[pkh]

I suppose I should be flattered to be imitated twice in one day, but I am not.

See my comment below for a comparision.

https://www-secure.symantec.com/connect/forums/backup-exec-tape-help#comment-9646641

Also see my other comment.

https://www-secure.symantec.com/connect/forums/cannot-start-backup-exec-2010-server-service#comment-9763411

[CraigV]

...would you mind editing everything from this post that wasn't mentioned by me on the 24th January 2014? Please read previous posts to make sure you don't post redundant, and already advised, information...

Thanks!

[Sush---]

Hey CraigV,

   Sorry somehow I missed that upgrade was already suggested here. my bad.... Sorry its not allowing me to edit the post.... may be you can edit it for me. I completely agree on your point here.

But I am not sure why the other guy is trying to press his point so high. He may not be understading that here it is point of providing the link and not checking anyone else sentence creation or English Grammer.

With all due respect to you CraigV please edit my above post.

Regards,

-Sush...

[kidtrebor]

Hi Craig,

What are the implications of doing the upgrade in terms of existing jobs, alerts, recipients, media sets, connection between Central Admin Server and branch?

Is this an update or more like a new product?

This might sound daft, but it sounds like more work than I have reasonably the time for.

[pkh]

If my previous answer is too subtle for you to grasp, then I will make it clear for you.  DON'T PLAGARISE MY PREVIOUS COMMENTS.  It is quite unbecoming for a Symantec employee to do such things.

[CraigV · Accepted solution]

MOving from R1 to R3 is straight-forward. Stop the BE services, grab copies of the Data/Catalogs folders and then perform the upgrade. Patch with the latest SP and any subsequent patches before push-installing these agents to your remote servers.

Since this is a CASO/MMS environment you have to start with the CASO first. I've done this in an environment from R1 --> R2, and then R2 --> R3. From my side, with slow WAN links, I copied the ISO of BE out to the sites (there were 28 at 1 stage) along with the latest SP. Upgraded the CASO, upgraded the MMS servers as soon as I could, and the then ran LiveUpdate from each site.

Check the TN below for more information:

http://www.symantec.com/business/support/index?page=content&id=HOWTO23417

Thanks!