e00081d9 error with all backups since power outage

[Fairholme]

Hi,

We've just started using Backup Exec recently, with a new Dell TL2000 tape library. After an overnight power outage, all jobs appear to end with this error. 

e00081d9 The job failed with the following error: The Backup Exec job engine system service is not responding.

I have tried the following, under advice from blog posts, Symantec technotes and such:

* A repair install.

* Repairing the database with beutility

* installing all hotfixes from LiveUpdate

* Creating new jobs

* Using disk instead of tape storage.

* Removing the catalog files

In all cases, the backup proceeds at a normal pace, and changes to the status: completed. Then the job sits there for 5-7 minutes and changes to stalled. Then after a total of about 15 minutes, the job auto cancels with the error as above. This happens in disk storage as well as tape. It also attempts to re-run the job.

I'm running put of ideas and coming up blank here. This is Backup exec 14.0 Rev 1798 and the server is 64-bit Windows Server 2008 R2. I'm backing up locally stored data. In addition, the diplay error where tested credentials state they have failed has also appeared, seemingly at the same time.

[pkh]

Have you tried going to Add/Remove Programs and repairing BE itself?

[Fairholme]

Yes, as stated, the first thing I did was a repair install.

[VJware · Accepted solution]

Try this on the media server ~

• On the Backup Exec media server use the Windows Registry Editor (regedit.exe) to  locate HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Server
• Create a new DWORD value called MaxRpcDatablockSize
• Set the value data to 5242880 (Note: this is a decimal value, default setting is 1048576)
• Restart the Backup Exec services.

Rerun the backup jobs now.

In addition to the above, any errors in the Event Viewer. Are all Backup Exec services running along with their dependencies ? Lastly, under Task Manager - Processes, there should be only 1 instance of each Backup Exec service running.

[Fairholme]

Well, that registry alteration seems to have fixed things. But since I wrote up the rest of the information you wanted while waiting for it to execute, I might as well post it. My tiny little test job has run with no issues, I'll just get a bigger job running overnight and see how it goes.

The only errors I am getting are 34114 and 34113. Nothing else appears correlated to the problem. The 34113 is the job engine error, and 34114 states that the user 'recovery' has cancelled the job.

No duplicates in task manager as far as I can see.

Services running are: 

Agent Browser

Device and Media Service

Error Recording Service

Job Engine

Management Service

Remote Agent for Windows

Server

I have created a very simple single folder ~500mb backup job to test this issue with so I can fairly easily re-run it and check. The backup itself only takes about a minute to start, then the transfer only another 30 seconds. Then for whatever reason the backup can't terminate.

[Fairholme]

OK, that's working with proper jobs as well now.

Do you mind explaining what that setting is and how it fixed the issue?

[VJware]

This setting allows for a larger incoming RPC data size.

Why it occurs, not really sure (most cases, the reason was due to high number of shares, though may be other reasons too). We have a KB for this  - http://www.symantec.com/business/support/index?page=content&id=TECH188880

[Fairholme]

OK, that's accurate as the server being backed up is the main file server with many shares for individual users. Probably around 1000-1500 total.

Edit: Actually this KB describes our issue and fix perfectly: http://www.symantec.com/business/support/index?page=content&id=TECH201369

We just created about 4 new users/shares so we must have hit a tipping point.