Final error: 0xe00084f8 - The network connection to the Backup Exec Remote Agent has been lost. Check for network errors. Final error category: Resource Errors

[MKatzung]

I’m getting this error message from BE when attempting to run jobs on two separate servers.

The jobs are attempting to backup SQL server databases

Final error: 0xe00084f8 - The network connection to the Backup Exec Remote Agent has been lost. Check for network errors. Final error category: Resource Errors

This error happens on the Full and Daily Differential backups, not the log backups.

This happens when initiated manually or when run automatically

So background on the servers/other info

One is a physical machine, one runs on VMware ESXi 5.1

Both have 4 gigs of ram and similar CPU (8 cores)

OS: Windows 2003 R2

SQL Server: 2005 SP4

BE: 2010 R2

Job size is 250 gigs for one server, 300 gigs for another (We have several TB size databases that do not fail)

When failing on a full backup fails significantly % wise, 250 gig db only hits about 40-60gigs

Backups have their own dedicated network with 1gig access

Permissions are not an issue, tested via elevation

I've noticed that after a time if I'm watching the job run, the byte count will just stop increasing, it will fail 5-10 minutes after that.

So I’ll go through all of the things I’ve done thus far to try and solve this problem

-Restarted the be agent

-Restarted the server

-Uninstalled/Reinstalled the be agent

-Upgraded to SQL Server 2005 SP4 (This seemed to fix the problem for a couple of days)

-Originally these databases and servers were part of a large backup job with a number of SQL databases they have been since isolated and separate jobs created

--Tested each database individually and it appears size related (maybe?), as the larger databases fail, but the smaller ones grouped together also cause a fail.

-netstat –a shows nothing on port 10000

I get 4 or 5 error messages in the event log on the agent’s server, these pop up around the time the backup is running and are likely related. Here is the output from the latest attempt to run the job. (I’ve removed what I think might be sensitive info with ****)

---

Event Type:        Error

Event Source:    SQLVDI

Event Category:                None

Event ID:              1

Date:                     6/3/2014

Time:                     11:56:30 AM

User:                     N/A

Computer:          ****

Description: SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=4572. Thread=4492. Client. Instance=. VD=Global\****_00__215eddd0_1765_4cb6_978b_815fbd697760__SQLVDIMemoryName_0.

---

Event Type:        Error

Event Source:    SQLVDI

Event Category:                None

Event ID:              1

Date:                     6/3/2014

Time:                     11:56:30 AM

User:                     N/A

Computer:          ****

Description:SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1672. Thread=944. Server. Instance=MSSQLSERVER. VD=Global\****_00__215eddd0_1765_4cb6_978b_815fbd697760__SQLVDIMemoryName_0.

--

Event Type:        Error

Event Source:    MSSQLSERVER

Event Category:                (6)

Event ID:              3041

Date:                     6/3/2014

Time:                     11:56:30 AM

User:                     ****

Computer:          ****

Description: BACKUP failed to complete the command BACKUP DATABASE **** WITH DIFFERENTIAL. Check the backup application log for detailed messages.

--

Event Type:        Error

Event Source:    MSSQLSERVER

Event Category:                (2)

Event ID:              18210

Date:                     6/3/2014

Time:                     11:56:30 AM

User:                     ****

Computer:          ****

Description: BackupVirtualDeviceFile::ClearError:  failure on backup device '****_00__215eddd0_1765_4cb6_978b_815fbd697760_'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).

---

Event Type:        Error

Event Source:    MSSQLSERVER

Event Category:                (2)

Event ID:              18210

Date:                     6/3/2014

Time:                     11:56:30 AM

User:                     ****

Computer:          ****

Description: BackupVirtualDeviceFile::RequestDurableMedia: Flush failure on backup device '****_00__215eddd0_1765_4cb6_978b_815fbd697760_'. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).

--

I’ve got the last bit of output from SGMon from a job I ran on Friday and truncated the log down to around the time the job failed if that would be helpful.

I’m new to systems administration and none of my coursework in the past covered third party backup applications. So please any help would be appreciated.

Also I know very little about SQL functionality so if I need to run queries I’ll need some help with that.

Thanks

[lmosla]

Hello,

 Make sure the Backup Exec processes are excluded from any antivirus scans.

 Is AOFO enabled in the job?  If it is disable it and run the job again.

also, make sure utility jobs or another third party application isnt running during the backup job.

[MKatzung]

Specifically which processes? I've added an exeption for beremote.exe from the Endpoint protection client and am still having this issue

AOFO is off.

The backup of the log file is the only thing I'm aware of that is accessing these servers besides the biztalk production application we use. Even when I pause the log file backup it fails.

[pkh · Accepted solution]

ESX 5.1 requires BE 2010 R3 with SP3.  See the SCL below

http://www.symantec.com/business/support/index?page=content&id=TECH137682&key=15047

To upgrade to BE 2010 R3, you can download BE 2010 R3 from either the fileconnect site or

http://trialware.norton.com/files/fc/Backup_Exec_2010_13.0.5204_MultiPlatforms_Multilingual_DVD.iso

Your existing licence keys will work.

After your upgrade, do not forget to run LiveUpdate a couple of time to update it to SP4 and the latest hotfixes.

After you have upgraded to R3 SP4, push out the remote agent again

[MKatzung]

One is a physical box the other is a VM

This problem is unrelated to VMware

[Colin_Weaver]

We stopped updating 2010 R2 ages ago (more or less immediately after R3 was released because R3 uses the same software licenses as R2.)

As such there are likely to be changes in R3 (both the initial release and the later Service Packs) that never made it to R2 and never will.

As such the recommendation to update to R3 (plus apply latest Servcie packs etc) is a valid one even if the justification against Vmware versions is perhaps invalid.

2010 R3 downlaod is here: http://trialware.norton.com/files/fc/Backup_Exec_2010_13.0.5204_MultiPlatforms_Multilingual_DVD.iso

As of Monday's release of BE 2014 we of course would now recommend this version - but you will need updated licenses which might will mean additional costs unless you have valid maintenance agreements in place. You should also carefully check the compatibility lists before going to BE 2014 just in case you are running something so old it is no longer supported.

[MKatzung]

Alright, I'll work to upgrade but will need some time to clear with management. Please do not close this thread, the issue is not solved yet. I will reply back once it is installed and continues to not function :)

[Colin_Weaver]

Just to be clear I am not guaranteeing that 2010 R3 with latest Service pack will fix your condition, I am just recommending an update because your software is very out of date

Also if you are talking to your management you should possibly mention that Backup Exec 2010 R3 will very soon  also be subject to limited (partial) support and no further software patches from Symantec as it itself is now 2 versions old.

[MKatzung]

You're suggesting it's not necessarily going to fix the problem and I honestly agree. We have other 2003 servers with SQL2005 instances that back up just fine.  I'm actually MORE concerned that updating will cause even more headaches for me for servers that are functioning fine now, but will be broken by the upgrade.

What else can I do to troubleshoot this issue without upgrading the software?

[VJware]

BE 2010 R3 was the final release of BE 2010 version and is more reliable, consistent than it's predecessor. Would recommend to keep a copy of Data & Catalog folders prior to the upgrade and it should be OK.

Description: BackupVirtualDeviceFile::RequestDurableMedia: Flush failure on backup device usually refers to low virtual memory.

Are you able to backup the DBs successfully using SQL native backups itself ? Lastly, without upgrading, would need to have a look @ the debug logs for troubleshooting.

[MKatzung]

Updating to R3 SP4 appears to have resolved the issue.

If it's okay, I'd like to hold off on marking this officially solved... When I updated to SQL2005 SP4 the jobs functioned for about a week or two and started failing again.

I'll report back on the 16th and let you know if the jobs are still working as this will give me 2 full backups and a weeks worth of differential backups.

Thanks for the assistance.

[MKatzung]

I believe I spoke to soon, I ran the differential again as a test and both jobs failed.

[MKatzung]

It's looking like the log file backups were some how causing an issue with these backups. I noticed the jobs would stop increasing in size when the log jobs ran.

These were part of larger backup jobs and I've split them into their own individual jobs that have a much shorter run time. I've encounter no failed jobs since splitting them off but I'll know for sure 100% the next time the full runs tomorrow night.