Jobs failing with 0xe000fe28 - A network connection to the server could not be established.

[soca-boi]

All,

In hopes that someone can offer some assistance with a very frustrating issue. Have a CASO environment with about 8 Media servers and from time to time my jobs fail with the error:

Error: 0xe000fe28 - A network connection to the server could not be established.

When i get in the morning I see that it failed and can run it without issue. I thought it was a firewall issue (traffic goes over firewall... Ugh) so we opened port 1025 to 65535. They helped somewhat but everyday some jobs fail with the error.  Noticed that trying to browse the job selection from BE the dialog just sits here. See attachment. In the firewall I am seeing errors saying "First packet isn't SYN". I have talked to the network guys and they say that a timeout is happening.

Contacted support and they suggested creating a registry entry called KeepAliveTime and setting it to 900000. That hasn't helped and support is out of ideas (not surprised). I cannot run my backups in this manner. Can someone help me out?

Very Frustrated

[EliasA]

Can you send me your support case # so that I can follow up and take a look at the details of your support call?
Thank you.

Elias

[soca-boi]

Sure! #21364686.I decided to close the ticket because they wanted me to open another one specific for this error. I think it boils down to communication issues with BE for both errors. Thanks for responding.

[EliasA]

Thank you for the response.  Are you still encountering an issue you need assistance with?  As I did not see a related/new support case opened.

WIth your installation of Backup Exec 2014, do you have the latest patches installed?

As you are indicating this is intermittent, is there a specific time / day where the jobs work more successfully than others?  Some jobs more successful than others?  Is this happening on one of the media servers more than another?  Just trying to narrow down the issue to help in troubleshooting...

Elias

[soca-boi]

Yes I am. I just haven't open the other ticket yet. Yes running latest patch (SP2). It doesn't seem to matter what time of day. But I can't seem to replicate the error. If I run the failed job during the day most likely it will work. ....sometimes it will fail. It happens to Exchange, File Server or Domain Controller Backups (essentially all jobs). Doesn't matter which media server. Seems to happen with jobs that cross the firewall. I checked the firewall and not seeing any packet drops or denies for the traffic. We have allowed the entire dynamic range 1025 - 65535.

Thank again

[EliasA]

Thanks for the update on the current situation.  Definately an odd situation you are running into.  So this happens more (only?) with jobs that cross the firewall, but there are no packet drops based on the logs.  Thanks for checking that.

Any firewall level rules in place that could be related?  So if you have a backup job that does not cross the firewall it works every time?  Permission related? (but that would seem odd why it sometimes works and sometimes doesn't)

Do you have any anti-virus software installed/enabled that can be causing the delay in files moving across the firewall? 

I would suggest to go ahead and open up a support case to have an advanced support technician get on the phone and troubleshoot.  This doesn't sounds like the quick one shot do this and it'll fix the issue, I wish it was. :)

Sorry I could not be of more immediate help - but once you do open the case, please let me know the case # and I will follow it up for you.

[soca-boi]

Again really appreciate it. I think it's something with the agent. For the entire week I wasn't able to back up a certain resource. I would get that same error over and over. No blocks in firewall nothing. Even ran Wireshark and was able to see that all packets being sent were received. I decided to restart the agent which was hung as always. Couldn't browse selections on the resource. I restarted the agent and the job worked last night... WTH?!

I disabled anti-virus, disabled malware-bytes.... no difference.

Going to open ticket

Thanks again.

[soca-boi]

Just posting an update:

Made some progress with troubleshooting. On the domain controller jobs that are failing I decided to break out the job into two. Was backing up server groups (in order to minimize job count) before. Instead I have created seperate jobs. Since then I've had better results. I think there was some timeout happening when BE opens a connection to the remote agent and then kept it opened while backing up the other resouce in the group. If this is the work around I am fine with that.

Now to figure out why i get the  same error with my Exchange jobs. I have a seperate job to backup each DB. Hopefully there is something to address this.