BE 2010 r3 Agent issue: "A communications failure has occurred"

[Pete_Eldridge]

Hi everyone, new to the forums and fairly new to BackupExec as well. Taken over a system from a previous worker who had things running quite nicely, however I've hit a snag with a file server which is baffling me.

The agent seems to have stopped working on a file server. It's a VM Guest Windows 2008 r2 server, fully updated and one of several very similar machines being backed up. It has the BE RAWS agent installed with AOFO option enabled. It was working happily until earlier this week, when it started throwing the error "A communications failure has occurred" and "Final error: 0xe000846b - The resource could not be backed up because an error occurred while connecting to the Backup Exec for Windows Servers Remote Agent". There's no apparent problems with the target server itself, it's happily on the network and can be connected to in all the usual ways. Nothing has changed aside from a round of Windows Updates being installed, which happened after the issue had begun.

Checking the Selection List, the server appears as a non-Agent machine. I can see file shares but no system state info or disks etc. Testing the Resource Credentials fails for this server, but works for all of the other resources in the Selection List, which are also agent-enabled VM Guest servers and are using the same System account

I have re-installed the agent several times, both as a local install and as a remote install from the BE server. Both appear to work fine and the BE Remote Agent Utility on the target server reports that the media server is registered for it to publish information to. I can't add the target server as a Favourite Resource on a Selection List  - the process appears to go through but the server subsequently doesn't appear in the Favourites list. I've turned off AV and the DeviceLock service on the target server, no joy.

I ran debugging and it generated the following when trying to connect to the agent (full domain redacted):

UNK:      [07/26/12 10:49:37] [0000]     07/26/12 10:49:37 [BESocket]           - @@@@@@@MyCloseSocket called with sockfd = 1556(0x614) retval = 0
UNK:      [07/26/12 10:49:37] [0000]     07/26/12 10:49:37 [ndmp\ndmpcomm]      - ndmpEstablishConnectionUsingNoSpecificAdapter: Could not connect to remote address 10.120.200.5 and port 10000 Errorno : 10061
UNK:      [07/26/12 10:49:37] [0000]     07/26/12 10:49:37 [ndmp\ndmpcomm]      - ndmpConnectEx: unable to connect using NetworkOptions to scsfs01.xxxxxx.co.uk
UNK:      [07/26/12 10:49:37] [0000]     07/26/12 10:49:37 [BESocket]           - BETCPConnection::CreateConnectionFromHostAndPort: Remote Host: "scsfs01.xxxxxx.co.uk": There were no addresses returned, belonging to family: IPv6.
UNK:      [07/26/12 10:49:38] [0000]     07/26/12 10:49:38 [BESocket]           - BETCPConnection::LoopThroughListAndConnect: Could not connect to remote address "10.120.200.5" Error:10061.
UNK:      [07/26/12 10:49:38] [0000]     07/26/12 10:49:38 [BESocket]           - @@@@@@@MyCloseSocket called with sockfd = 1556(0x614) retval = 0
UNK:      [07/26/12 10:49:38] [0000]     07/26/12 10:49:38 [BESocket]           - BETCPConnection::CreateConnectionFromHostAndPort: Could not create a connection to "scsfs01.xxxxxx.co.uk" because attempts with both IPv4 and IPv6 protocols failed
UNK:      [07/26/12 10:49:38] [0000]     07/26/12 10:49:38 [ndmp\ndmpcomm]      - ndmpConnectEx: All attempts to connect to 'scsfs01.xxxxxx.co.uk' failed
UNK:      [07/26/12 10:49:38] [0000]     07/26/12 10:49:38 [ndmp\ndmpcomm]      - ndmpConnectEx: failed to connect to 'scsfs01.xxxxxx.co.uk'
UNK:      [07/26/12 10:49:38] [0000]     07/26/12 10:49:38 [fsys\shared]        - FS_ResolveDevName: [\\scssql01.xxxxxx.co.uk\D:]

Just can't get the BE server to recognise it any more! Any thoughts? Anything else I can do to get it to pick up?

Any help much appreciated,

Pete Eldridge

[CraigV]

Hi,

Make sure that the RAWS agent is running on the file server. You should also make sure that if any firewall is running on that server, that port 10000 is open. This allows the communication between the agent & the media server.

Thanks!

[AmolB]

Have you tried adding IP address of the remote server under "User Defined Selection"

On the remote server make sure port#10000 is used by BEremote.exe 

EDIT: Refer to http://www.symantec.com/docs/TECH182405

[Pete_Eldridge]

Thanks for the quick responses guys!

@CraigV: Windows firewall is off so no issue there, Trend AV also doesn't run the FW so I don't think that's the issue. Will double check that though. Agent is definitely running on on the file server.

@ Amol: Will try the IP route. How do I ensure BEremote.exe is uing port#10000? It hasn't been changed to my knowledge so should be using the default (which I gather #10000 is). As mentioned to CraigV, no FW is running and I've tried it with AV disabled as well, no luck either way.

[AmolB]

On the remote server use netstat command to check which application is using port#10000

[Pete_Eldridge]

Quick follow up: Adding the IP in User Defined Selections makes no difference. I get the same behaviour - it looks like it's been successful but can't connect to the device and there's the same communications error.

There's no firewalls running and the AV isn't affecting it. Other similar machines with the AV turned on are not affected by this issue.

However Netstat isn't reporting port#10000 as being in use, which makes me wonder if it has been changed from the default. I'll look into that side of things.

EDIT: I've been testing further and I can Telnet to all the other servers in the Selection List on port #10,000, however the file server in question is refusing a Telnet connection on that port. WOuld appear that the Agent isn't installing properly or isn't listening on that port. But it does indicate that the Agent should be listening on port 10,000. Will try and do a better uninstall/cleanup/re-install of the agent and see how far that gets me.

Thanks again for the help guys.

[CraigV]

Pete: Check the link below and see if changing the port will help:

http://www.symantec.com/business/support/index?page=content&id=TECH24256

[Pete_Eldridge]

Thanks Craig, that was a useful document. There's nothing in the Drivers\ETC\Services list that's clashing, and NETSTAT on the file server isn't reporting anything as using port 10000, so it would appear I'm OK on that front.

I'm a little reluctant to change the port being used by the server as it's working fine on the other 20-odd servers in our farm, seems a little risky to make that change for a single finicky server.

The question really is then, how do I get the BE Agent to kick in on this server and start listening on the port? The install appears fine, I've run the BE_SupportTool.exe program and get a few errors, but they're the same as I get on other, working machines. Nothing in Event Logs, I've put the machine in an AD Group Policy area that has a totally blank policy, meaning it's not being externally controlled by that. Firewall & AV are completely turned off. It's like a firewall or something is blocking it, but there's nothing turned on to do that. Confusing.

Might be time for a support call to Symantec.

[CraigV]

It looks like you have eliminated the usual suspects (firewall, service, AV)...if you put in that server's IP address in a custom selection list and browse to it, does it show up?

If you browse in a selection via the Active Directory domains...?

[Pete_Eldridge]

Hi Craig, when browsing by AD domain the server shows up as a normal network resource, shares are browsable but no system resources are visible (system state, root drives, shadow copy componants etc). Other machines with the agent working do display this info.

P.

[CraigV]

...and can you ping/nslookup from the media server to that affected server via name/IP address successfully, and vice versa from the affected server to the media server?

Otherwise it might be time to log a support call with Symantec, and if they assist you, you can close it off here.

[Pete_Eldridge]

Hi Craig, yes, ping, nslookup etc all working fine from either server. No impaired network functionality that I can detect, file server is behaving perfectly normally for the users attached.

I've logged a call, will report back and close the thread when it's resolved. Thanks for your help!

P