Communications failure with the DMZ

Martin-B · ‎12-08-2014

We are using Symantec Backup Exec 2014 installed on a Windows Server 2012 platform. All backups work without issue (normally) and we are only experiencing issues with those servers in our DMZ.

The DMZ servers (neither virtual ones or the physical ones) would not accept the agent via push from the BUE console. We opened the firewall so these servers could communicate using any port in both directions with the BUE server. The issue appeared to be UAC so we removed that via the registry and it allowed us to install the agent, establish trust and setup a job which worked without issue.

We come in the following day and it cannot establish a connection - the very same issue again. No changes have been made since yesterday, yet it cannot communicate with the DMZ server.

I can ping successfully in both directions with both IP address and the servers name. I can also navigate to the c$ drive on the server if I provide the right credentials. I am using the same credentials in BUE and this also works in both directions.

So my question: why do the credentials work when navigating manually, the network and DNS etc all works ok, ports are open but something is preventing communication from within BUE 2014. What? Is it a UAC issue and has anyone an idea of how to correct it? The servers in the DMZ are a mix of Windows Server 2003, 2008 R2 and 2012 R2 versions, but all report exactly the same failure.

We have tried installing the agent on the target server, but this does not change the communication error between the servers.

CraigV · ‎12-08-2014

...you need to open Port 10000 on any firewall. This is used by BE to communicate to a remote server's RAWS.

Thanks!

Martin-B · ‎12-08-2014

All ports are open for all communication on that address range in both directions.

From my post: "We opened the firewall so these servers could communicate using any port in both directions with the BUE server. "

CraigV · ‎12-08-2014

...while i understand that, my intention was to see if this port needed to be opened explicitly.

Thanks!

Martin-B · ‎12-08-2014

Sure. I have used netstat -an | find "10000" to ensure the port is open. I get a successful result from both servers.

VJware · ‎12-08-2014

Would you post the push install log ? Thanks.

Colin_Weaver · ‎12-09-2014

Is the firewall also handling a NAT configuration between the media server and the DMZ? If it is then because remote agent publishing is in the opposite direction it it possible some elemenst of the backup process will not work, although this should not stop a basic file system backup from running.

Also do you know if the trusts have been setup as a missing trust (between RAWS agent and Media server) will cause authentication style issues.

Martin-R · ‎12-17-2014

Not sure what has happened. Had to cerate a new account here.

Below is the latest failure log. There have been a few developments: on investigating more and setting Windows UAC off in the registry, the agent can install, and backups can begin. The failure is when the backup has been running for a few minutes (including transferring data), but the connection is lost and this cancels the backup set.

Job Log for server1.location.dmz Backup 00159-Full

Completed status: Failed See error(s)

Job Information
Job server: RRLIBACKUP Job name: server1.location.dmz Backup 00159-Full Job started: 17 December 2014 at 11:35:36 Job type: Backup Job Log: BEX_RRLIBACKUP_06109.xml Job Backup Method: Full

Backup Exec will not follow mount points, junction points, and symbolic links when the Simplified Disaster Recovery indicator is ON.

Drive and media mount requested: 17/12/2014 11:35:39

Device and Media Information

Drive and media information from media mount: 17/12/2014 11:37:59
Drive Name: Deduplication disk storage 0001:1
Media Label: OST00001213-45D3195C02B6CAD7
Media GUID: {13cb5232-9469-4aa0-bb10-17ed5f4a81ba}


All Media Used
 OST00001213-45D3195C02B6CAD7

 OST00001245-45D3195C02B6CAD7

Job Operation - Backup

Backup Options

Backup Set Retention Period: 2 Weeks.
Compression Type: Hardware [if available, otherwise none]

Encryption Type: None

WARNING: The option 'Verify after backup completes' was not selected.

Performing a verify operation to make sure that media can be read after the backup has completed is recommended.

Server -server1.location.dmz

Backup Exec server is running Backup Exec version 14.1.1786.1068 with SP-1,HF-225092.

Agent for Windows(server1.location.dmz) is running Backup Exec version 14.1.1786.1068 with HF-218257,SP-1,HF-225092.

The following resources that were selected for backup can be used for future conversion-to-virtual jobs: 'C:, System?State' 

Snapshot Technology: Started for resource: "\\server1.location.dmz\C:". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS).
The snapshot technology used by VSS for volume C: - Microsoft Software Shadow Copy provider 1.0 (Version 1.0.0.7).
Network control connection is established between 172.19.24.157:63176 <--> 172.16.1.22:10000
Network data connection is established between    172.19.24.157:63281 <--> 172.16.1.22:65347

Set Information - \\server1.location.dmz\C:

Backup Set Information

Family Name: "Media created 17/12/2014 11:35:39"
Backup of "\\server1.location.dmz\C:"
Backup set #1 on storage media #1
Backup set description: ""
Backup Method: Full - Back up files (using modified time)

Backup started on 17/12/2014 at 11:38:50.

Backup Set Detail Information

V-79-57344-65072 - The connection to target system has been lost. Backup set canceled.
V-79-57344-41488 - Due to one or more errors in \\server1.location.dmz\C:, this backup cannot be used for conversion to virtual machines, Simplified Disaster Recovery (SD...

Backup completed on 17/12/2014 at 11:44:51.

Backup Set Summary
Backed up 8736 files in 1060 directories. Processed 1,990,392,710 bytes in 6 minutes and 1 second. Throughput rate: 315 MB/min Compression Type: None

Snapshot Technology: Started for resource: "\\server1.location.dmz\System?State". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS).
The following volumes are dependent on resource: "C:" .
The snapshot technology used by VSS for volume C: - Microsoft Software Shadow Copy provider 1.0 (Version 1.0.0.7).

Drive and media mount requested: 17/12/2014 11:45:26

No appendable media could be mounted.
Switching to overwrite operation on scratch media.

Drive and media information from media mount: 17/12/2014 11:48:55
Drive Name: Deduplication disk storage 0001:1
Media Label: OST00001245-45D3195C02B6CAD7
Media GUID: {eb130457-018d-4ff4-b70a-3a1d31a54de0}

Network control connection is established between 172.19.24.157:63176 <--> 172.16.1.22:10000 Network data connection is established between 172.19.24.157:64151 <--> 172.16.1.22:65427

Set Information - \\server1.location.dmz\System?State

Backup Set Information

Family Name: "Media created 17/12/2014 11:35:39"
Backup of "\\server1.location.dmz\System?State"
Backup set #1 on storage media #1
Backup set description: ""
Backup Method: Full - Back up files (reset archive bit)

Backup started on 17/12/2014 at 11:48:58.

Backup Set Detail Information

Backup completed on 17/12/2014 at 12:20:42.

Backup Set Summary
Backed up 19 System State components Processed 9,730,917,445 bytes in 31 minutes and 44 seconds. Throughput rate: 292 MB/min Compression Type: None

Deduplication Stats::PDDO Stats for (RRLIBACKUP): scanned: 11486167 KB, CR sent: 17229 KB, CR sent over FC: 0 KB, dedup: 99.9%, cache hits: 28809 (20.4%)

Job Completion Status

Job ended: 17 December 2014 at 12:21:43
Completed status: Failed
Final error: 0xe000fe30 - A communications failure has occurred.
Final error category: Server Errors

For additional information regarding this error refer to link V-79-57344-65072

Errors

Click an error below to locate it in the job log

Backup- \\server1.location.dmz\C:
V-79-57344-65072 - The connection to target system has been lost. Backup set canceled. V-79-57344-41488 - Due to one or more errors in \\server1.location.dmz\C:, this backup cannot be used for conversion to virtual machines, Simplified Disaster Recovery (SD...

Colin_Weaver · ‎12-17-2014

Are your jobs set to use client side deduplication - if yes you might have to turn this off because it is likely client side deduplication is trying to make a connection back from remote server to media server and the DMZ is probably blocking this direction of connection request.

Colin_Weaver · ‎12-17-2014

Actually just noticed the system state backup of the same server seems to comeplete, as such it is probably not client side dedup. Are you seeing any process crashes with auto restart of process enabled

Note if some resources backup fine and some don't where all are in DMZ then it might be time to log a formal support case as there is something odd going on.

Martin-R · ‎12-17-2014

The jobs are all set to BE server side dedup.

There is nothing apart from the BE entry in the logs saying the backup failed.

It is only the DMZ servers that fail. The other 94 run without issue.

Martin-R · ‎12-17-2014

Information in the system event log shows the Virtual Disk Service stopping and starting, not as an error, but during the backup. I was watching the data transfer and there was successful data being written to the dedup storage before and after this service stopping and starting.

I didn't notice before, but there is a corresponding entry in the Windows security log: audit failure with an invalid user name for the DMZ. It tried to authenticate with an internal only username and password (used for the internal servers). I will try to see where this could have come from.

Martin-R · ‎12-19-2014

The normal issues with connectivity do not seem to be an issue here. There is obviously something going on with the communication or trust between the BUE server and the DMZ servers. A call is being logged with Symantec. I will post any solution when they find it.

VOX

Communications failure with the DMZ