Highlighted

Failed to Connect - GRT Error Message

Hello Folks,

Using Backup Exec 2012 with SP4.  Running into two issues and would very much appreciate any help.

Issue 1:

After every backup the following event is logged for one of my servers: 

V-79-57344-38726 - Backup Exec failed to connect to virtual machine 'SERVERNAME' and was unable to collect the necessary metadata to restore individual application items. You cannot perform GRT-enabled restores of application data from this backup. You must install the Agent for Windows and the virtual machine tools provided by your host software on the virtual machine.
 

These are some of the steps to troubleshoot:

1. Remote Agent for Windows Servers is not installed or is installed however the Backup exec remote agent service is not started on the Virtual Machine
2. GRT is disabled for an application.
3. Backup Exec cannot connect to the virtual machine.
4. Incorrect credentials were entered for the virtual machine.
 

NOTE:  The two most common issue for that generate this exception are no Backup Exec remote agent installed inside the Virtual machine or Incorrect credentials are supplied to connect to the Virtual machine.  If the Virtual Machine is in a workgroup or different domain than the Vmware Host or Hyper-V host and the wrong credentials are supplied for the Virtual machine the job will complete with the exception.

I have installed the new remote client on this server after my SP4 install and the credentials test successfully, but the issues remains.  Just wanted some clear concise steps of where to go from here.

Thank you in advance!

Jonathan

 

 

20 Replies

are there any application

are there any application errors logged in event viewer on the virtual machine when the backup fails ?

Hello Jonathan are you

Hello Jonathan are you backing up this VM as VMDK? a physical box?

Yes there are. On the VM

Yes there are.

On the VM there are a few errors; some are related to beremote.exe and the one other is unlikely.  The beremote.exe errors occur about 4 hours before the backup begins.

Errors:

The Windows logon process has unexpectedly terminated. (Occurs a various times before and during backup.

---

Faulting application name: beremote.exe, version: 14.0.1798.1364, time stamp: 0x5318f09f
Faulting module name: bedsnt5.dll_unloaded, version: 0.0.0.0, time stamp: 0x5318f0ad
Exception code: 0xc0000005
Fault offset: 0x0000000063f7d5f0
Faulting process id: 0xfe4
Faulting application start time: 0x01d0114cc33a6e3c
Faulting application path: C:\Program Files\Symantec\Backup Exec\RAWS\beremote.exe
Faulting module path: bedsnt5.dll
Report Id: 15c5eff0-7d40-11e4-b17f-00155d003003

---

The Backup Exec Remote Agent for Windows Servers Service did not start.  The application failed to listen on the NDMP TCP/IP port.  Check the network configuration. 

 For more information, click the following link: 
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml

---

Process beremote.exe requested memory dump but the maximum number of memory dumps has already been reached.

---

On the VM Host Server the following error is seen:

The shadow copies of volume \\?\Volume{2dac35df-fc13-11e0-ac98-782bcb577d71} were aborted during detection because a critical control file could not be opened.

The machine that is having

The machine that is having the issues is a Windows Server 2008 R2 w/ SP1 VM running on a Hyper-V Host that is  Windows 2008 R2 w/ SP1 with all critical windows updates as of 12/1/14.

Try the steps from this

Try the steps from this article :
http://www.symantec.com/business/support/index?page=content&id=TECH155335

I would also recommend you to reinstall the backup exec remote agent on this virtual machine. Please ensure the credentials are ok as well.

Gurvinder, Thanks for the

Gurvinder,

Thanks for the suggestions.  After uninstalling and reinstalling the Remote Agent I still could not start the service.  Instead of any app crashes only one error appeared:

The Backup Exec Remote Agent for Windows Servers Service did not start.  The application failed to listen on the NDMP TCP/IP port.  Check the network configuration.

Found this tech article: http://www.symantec.com/business/support/index?page=content&id=TECH60245

 

  1. Used command: NETSTAT -abno
  2. Found the java.exe was already using port 10000
  3. Opened c:\windows\system32\drivers\etc\services
  4. Added line: "NDMP   11000/tcp" with an carriage return after.
  5. I was able to start the Backup Exec Remote Agent Service successfully.

This may have been the issue.  I will report back to you after a backup has been completed to see the results.

JV


 

 

After fixing the network

After fixing the network service port issue, reinstalling the Backup Exec Remote Agent and rerunning the backup the GRT error originally listed in this post for the same VM still appears.  No errors are found in the event logs of both the Backup Exec and VM.  The Backup Exec Remote Agent service is continually runningo on the VM.  No firewall turned on the VM.  I have completed the steps outlined in the document outside of the debug.  Let me know if there is anything more to try.

Use this KB to enable

Use this KB to enable debugging and pls attach the logs - http://www.symantec.com/business/support/index?page=content&id=TECH204389

 

The debug logs seem to be

The debug logs seem to be incomplete. A quick Q -  Which application is running on this VM ?

 

 

Just SQL 2008 R2 and other

Just SQL 2008 R2 and other client specific software.  No exchange and it's not a DC.  What do you feel is missing from the job logs?  The main backup to tape was completed but still had another piece that was a duplicate to tape.  I stopped the capture since the error appears in the tape backup portion.  Do you want me to try and run a smaller job with just that VM server involved to recreate the exception?

Yes, if you would pls. I

Yes, if you would pls. I would recommend to remove the existing logs before running the debug again for just that VM.

Attached are the debug logs.

Attached are the logs.  They

Attached are the logs.  They are quite large once unzipped.  Thank you again!

These are the RAWS connection

These are the RAWS connection errors in debug when trying to collect GRT metadata:-

NDMPAgentConnector::GetNotifyConnectedStatus: : Wait for notify_connected timed out after 30 seconds


@@@@@@@MyCloseSocket called with sockfd = 1832(0x728)    retval = 0


NDMPHostBackupEngine::ConnectToVirtualMachine: failed to connect to RAWS in virtual machine, error: 0xe0000f04


NDMPHostBackupEngine::ConnectToVirtualMachine: returning 0xe0009746

 

 

Thank you for finding that

Thank you for finding that info.  Is there a proposed fix or a common root cause?

Common causes are if the

Common causes are if the remote agent service isn't running, port 10000 is not available or in use by another process, AV blocking, name/ip resolution etc.

Are you able to get a RAWS backup of this VM directly without any errors ?

Hey VJware, a few posts above

Hey VJware, a few posts above I completed the steps outlined in http://www.symantec.com/business/support/index?page=content&id=TECH60245 changing the port to 11000 on APP server.  I found the BE Remote Agent service was not starting on APP due to a java application taking port 10000.  Once the change was completed the service started.

Do I need to make that adjustment on all the servers involved in backup (BUExec and Remote Agent Servers) or is there something else that needs to be done to ensure the BE Remote service can connect to the BE server?

Nopes, the adjustment doesn't

Nopes, the adjustment doesn't need to be done on any other servers. On the affected server, open the remote agent utility, click on the Publishing tab and ensure the right information is being published.

If the VM is running Win 2k8 and above, would recommend to reboot the machine for the changes to be permanent. And then, try a RAWS based backup to confirm if its successful or not. If successful, then try a backup using the virtual agent.

VJware, there were no BE

VJware, there were no BE Servers listed in the top window of the Remote Agent Utility on APP.  The checkmark to publish was also not selected.  I looked at the other server that has been working correctly and found several entries (NETBIOS/IP).  On the APP server I added the backup server to the list in IP and NETBIOS form.  I will wait and see if the backups are successful tomorrow and report back.