VM Instant Recovery fails from 5230 Appliance

[arctics2]

Trying to perform a VM instant recovery, but it fails with the following:

Error bprd (pid=16460) Failed to mount a backup image on the server ESXHOST

system error occurred  (130)

Running the following command from a Windows 2008 R2 recovery host:

nbrestorevm -vmw -ir_activate -C Restore_VM -temp_location datastore_on_ESXHOST -vmproxy recovery_host -vmpo

NFS services are installed and Client for NFS is running on the recovery host.

NFS client services are enabled on the ESX Host.

Not sure where to go from here.

Thanks!

[arctics2]

Not sure if this is related, but the nbfsd log shows:

createNasDatastore: create NAS datastore failed with 6
performVMOpn: return error on mount : 130

bpVMutil log shows:

bpVMutil main: checkifDatastoreExists failed
bpVMutil main: EXIT STATUS 71: None of the files mentioned in the file list exist or may not be accessible

Also note, I have tried running this directly from the appliance using the appliance as the recovery host, but same thing.

[mnolan]

Check for VMWare events in vsphere.

Could be that the ESX server has no name resolution for the media server so it cannot mount the nfs image.

To be sure, add the media server to the esx server host file.

[Marianne]

We have tested this at a customer site shortly after NBU 7.6 was released - also backing up to 5230 Appliance.

We have summarized the steps as follows (see if there is anything that may have been missed in your environment):

Requirements

• The virtual machine to restore must have been backed up from a VMware policy.
• The backup image must reside on NetBackup disk (basic, advanced or MSDP).
• The target ESX server for the restore must be at vSphere 5.0 or later.
• The restore host (NetBackup client) must be on Windows. (If backup host is NBU Appliance, the restore host can be a Windows VM)

• Restore host must have SERVER entry on NBU master
• The NFS Client service must be enabled on the ESX/ESXi host.
• The Services for Network File System (NFS) must be installed on the Windows restore host.
• The media server must use IPv4 (IPv6 is not supported).
• NetBackup requires login credentials for the vCenter server and the restore host.
• TCP ports 7394 and 111 between the target ESX server and the media server where image resides.

Steps to restore

• Use cmd on ‘restore host’ (Future NetBackup releases will feature a GUI for the IRV process):
  nbrestorevm -vmw -ir_activate -C <virtual-machine> -temp_location < temporary_datastore > -vmproxy < Windows restore host> –vmpo
• Monitor:
  Both the vSphere Client and the NetBackup activity monitor show the status of the VM being powered on.
• Use:
  View newly powered on VM from the vSphere client just like any other VM and is 100% user accessible.
  vSphere capabilities (e.g. Storage VMotion) can be implemented on this VM.
  The NBU disk based storage unit appears as a “read only” Datastore to the selected ESXi host.
• Restore entire VM:

Storage VMotion restore process can be initiated. This copies the VM image from the NetBackup disk storage unit to the standard ESXi Datastore.

• Restore individual files:
  Use file sharing (e.g CIFS) between the restored VM and another virtual machine to copy the files.
• Complete the IRV Process:
  Cancel the VMware Instant Recovery job in NBU activity monitor.

More Info

Details and more options can be found in :

• NetBackup 7.6 feature Briefing - Instant recovery for VMware: http://www.symantec.com/docs/TECH211113
• Restoring individual files with Instant Recovery for VMware while the current virtual machine is running : http://www.symantec.com/docs/HOWTO92095
• NetBackup for VMware Administrator's Guide
  
  
   

[arctics2]

Thanks for the responses.

Not seeing anything in ESX logs.

Marianne,

I've thoroughly gone through the VMware admin guide and reviewed all of the items you mention above.  Except that since we are on 7.6.1.1, I read that the 5230 appliance can be used as the recovery host, so tried direct.  Once it failed, I configured a recovery host on a Windows 2008 R2 machine and getting the same thing. 

Some more logs that may mean something to someone:

nbfsd log:

10:50:33.556 [124890] <2> performVMOpn: starting mount on vmserver: vcenter ESX host: ESXHOST_IP

10:50:33.556 [124890] <2> createNasDatastore: stating creating a nas datastore vmserver: vcenter esxHost: ESXHOST_IP nasserver: NBU_Appliance_Name nasPath:/NBUFS_F_ESXHOST_IP_NBU_Appliance_Name label: NBU_IR_NBU_Appliance_Name_172

10:50:33.560 [124890] <2> vnet_pbxConnect: pbxConnectEx Succeeded

10:50:33.561 [124890] <2> logconnections: BPRD CONNECT FROM NBU_Appliance_IP.38525 TO NBU_Appliance_IP.1556 fd = 5

10:50:34.885 [124890] <16> createNasDatastore: create NAS datastore failed with 6

From bpVMutil log:

10:50:31.847 [124881] <2> vnet_pbxConnect: pbxConnectEx Succeeded

10:50:31.847 [124881] <2> logconnections: BPRD CONNECT FROM NBU_Appliance_IP.38836 TO NBU_Appliance_IP.1556 fd = 8

10:50:33.207 [124881] <16> bpVMutil main: checkifDatastoreExists failed

10:50:33.207 [124881] <2> bpVMutil main: EXIT STATUS 71: None of the files mentioned in the file list exist or may not be accessible

[nbutech]

Hello,

I think you should have a look on Vmware side for the hostd logs and Vmkernel logs /var/logs/messages

You can also review below links

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2002957

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036182

[arctics2]

Thanks for the suggestions to look at the ESXi logs ... I've been able to get one step further.  The above issue was due to name resolution from the ESXi host to the appliance.  Once that was resolved, it proceeds further, but now it fails at:

12/21/2015 15:38:14 - Error bprd (pid=24308) Failed to activate a virtual machine on the server ESXHOST

12/21/2015 15:38:14 - Info bprd (pid=24308) Starting to unmount a backup image on the server ESXHOST

the restore failed to recover the requested files  (5)

Note that this is being run from an appliance on 2.6.1.1.  The command used was

nbrestorevm -vmw -ir_activate -C Client_VM -temp_location ESXHOST -R /usr/openv/netbackup/bin/renametest.txt -vmpo -vmserver ESXHOST

 

[Marianne]

Still looks like an issue on the ESX server. Have you checked the logs again?

[nbutech]

Did you check this article ? Though it is for Windows you shall get some headsup in bpvmutil and nbfsd logs

Instant Recovery of a Virtual Machine failing with a Status 5.
Article:000022056 Publish:September 14, 2015 Article URL:http://www.veritas.com/docs/000022056

[arctics2]

I've checked the logs on the ESX server but not seeing anything that jumps out at me.

I've also read the article.  Can someone suggest how to restart client for nfs services on an appliance?

Some additional info.  To get this far, I'm now running the command as follows:

nbrestorevm -vmw -ir_activate -C CLIENT_VM -temp_location ESXHOST -R /usr/openv/netbackup/bin/renametest.txt -vmpo -vmserver ESXHOST

Note I am restoring to a test standalone ESX Host now.  Not connected to Vcenter.  The content of renametest.txt is:

change esxhost to ESXHOST
change datacenter to None
change folder to None
change resourcepool to /ha-datacenter/host/ESXHOST/Resources
change datastore to datastore1
 

Is this line item required in the file:

change /original_datastore/original_VM_folder/original_vmdk.vmdk to /new_datastore/new_VM_folder/new_vmdk.vmdk

If so, how do I locate this info??