Netbackup 6.5.1 and VMWare

[Mark_Casemore]

I have created a policy to backup our virtual servers (running on VMware ESX server v3). I have followed the document "how to configure and use NetBackup 6.5.1 for VMware". At the end it runs a validation on the new policy which fails with the following error message:


Policy: VCB-TEST -- client: L-curric-prt-01 --
Snapshot could not be created with VMware method

Policy: VCB-TEST -- client: l-admin-prt-02 --
Snapshot could not be created with VMware method

Policy: VCB-TEST -- client: d-admin-prt-01 --
Snapshot could not be created with VMware method

Policy: VCB-TEST -- client: d-curric-prt-02 --
Snapshot could not be created with VMware method

Anybody seen this error before, thanks.

[Mark_Casemore]

Just to keep you all informed, had an update regarding this (yesy I am STILL working on a resolution)

Hello Mark

I hope that you had a good break away from work (and Netbackup!).

Since you have been away I have been made aware of several problems in VCB backups in Netbackup and I believe that they are applicable to your installation. The fixes are in Netbackup 6.5.2. but this is not available until the end of May at the earliest.

There are binaries available meanwhile and I am preparing an escalation today so that my backline engineers can review your errors & may provide the binaries if applicable.

I will be in touch as soon as I have more information for you.

Best Regards
Alan Thomas
Associate Technical Support Engineer

[Mark_Casemore]

As he promised, I was sent the new binaries and hey presto, everything works fine.  Watch this space for 6.5.2 when all our woes are fixed

[Anonymous]

I am really interested in the fixes in the 6.5.2 updates now as I have come to the end of my troubleshooting and about to contact Symantec support with my problem.

But thought I would share my experiences in configuration, and my fixes.

Hardware Environment:
Fibre Channel SAN
2 x Tape Library (7 drives each using SSO license option)

NetBackup Environment:
Master Server: HP-UX 11i v2 (IA64) / NetBackup 6.5.1
Media & VCB Proxy Server: Windows 2003 R2 64BIT / VCB 1.1 / NetBackup 6.5.1A
NetBackup Client software disabled or not installed on Virtual Machines.

Virtual Environment ( April 2008 updates):
VI: 2 x ESX 3.5.0 Build 82663 servers, VC 2.5
VM's with latest VMware Tools


64BIT Install Problem
=======================
First off the VCB Framework won't install into 'Program Files' subfolder always defaults to 'Program File (x86)'; even when I tried to specify this as a custom path
Have to move the folder into 'Program Files\VMware\'

(the bpVMutil debug logs came in handy here)

Could not get VM Server list
=============================
This was down to two things.
1. GUI Timeout of the Admin Console - increased to 500
2. Host name resolution on the VCB Proxy server. I forced it with some edits to local hosts to confirm this.
After this able to get list of VM's from my VirtualCenter

Some VM's missing from list
===========================
A couple of VM's were missing from the VM Server list when it was displayed, and this was down to inactive or unconfigured secondary NIC's in the VM
If a VM when queried has a disconnected NIC with 0.0.0.0 this seems to place it at the beginning of the NIC list and this VM is excluded - so watch out.
We had added a Backup LAN into our Virutal Infrastructure environment but hadn't implemented or used it yet.
Suffice to say we've now scrapped it in the hope of using VCB.

GOTCHA
=======
Don't forget to present all your datastore disks to the VCB proxy server in your SAN environment
- I quickly caught this. Yes, I know it is mentioned in the config guide.

OTHER INFO
==========
Have not touched the config.js file under VCB Framework folder
Using port 443


CURRENT PROBLEM
================
So after configuring a policy and pointing it at my large 'mount' disk on the VCB proxy server, I have been unable to perform a complete backup to tape of my Virtual Machines.

I get a overall status 130 error message within NetBackup, with child status 59 messages.

Now looking above someone mentions licensing each ESX server you have as a ENTERPRISE CLIENT

I have this type of license installed locally on my VCB Proxy / NetBackup Media Server
Should I install a second and third one on my master server to cover my 2 ESX servers?
- I have not seen this stated in any documentation.

Only seen the following technote
http://support.veritas.com/docs/298951

The type of license I have is this
VRTS NETBACKUP ENTERPRISE CLIENT 6.5 WIN TIER 2 STD LIC

This technote says it needs
VRTS NETBACKUP ENTERPRISE CLIENT 6.5 WIN TIER 3 STD LIC

Is their a difference.?

Here's my netbackup status log

26/05/2008 14:32:23 - started process bpbrm (3500)
26/05/2008 14:37:25 - Error bpbrm(pid=3500) cannot connect to homer.xxxx.com, The operation completed successfully.  (0)   
26/05/2008 14:37:26 - Error bpbrm(pid=3500) terminate wait failed, The handle is invalid.      
26/05/2008 14:37:27 - end writing
Status 130
26/05/2008 14:37:32 - end Flash Backup Windows , Create Snapshot; elapsed time: 00:05:11
26/05/2008 14:37:32 - begin Flash Backup Windows , Stop On Error
Status 0
26/05/2008 14:37:32 - end Flash Backup Windows , Stop On Error; elapsed time: 00:00:00
26/05/2008 14:37:32 - begin Flash Backup Windows , Delete Snapshot On Exit
Status 0
26/05/2008 14:37:32 - end Flash Backup Windows , Delete Snapshot On Exit; elapsed time: 00:00:00
26/05/2008 14:37:32 - begin Flash Backup Windows , End Notify Script
26/05/2008 14:37:33 - started process RUNCMD (26210)
Status 0
26/05/2008 14:37:33 - end Flash Backup Windows , End Notify Script; elapsed time: 00:00:01
Status 130
26/05/2008 14:37:33 - end Parent Job; elapsed time: 00:05:12
system error occurred(130)

My bpbrm log on the VCB Proxy server is full of

13:37:24.633 [3500.2268] <2> vnet_async_connect: vnet_vnetd.c.3983: connect in progress: 1 0x00000001
13:37:25.696 [3500.2268] <2> vnet_async_connect: vnet_vnetd.c.4181: getsockopt SO_ERROR returned: 10061 0x0000274d
13:37:25.696 [3500.2268] <2> vnet_async_connect: vnet_vnetd.c.3765: ran out of time before connect: 302 0x0000012e
13:37:25.696 [3500.2268] <2> vnet_async_connect: vnet_vnetd.c.3765: ran out of time before connect: 302 0x0000012e

What I don't quite get is that netbackup it is trying to connect to is homer.xxxx.com  - the virtual machine?

However, using the vcbmounter.exe on the VCB proxy server I can manually initiate snapshots of virtual machines into my mounted area.

So its definitely a NetBackup issue of sorts.

Message Edited by Stuart Green on 05-26-2008 07:43 AM

[NBU_Support]

Hi Stuart,

You can have a Check on the Vmconverter path onto the registry.
  Netbakup  will search for the default path under Program files.  If the path varies you can edit registry HKLM>Software>Veritas>Netbackup>Current Config> to point to the Correct path. This is onto the NBU Proxy / Media Server.

[NBU_Support]

Hey.


Apologies ...Kindly check for the vcbmounter path instead of Vmconverter.

[ahlip]

The info I have is that 64-bit Proxy server will only be supported in 6.5.2. Meaning that at this stage the 6.5.1A x64 Windows binaries do not have the engineering work done to properly support VCB integration. Note that before VCB1.1, VMware only provided a Proxy driver for a 32-bit Win server

[Anonymous]

Well I can confirm that 64 bit proxy works and it WAS a license issue. I applied a temporary 30-day 'Enterprise Server' license onto my Media /VCB Proxy server, and straight away the backup of my VM's worked.

So to confirm the working environment is

Windows 2003 R2 Standard Edition 64BIT
VCB 1.1 Framework
NetBackup Media Server software 6.5.1A
SAN attached tape drives

...and the correct license; which is not that clear in documentation, other than the recent techdoc.

[Rajesh_s1]

Hi ,

i too had the same issue of system error occurred(130) , but i tried to backup to the Disk staging server which i was using for LAN clients it went fine .....


Then we used the remote client license .... After that backup started working fine....


Regards,
Rajesh

[Omar_Villa]

Have any check if the clients who fail have Raw partition mounted? I understand that VCB will not work with clients who have an RDM mounted, this if you are using the SCSI Hot-Add Mode. Other restriction to mention is with the LAN Mode is that your virtual disks cannot be bigger than 1TB each.

There are a lot of more details under the Virtual Machine Backup Guide from VMware I will highly recommend to read it and see if your configuration is messing up with your backups.

hope this helps
regards.

[Omar_Villa]

Under certain circumstances, you cannot use Consolidated Backup for backing up data
in a virtual machine.

  Consolidated Backup in SAN mode cannot back up virtual machines with disk
images stored on a storage device that the VCB proxy cannot access.

  Consolidated Backup cannot back up RDMs in physical compatibility mode.
CAUTION Because the VCB proxy can recognize only volumes that have drive
letters assigned to them, make sure that each volume on a virtual disk has a drive
letter.

  Consolidated Backup cannot perform file‐level backups of virtual machines
running operating systems other than Microsoft Windows.

 You cannot use Consolidated Backup to backup virtual machines in a cluster.
These virtual machines share an SCSI controller and snapshot operations cannot
be performed on them.

If you are unable to use Consolidated Backup, deploy a backup agent in the virtual
machine and perform the backup from within the virtual machine. The backup agent is
supplied by your third‐party backup software vendor.
For details on installation of the backup agent, refer to the documentation provided
with the integration module matching your backup software.