Error when backup vmware image by Backup Exec 2010

[perre]

Dear,

I have a strange problem when I backup the vmware image by backup exec 2010 R3,

My Infrastructure :-

- IBM Blade Chassis E

- 3 pcs of IBM Blade Server HS21, it total has 9 window 2003 server run on this blade

- 3 pcs of IBM Blade Server HS22, it total has 5 window 2008 R2 server run on this blade

- All Blade Server is installing vSphere ESXi 5

- another blade server HS21 install window 2003, which is running vSphere vCenter 5 + backup exec 2010 R3, the backup exec 2010 R3 is up to date already.

- All blade server is connect to IBM DS3400 SAN storage and all vmware machine is running at SAN

- I have 14 vm machine, 9 window server 2003 + 5 window server 2008 R2, all window update is up to date and the vmtools is latest too.

- the backup exec is using SAN transport mode at vmware setting.

My Question :-

In above setting environment, I can able to take a snap shot and backup all window server which is running at HS21 ESXi 5, but all server which is running on HS22 ESXi Server is fail, the error message is cannot take a snapshot, this error will see at backup exec log and vCenter task and event.

So I try to move all vm machine from the ESXi 5 of HS22 (5 pcs of window 2008R2) to HS21 by vmotion, and try to backup again, oh .... all vm machine can backup and no error message, and then I try to vmotion some window 2003 server from HS21 to HS22 and try to backup again too, oh .... all vm machine is failure, the error message is same, the snapshot error.

After try above testing, the first idea is the IBM Blade server HS22 have problem, because the same VM Machine can able backup at HS21 ESXi 5 but failure at HS22 ESXi 5. So we call IBM engineer onsite take a full hardware test, it is pass. and we call VMware support remote check all setting on our VM Infrastructure, and all setting is standard.

At this moment, I still try and try .........

At the end, I try to install a new server on the ESXi 5 of HS22, because this server 2008 R2 is new install, so no any window update or software install, I try to backup again, it is OK ......

So I try to make all window update and try to backup again, it is OK too ......

And then I try to install the latest vmtools and try to backup too, it is Failure .... So the main point of the backup failure is causing by the VMTOOLS, I think the backup exec not support the latest vmtools which is running on IBM Blade Server HS22.

And I try to using custom install of vmtools, but if the vmtools is install at any window virtual machine, even only 1 selection on custom install, the backup exec must FAILURE .......

So I need to remove all VMTools at all window virtual machine to make sure the backup is OK now .....

Anyone have above case? and have any solution? Thanks a lot.

[pkh]

To use vSphere 5, you need BE 2010 R3 SP1.  If you do not have SP1, then run LU to update to SP1.  There are also a couple of hotfixes after SP1, so you have to run LU again after you have updated to SP1.

[perre]

Yes, the backup exec R3 SP1 is installed.

[ZeRoC00L]

can you ping the esx hosts from the BE server ?

http://www.symantec.com/business/support/index?page=content&id=TECH77785

otherwise, enable debug:

http://www.symantec.com/business/support/index?page=content&id=TECH82942

and post the logs.

[perre]

I can ping the ESX host 1 - 5 and the selection of backup list is select vCenter's vm machine, the vCenter is same server with BE 2010. The error message as attachment.

I will try the log enable with the debug enable, but I need to install the vmtools again and then backup, but my server cannot reboot now (it needs after install vmtools). So I need to test again 1 - 2 days after.

** The error message will appear at BE job report and vCenter Task and Event. See Attachment Picture

[perre]

And when the job failure with above error message, the vmtools is running mouse and vga drive only, I use custom installation and disable others. same as attachment screen.

[teiva-boy]

Just as a test, can a VMware NBD backup be made, rather than SAN?  Perhaps all the SAN LUN's are not provisioned cocrrectly?

Are there any permissions from one host to another that are not the same?  

The only custom VMware Tools install you need to do for BackupExec, is to remove the VMware Tools VSS Driver.  BackupExec's is the one to use for the GRT functions...

[perre]

In the early backup, I used NDB option, but the speen is too slow, only 500MB/Min, and then I changed to SAN option for backup, the speed increase to around 6000MB/Min. Because I need to backup around 1.2 TB VM, so I use SAN transport at the end.

But the error message happen at NDB and SAN transport mode, even the GRT is selected or not. But if move the VM Machine to HS21 server, it is success backup, it is failure on HS22 server.

But if I remove the vmtools, it can success backup at HS21 and HS22 blade server too.

So is it relate the vmtools? but if yes, why the vm machine can sucess backup at HS21 blade server but failure at HS22? And why if remove all vmtools, it can success backup at both HS21 and HS22 blade server?

[ZeRoC00L]

Do you have installed exact the same ESX(i) version and patches on all hosts ?

[teiva-boy]

Then your best bet, is to call Symantec or use the mysupport portal to open a support case to troubleshoot further.  It's very possible you have a config that is unique or special with vSphere5.

I would open the case, try the level 1 support for a day, and if they are useless, escalate to a "Duty Manager."  They will bump you up to a more senior level technician to diagnose and hopefully solve.

[Lesta_G]

run a test backup with NBD as tevia-boy suggests, you do not need to let it finish, just do it to prove it is an issue with SAN transport.

Also take a snapshot (with quiesce) via vcenter client to see if a snapshot will actually work.

This helps to eliminate it as a BE problem and may just be a vmware issue

[Colin_Weaver]

In your descriptuion you state that You Have ESXi on all blades (including Blade HS21). You then state the you have a Virtual Center and Backup Exec media server also running on HS21 - if this media server is also a VM and not just a separate physical server then SAN transport is not supported (and neither is tape library/drive pass through)

[perre]

It is not related SAN or NBD transport, because same vm machine can backup at ESXi 5 of HS21 or HS22 blade server.

But it can backup at ESXi 5 of HS21 if the vmtools installed, but cannot backup at ESXi 5 of HS22 when the vmtools installed.

So I can make sure the problem is on vmtools

[VJware]

Compare the VMWare Tools install on both servers....Does the HS21 have a custom install with the VMWare VSS not installed, whereas it is installed with HS22 ?

If yes, modify the tools install for HS22 & remove the VMware Snapshot Provider and VMware Filesystem Sync Driver options, reboot the machine & retry.

[perre]

I already the try to use custom install and one by one to install and try the backup. the result is, if I uncheck all function, including all drivers, but the backup still failure when the vm running at HS22, but I can backup even I select completed vmtools install.