Windows thin vmdk files now backed up as thick disks after 2010 R3 upgrade

[thebigred]

Hi,

Sadly i've inherited this issue so you'll have to excuse me if I'm a little slow on the uptake, this is the frist time I've worked with VADP / VIII and documentation of anything to do with it is nice and wolley, very much feels like a "It should just work" product - which is annoying.

We have ESXi 4.1 U1 hosts running in a VSphere VCentre (versions all match etc). We are trying to backup VM's on two hosts in the vCentre which are not attached to our main SAN, therefore the backup server does not have direct SAN connectivity. I accept that this means we will be running in NBD and there are some performance hits there, but that's just life *shrugs*

When we first started backing up these VM's we were using Backup Exec 2010 R2 the size represented the usage within the vmdk file, for example a DC was backing up at about 15GB, however since the upgrade to Backup Exec 2010 R3 I've spotted that the vmdks are being backed up as though they were thick disks, this is causing the backups to run far too long, in some cases 45 hours when previously they ran in 45 minutes!

The VMDK's are stored on a fibre channel HP SAN, as mentioned before for various reasons the backup server does not have direct fibre access to the SAN. We are not using NFS.

Another issue is speed - originally we were seeing speeds of about 2,400 mb/s, we are now seeing speeds of about 500 mb/s, a fairly significant drop.

We've not changed anything networking wise since the upgrade, and no new jobs have been added. The speed is the same regardless of the number of backups running in tandem.

We are backing up to LTO4 drives, I know this part of the backup route is working fine since we have another different job which backs up VM's on our main SAN and this runs at roughly 2,400 mb/s. This SAN is the same setup but a netapp product, it's an FC san - we don't have direct fibre connectivity from the backup server so again these backups take place over nbd.

I'm sure there are a million questions about our setup, i'll try and answer any that come up, but really what I need to know is if this is a known issue. A good old google hasn't turned much up

Finally the VM's themselves are Windows Server 2008 R2

Over and out

Red.

[ZeRoC00L]

Are your virtual machines updated to Vmware Hardware Lever 7 ?
As far as I remember this is a requirement for 'thin' backups.

[thebigred]

Yep, all VM's are version 7

Anyone else? - I've never raised calls with Symantec for backup exec before but I know we have active support contract with them

[ZeRoC00L]

You can contact Symantec here:

http://www.symantec.com/business/support/contact_techsupp_static.jsp

Select your country and call the number.
You will need your (active) support agreement number.

[ZeRoC00L]

Did you enable CBT (Changed Block Tracking) on the VM's ?

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=102012...

[thebigred]

Yep - CBT is enabled.

I don't think it's a CBT problem to be honest, we are only doing full backups at the moment (one step at a time)

The problem is that it's definatley not backing the disks up as thin - and the speed has dropped off since the upgrade to R3.

Do any of the symantec guys answer forum posts or do I have to poke someone?

[thebigred]

Anybody? - Suprised none of the symantec guys/girls have commented yet!

[Ben_L_]

Check this registry key.

HKLM\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent

NTFS Used Sector Tracking Preferred

What is it set to?  If it's set to 0, change it to a value of 1 and cycle the Backup Exec services and run the backup again.

[thebigred]

Ben,

Thanks - I've just checked and that key is already set to 1.

Ol.

[Ben_L_]

Change that registry key back to 0, cycle the BE services and try the backup again.

What type of datastore are these VM's stored on, VMFS?

Also are the disks in the VM GPT or MBR (you'll have to check disk manager in windows to get this information).

[thebigred]

Cheers, I've just set that back and am running another backup now - will update when it's done.

They are MBR based and stored on VMFS3's

Thanks

ol

[thebigred]

OK - Think we've solved the thick / thin issue. That seems to be fine.

Next up is the speed issues - need some help/guidance on those.

We are def using NBD as there is no direct SAN connection, and since the upgrade from R2 to R3 there is definatley a massive performance drop from around 1200 mb/s to 500 mb/s

[ZeRoC00L]

So the selution was the option Ben L suggested ?

You are not the only one with slow backups after R3 upgrade. Lately I see lots of slow backups since R3 upgrades...........

[thebigred]

Actually it wasn't what Ben suggested - sorry Ben :)

It was more a case of a bit of confusion our end, I completley got confused over two VM's with similar names, one ended up having thick disks and the other thin - backing up the thin one results in the right sized backup - so my bad.

So is this a bug which symantec have admitted to then? If so I need to know and ideally get some kind of bug / KB number so I can report this back to the customer - currently the R3 upgrade is looking like a bit of a dogs dinner and they don't seem to impressed with the product.

[ZeRoC00L]

Thanks for sharing your solution !

No I have not (yet) seen a know issue / technote about the slow backups after R3 upgrades, just an increase of these messages in this forum.

Maybe it is wise to open a call @ symantec for the slow backups.

[thebigred]

Yeah I've done this - like the forums they seem a bit slow on the response, not the most impressive service i've ever had - will chase the call tomorrow i've got the number somewhere.

Our NDMP based backups on the same box (backing up from a NetApp SAN) run absolutley fine, very happy with the speed. It's just NBD based backups.

Again I'm not expecting them to set the world on fire but they were running much quicker before the R3 upgrade. Going back to R2 really isn't an option for us as the R3 upgrade fixed another bug which was preventing us from backing up other systems.

[ZeRoC00L]

Please note that these forums are based on best effort, and there is no guarantee that someone from Symantec even WILL respond.
(I think they only will respond when there is time available, but registered calls @ support should get priority above these forums.)

[thebigred]

OK Thanks for the note, I'm going to chase the call today (again)

-

[pkh]

Do note that these fora are best effort help from the user community and NOT from Symantec.