Backup Exec 2010 / VMware AVVI - Very slow backup ?

[mRyOuNg]

Hi there, I recently update for a customer a working backup exec 12.5 (using vmware agent, via a vcb proxy) to a backup exec 2010 and avvi. My jobs are the following: . A full backup to disk of the Windows VM with GRT enabled . A duplicate to tape of the previous job With backup exec 12.5, my vcb backup was acting really fast (something like 4h for a particular 500 GB VM). Now, with be2010, I've got something like 30h for the same 500 GB VM, while backup of small VM (15 GB) are fast. I suspect that vmdk file size is related to the problem. Before, with be12.5, i was not doing monolothic export, so i used to get 2GB chuncks of a VM. Now, with be2010, I'm not able to change this parameter, and the backup is working only on a monolithic file, which is something like 400 GB (even if i enable software compression). I exclude the backup directory from the antivirus (sep) analysis, but it doesn't help much ... It's a bit difficult to explain to the customer (who have bought be2010 after going to a symantec convention on paris, where a guy from symantec told him that be2010 brings a huge improvements on vSphere) that be2010 is acting slowly compared to be12.5, just because it's now using vstorage api. Anyone here having an idea on how to improve backup performance ? (maybe a way to split vmdk export with 2GB chuncks file will help ... if available). Thanks in advance for your answer. Cya PS: Btw, be2010 is plainly up to date. Same for the remote agents.

[teiva-boy]

Is vSphere up to date?  There was an U1 that fixed slow backups via NBD, and I'm sure more since then....

[mRyOuNg]

Yes, vSphere is up to date too ... vCenter 4 with U1, and ESX4 with U1 (and latest fixes via Update Manager).

[bobs_renasa]

I to have been having the same issue. When backing up VMs to disk storage. In 12.5 the 2GB split made the backups run 100% faster. If I recall correctly the issue (in my case for version 12.5) was that NTFS does not like large files in the hundreds of GBs.

I have used both SAN and NBD transport methods and in both cases the results are pretty much the same.

Any solution to this issue in BE2010 would be very much appreciated. 

[mRyOuNg]

It's clearly impossible ...

When writing my first post here (so yesterday), i was starting a backup of 3 Windows VM with GRT (so exporting vmdk to IMG folder)
. a 2003 VM with only 15 GB used
. a 2003 VM with 130 GB used
. a 2008 VM with 500 GB used

Right now, one day later, i've got the 2 first VMs stored on my backup drive, and the last one still in progress ... The whole job is giving a total of 612 GB already done, which means i still have something like 40 GB to go...

And of course, the backup snapshot made for the 500 GB VM is growing, consuming my datastore space...
And i should now explain to my customer that even if the job ends today, the data in it are already 1 day old ...

Anyone has tried a direct backup to tape drive or library ? Does this help on the backup performance ?

Thanks in advance for your answer.
Cya

[teiva-boy]

If SAN and NBD are the same speeds, than something is a miss.  SAN is almost always faster, 50%-300% faster than NBD in all of my setups, be it iSCSI or 4GB FC.

If going from 2GB files to singular large files, nothing changes in speed regardless of transport type, than I'm not sure.

[bobs_renasa]

Hi

NBD is much slower the SAN to begin with but by the time 300 or 400GB of data has been written SAN has slowed down considerablily. Which is why I don't believe that the problem is with the SAN/NBD transport methods but with the storage on the "backup proxy" server where BE2010 is writting to. With BE12.5 with 2GB splits enabled SAN backups were much much faster then NBD the whole job through.

The server on which the backups are being stored is the exact same machine as it was with BE 12.5 and the disk subsystem is a 8 disk hardware RAID 5 array. This was also the same with BE 12.5 but the 2GB splits made the diffrence.

[bobs_renasa]

@ mRyOuNg

I will test writing the data directly to a tape drive when I get into the office tomorrow and let you know what the diffrence in performance is...

[mRyOuNg]

Thanks ...

I'll try to do the same with my customer jobs ...

Finally, it ended correctly, after 33h ... The job is correct, but i really should find a way to accelerate the job.

Cya

[mRyOuNg]

So, i change all my jobs, and started it ... with Direct2Tape.

To compare them,
. Direct2Disk job is working during 33h (without any export to tape, just a backup on a disk device) / 410MB/min
. Direct2Tape job is working during only 4h (same selection list). / 5700 MB/min

That's unbelievable ;)

Anyway, this tells us that:
. vStorage API export are working perfectly.
. MediaServer is clearly showing his performance.

The problem is exporting data to disk ...
Before, with 12.5 and 2GB chuncks, export bandwidth was something like 6500 MB/min.
Now with 2010, and full vmdk export, bandwidth is 410MB/min.

I think Symantec should really add an option to split vmdk into 2GB chuncks
. It corrects the export to disk problem.
. It corrects the export to SMB NAS problem

In fact, I'm pretty surprised that that this options was removed.

Cya.

[teiva-boy]

You sure Symantec removed it, or it's a limitation to vStorage?  There were a lot of limitations to VCB that Symantec had to work around, and there are still some to vStorage as well like not being able to backup fault tolerant VM's, that is not Symantec's fault.  

[bobs_renasa]

Hi,

I appologize for not coming back to you guys sooner. The end of last week...

Ok so I backed up a VM of 880GB the process ran at 472MB/min average and took 29 hours and 42 minutes. I then ran the same backup to an LTO 4 SAS tapedrive and it was substantially faster writting the data to tape at 4109MB/min.

I agree with mRyOuNg 100%, Symantec needs to add the option back as currently we can't use the option of full VM backups on large VMs for which we paid a lot of money to Symantec!

Removing these options reduces the flexabilaty of the product which is most unfortunate.

[Nicolas_d2b]

Hi,

I have the same bad issue, VSphere 4, BE 2010.
I have File server (w2003 sp2 Standard) with 1.3 To on one vmdk.
In first, the custumer use BESR to do his backup, but since a couple of month, it should stop to use it, because they have a lot of problem and Symantec Support level 2 could not correct the bug.
So for try we use VMWare agent....The transfert rate is 450Mo\min.
So 40 hrs to backup all data, with all active mode on the ESX/VMware backup methode.
Do you have something to help us.
All of the VMWare and BE are up to date.....

Regards

[ynot21774]

I had the same issue.  I’m running a BackupExec 12.5 server running Server 2003 sp2.  It’s attached to an EMC Clariion AX4-5i.  I was getting 50MB per minute throughput using the SAN option for the VMWare Agent.  
 
I was talking to a VMWare rep and he did the following actions:
 
RegEdit and drill down to:
 
HKEY_Local_MACHINE\system\CurrentControlSet\Tcpip\Parameters\Interfaces.  Find the interface associated with the iSCSI interface.
Create the DWORD entry named TcpAckFrequency under the iSCSI interface.  Set the value to 1.
 
Reboot and re-test.  This brought my throughput rate to 1GB per minute.  

[mRyOuNg]

Sorry, for the moment, still no luck, and no update from Symantec ...
Moreover, the problem is not only on vmdk file, but I saw the same issue recently with a huge Exchange Database file.
In fact, the real problem seems to come from the lack of flexibility of the recents GRT changes.

For now, all planned migration to BE2010 are cleary in standby because the only solution from Symantec is:
. Save directly to tape (Which is interesting, except when you have to restore a 3KB file, waiting 3h for a sequencing procedure).
. Disable GRT for AVVI (which is useless, imho, because it then requires a job for VM Level backup and another for File Level backup (via agent) - it's like going back in time :))

Cya ...

PS: Symantec, wake up, we still waiting for an update or something !

[mRyOuNg]

Sorry, but we don't experience here problem with be12.5 neither performance issue ...
BE is 2010, and local disk or FC SAN i'm using are working perfectly...

But anyway, thanks for your help

Cya.