Backup Microsoft Virtual Server with GRT option takes very long time

[JorgenE]

Hi,

This is my scenario.

One Backup Server. One W2008 With Hyper-V server, 2 virtual machines. The virtual machines VHD files are stored on a HP Lefthand Volume connected through ISCSI which
the Hyper-V server has access to.

Between the Backup Server & the Hyper-V (host) server i have a dedicated backup network 1Gbps network.

Backup agent installed @ Hyper-V host server. NO backup exec agents installed in the Virtual machines.
Also installed Hyper-V @ the Media server.

I want to perform an online backup of the virtual machines.

I set up the job, select the target server (Hyper-v) select Microsoft Hyper-V option below the server.

Then in the settings option if i check "Use backup exec GRT to enable restore of invidual files ...." option the backup performance becomes totally crappy.

Without the option selected Backup perform @ 8 minutes 40 seconds (26GB) with a throughput @  3,618MB/min

if i select the GRT option enabled i get 1 hour 35 minutes (15GB) (just one of the virtual servers) throughput is 154MB/min !!!!!!!!!!!!!!!

i know that i have to have the agent installed inside each virtual guest to be able to restore the files directly into the guest system. But that's not what i'm looking for atm, just to be able to restore these files to another location (media server for example) will satify my needs.

Why this performance degration ?????

[Paul_Gazo]

I have a similar config.  Hyper-V, two VMs, the BE 12.5 agent, and plenty of hardware to make things screamingly fast.

I haven't reached the point where I NEED to back up the VMs online, but using GRT to get at the VMs is brutal slow.

[Ken_Putnam]

 I haven't reached the point where I NEED to back up the VMs online, but using GRT to get at the VMs is brutal slow

Just off the top of my head I'd say it is similar to backing up an Exchange Store or doing brick level backups  (& we ALL know how slow that could be, right?)

[Paul_Gazo]

In that case, it's not really GRT at heart.  What the agent should be doing is backing up the VHD in a mountable state so individual contents can be restored.  It should - in theory - be a streaming backup of a single huge file, just like doing an Exchange EDB.  There's not meant to be concern over the internal structure.  That's not what I'm seeing, at least performance-wise.  No idea what magic the agent is or isn't doing.  Funny thing is my VMs aren't very large and don't change much.  They should scream.  But don't.

I figured I'd post mostly to share what I'm seeing.  I don't need to directly back up the VMs on a schedule so it's not a huge concern to me.  It's just polite to let people know they're not alone.

[JorgenE]

With the GRT option i get a .vhd file in the IMG0000X folder in the B2D folder i backup to. That way it's very easy just to mount the VHD file in Gizmo or any other VHD aware application and very easy to snatch a file from the backed up virtual machine.

Without the GRT option i get a huge .bkf file, and the extract of VHD file from there are very time consuming and troubblesome.
Soo i really need to get some speed with the GRT option, i'll try running backup exec directly on the hyper-v server and see if i get some better performance.

Now i only got 2 virtual servers, but i'm looking @ atleast 7-8 virtual servers on 2 Hyper-V servers, with this speed with GRT option enabled i can forget using Symantec Backup Exec as backup software for my solution.

I'm also evaluating Microsoft DPM, but in the initial phase that software requires AD !! to even install & DPM cannot also be installed on a DC or application server. I only want 2 Hyper-V server & a backup server totally isolated from all other networks & DPM would require an addition server for my scenario. Any other backup software solutions capable of performing online backup of Hyper-V servers are appreciated !! if symantec can't resolve my issue here.

[JorgenE]

Hi,

Now i have tested Running Symantec Backup Exec 12.5 locally @ the Hyper-V server and running a backup with the GRT option enabled.
Still same crappy performance ..... 144MB/min ..

Without the GRT option enabled it performes in an acceptable time frame.
Is this by design ????  to have 1000% slower backup window with GRT option enabled ???

Are there any free app available so i can open a symantec .bkf file and export the .vhd file ???
Instead of doing a complete restore to another Hyper-V server ???

[JorgenE]

Bump ...

[gpz1963]

I also have the same performance issues with GRT enable



GPZ1963

[ScottJ]

Hopefully we can get an answer directly from Symantec on this soon.  I'm having the same issue - similar setup to Jorgen, except for direct attached storage.  Currently I have 140GB of data that, when backed up directly, runs at 3,140MB/min.  Once I turn on GRT, it drops all the way down to 105MB/min.  The job that used to only take 1 hour to complete, I had to cancel after 12hrs because I can't run the backups during business hours - and the job was only 50% complete.

I'm planning on adding four more VM's which, once configured, should add another 100GB to my backups.

I can't live without a backup of these .VHDs, but I can't live with a 2-day long backup run either.   (I have two more Hyper-V servers that I want to add in eventually, but with well over 1.1TB total in VHD files alone.....(let's see 105MB/min = 6GB/hr; 1,340GB/6 = 223.3hrs; 223.3hrs/24hrs in a day = 9.3 days to complete my backup??)

The only other option I can see would involve shutting down the VMs, copying the .VHDs, then running a files-only (no GRT) backup of the  .VHDs AND the individual files in each VM.  The issues with that are 1) I can't afford to shut down the VMs every day just to run backups and 2) I don't have 300GB of disk space lying around just to use as temporary storage.

I understand Ken's point, but with Exchange, you are running a block-by-block copy of data while the store is still mounted and in use.  With this, you're working off the snapshot image of the VHD and mounting it in a manner that the individual files are visible.  In all essence, you are running a second instance of your virtual machine off-network.  The backup of the individual files at this point, again as Paul pointed out, should "scream".  Granted I understand that you are still creating multiple backup files of the same data (Once for the snapshot, once to copy the snapshot's VHD, and then once mounted, copying the data from the VHD). 

Even adding in time to mount and unmount the VHDs, I can understand it taking 4 or 5 times as long as just the files themselves - but there is no way that I can understand why it takes 20 to 30 times as long.

[Ken_Putnam]

Since there has been no response from Symantec, at all, I set the "Tech Support" flag on this, so we should expect SOME official response fairly soon

[ScottJ]

Is there a definition for "fairly soon"?

[Shaun_Owens]

I haven't tried turning off the GRT but my backups go to a SAN disk, the SAN uses a Raid 5 array of SATA drives (7200) and I get about 600MB per minut throughput.  Just curiouse what your Hyper-V servers are hardware wise, 150mb per sec seems really slow and I have a really slow SAN DELL MD 3000i, only 512 cach.  My Hyper-V server are Dell R710's, 15K drives using local storage, 2 x quod core CPU, 48GB RAM and my BE Media server is a Dell R610, single quod core, 8GB RAM and 10K drives.  My backbone is a Dell gigabyte switch and my SAN switch is gigabyte as well.

[ScottJ]

Shaun,

I don't think hardware is the issue as the backups without GRT run at 3000MB/min.  But just for kicks, here are my specs:

Hyper-V servers are Dell R610's 48GB RAM 2x Xeon 5540 quad-cores, 6x 10K SAS drives, BE server is a Dell R300 with a single quad-Xeon 4GB RAM and local 7k RPM SATA drives.  The two are connected by a Cisco 3950 gigabit switch.  There are only 20 computers total (these two servers and 18 PC's) on the network, so I don't think there's anytihing interfering with the bandwidth anywhere else.