Poor performance backing up to iSCSI target

[DrDebate]

I am seeing very poor performance backing up my Exchange server to iSCSI-based Backup-to-Disk.  Here are my details:

Media Server Type:  Virtual (ESX 4.1)

Operating System:  Windows Server 2008 R2 Standard - fully updated

iSCSI Appliance:  Dell MD-3000i

NIC Configuration:  (2) VMXNet 3 - one on the LAN and one on the iSCSI (isolated) subnet

Backup Exec:  2010 R3 fully updated (Version 13.0 Rev. 5204 64bit)

Note:  The Dell MPIO driver is installed and iSCSI is configured per Dell's recomendations for Server 2008 R2

Exchange to iSCSI B2D Speed:  550 MB/min

Exchange to C: B2D Speed:  1350 MB/min

Media Server to iSCSI B2D Speed:  1150 MB/min

Windows Explorer C: to iSCSI Speed:  3000 MB/min

In other words, if I back-up the Exchange server to the Backup-to-Disk folder on the iSCSI appliance I get less than half the throughput I get backing up to the C: Drive on the Media Server or backing-up the Media Server to iSCSI.  If I do a straight file copy using Windows Explorer I get about 3000 MB/min.  I'm not particularly impressed with 1 GB/min but I can work with that.  500 MB/min is painful for the size of our Exchange environment.  I've searched the forum and the internet and I have found other people with similar problems but no solutions.  Any ideas out there?

[pkh · Accepted solution]

Symantec does not recommend the use of a virtual machine as a media server because you will have problems accessing real devices.  See Colin Weaver's comment in this discussion regarding this as an alternative configuration

https://www-secure.symantec.com/connect/forums/backup-exec-2010-r3-installed-vmware-vsphere-4-esxi-4...

[DrDebate]

I appreciate the response but I'm not sure it applies at the detail level.  The sole purpose of this Backup Exec server is to back-up Microsoft Exchange to iSCSI.  There is no tape drive in the equation which, I believe, makes the Alternate Configuration a moot point.  I understand that running the media server on a virtual platform is not supported by Symantec - now - but I'm willing to bet Symantec is willing to put it on it's radar if it wishes to stay competitive in an era where everything is moving to a virtual platform.

To reiterate some points in my initial post:

1.  This performance issue is based on backing up from the Exchange server to the iSCSI target.  If I back-up to the local disk or I back-up the media server to iSCSI the performance is at least double.  I'm certainly willing to entertain the possibility that this is based in the server being virtual but I'm confident it would not be for the reasons outlined in the article you linked.

2.  Other people have had this issue and it does not sound like they are running a virtual media server.  I believe this bolsters my opinion that this issue is not based in the virtual platform.

https://www-secure.symantec.com/connect/forums/backup-exec-2010-r3-job-rate-running-extremely-slow

https://www-secure.symantec.com/connect/forums/backup-disk-very-slow-0

[pkh]

Regardless of what you believe, if you are no able to reproduce your problem with a physical media server, you will not be able to get Symantec to look at your problem.

[DrDebate]

I see.  Between Endpoint Protection, Altiris and Backup Exec I've logged many hours working problems with Symantec support.  You have cited an article which is specific to hooking up a tape library to a virtual media server and are using it as justification to ask me to jump through a hoop which, on face, has no logical connection to the problem I (or the other users) are experiencing.  I think most people would agree that's a text book example of denying support by forcing the user to jump through unnecessary hoops in the hopes the user will give up.  However, I have never worked with a member of Symantec support who wasn't more than willing to work the problem with me.  Do you work for Symantec?

[pkh]

No. I do not work for Symantec.  None of the Trusted Advisors are.

I am just warning you of the consequence of using a virtual machine as a media server.  It is not just using a tape drive with a virtual machine that is an alternative configuration.  Using a virtual machine as a media server is an alternative configuration.  This have been stated many a times in this forum.  If you can get Symantec to support you, so much the better, but it is within their rights to ask you to reproduce your problem using a physical machine before they support you.

[Backup_Exec1]

Hi

Going through issue you have given some good details test you have done

Exchange to iSCSI B2D Speed:  550 MB/min

Exchange to C: B2D Speed:  1350 MB/min

Media Server to iSCSI B2D Speed:  1150 MB/min

Windows Explorer C: to iSCSI Speed:  3000 MB/min

But then I don't see you have tried doing Non GRT backup, can you try doing Non GRT backup of exchange and see if the speed is same or you get speed as expected

Moreover I got one document for the slow backup issue of exchange on DIsk with GRT on , but this document is related to known issue in BE 2010 & in your case it is BE 2012 , but link don't specify even this has been resolved in BE 2012 so can be an issue in this version too

http://www.symantec.com/docs/TECH180073

So you can try doing non GRT backup first & see what are the results and also can open a case with Symantec so this is investigated  further

Thanks 

[CraigV]

...the other option is to turn off Dell's MPIO option and use Windows Server 2008's iSCSI option instead to see if this gives you a speed increase. If it works, don't use the Dell MPIO option.

I backup to Iomega NAS's using another 3rd party VM backup application with iSCSI LUNs mounted and no other software installed other than Windows's iSCSI option. It works like a charm.

You should also turn on jumbo frames on your target, and the source to see if this sorts things out...your switch should also support it otherwise it won't work.

THanks!

[CraigV]

...that TN has nothing to do with slow speeds, but everything to do with VSS snapshots taking a while and adding to the time taken to backup Exchange...something the OP never described. He has slow throughput. The TN has no relevance, so please read someone's post carefully before throwing a TN out (non-GRT or not, it has no relevance and shouldn't have been mentioned!).

[DrDebate]

Thanks CraigV.  I ripped out the Dell MPIO driver and went to a single IP target using the native iSCSI initiator.  I got a little bit better performance, about 700 MB/sec, but I still think it's far from where it should be.  Thank you for the tip and thank you for actually reading my post.

[robnicholson]

Regardless of what you believe, if you are no able to reproduce your problem with a physical media server, you will not be able to get Symantec to look at your problem

Which is another small but important reason why Symantec appear to be getting left behind in the backup market in terms of innovation. I could understand a small company with a niche product not been able to support every single platform but that doesn't really apply to Symantec does it?

Many solutions now (like this one) don't have tape in the equation and it's the world of backup to disk or cloud (which is in a way just a disk over a slower link).

When the core of Backup Exec was designed way back, the world it lived in was locally attached storage where throughput was important but not critical. To a certain extent, it's still better to have you backup system as isolated as possible from the rest of your infrastructure but that's an aside.

If one was designing a backup system today, one of the top requirements would be to reduce data packets across the LAN/WAN including the catalog system. My own guess is that the dedupe system fairs pretty badly when using iSCSI as maybe even PostgreSQL isn't optimised for such use (not sure if original poster is planning to use dedupe).

I know this is a bit of a Friday winge but we have used Backup Exec for many years and to be in the position of considering alternatives is a bit like seeing an old friend go the seed :(

Cheers, Rob.

PS. So therefore it is unacceptable for Symantec to not support customers when virtualised and using iSCSI SAN. What century are we living in?

[robnicholson]

To be honest, I don't think you'll fix this I'm afraid with Backup Exec as your chosen solution for backup. Performance is not one of it's strong points. Like you, we planned to rollout a replacement BE 2010 solution a while back using our new shiny iSCSI SAN (StarWind) with RAID-10 SATA-3 drive system (SAN had multiple enclosures, this was just the one we planned to use for BE).

We soon realised that throughput with a dedupe system on iSCSI just wasn't going to hack it.

So we switched to a lots-of-RAM but not that powerful PowerEdge bare metal server with a LsiLogic SAS/SATA card connected to a 16 disk external SAS/SATA disk enclosure. We ended up using 2TB SATA-2 disks because we had some of them lying around.

With this set-up performance is okay.

Reliability isn't so okay but that's another day.

Cheers, Rob.

[CraigV]

...I don't see the issue with backing up to an iSCSI-presented HDD, much like I don't see the issue with backing up to a NAS when BE is a media server. So this all seems invalid.

I've seen the TNs around BE being installed on a VM with a library/tape drive presented to it...this isn't a Symantec problem (trust me, been down this road already with virtualised media servers I didn't recommend!). It comes from a VMware side that Symantec seems to have picked up on. I don't see CA recommending that ARCserve R15/r16 can be installed on a VM, and EMC certainly don't recommend it with NetWorker for instance.

Disk is disk...a virtualised media server wouldn't care in this case as it appears to be a local disk anyway. This could very well be a misconfiguration somewhere in the Vmware environment.

[CraigV]

OK, good start...are your server NIC/s and the Dell storage appliance set to the fastest speed available (ie. 1000MB FULL)?

Then make sure that any AV isn't perhaps scanning the BE services, and if so, put in an exclusion.

[pkh]

As an aside, up till this day, I am still amazed that the various manufacturers managed to get iSCSI to work.  If you have taken an in-depth look at the Ethernet protocol, you will realise how fagile it is.  Remember Ethernet is designed in the 70's for a Cold War situation.  Any Ethernet packet is delivered on a best-effort basis.  A disk protocol on the other hand is designed to be as bullet-proof as possible.  When the protocol says the write is completed, almost 100% of the time the data is on the disk.  The timing requirements for such protocols is quite exact, especially SCSI.  It still boggles my mind how they managed to marry the two seemingly incompatible protocols.

[robnicholson]

It still boggles my mind how they managed to marry the two seemingly incompatible protocols.

Yes, I too am amazed it works at all but it really does work. We switched to StarWind iSCSI SAN for almost everything except backup (and the SAN itself obviously) for a 100 person company and most of the the time it works a treat. We do occasionally see slow downs but that's kind of to be expected. It would be amazing to think you could take four separate servers each with their own local high speed iSCSI RAID-10 disk systems and push the same IO through one SAN server and over "just" 1Gbit/s network. Actually we have multiple networks just for the iSCSI disks.

My gut feeling is that BE is sometimes pretty heavy handed with local I/O like the catalog (remember how long it takes to search!) and that when you throw in ethernet & TCP/IP overhead into the equation, it struggles.

Cheers, Rob.