Slow tape backup BE2008 R3 to Quantum SuperLoader3

[robnicholson]

Something has degraded on the new Windows 2008 R3 media server that I've built recently. We've just had the Quantum SuperLoader 3A replaced by Quantum as it was generating errors. But I wonder if that is just coincidence as the new unit is hobbling along at ~100MB/min whereas this same tape unit has managed over 800MB/min before.

Considering this is a new unit, I wanted to remove the hardware from the equation so I've taken an image of the Windows 2008 media server and just installed plain-vanilla Windows 2003 R2 x64 with Quantum drivers. The reason I installed Windows 2003 so I could try a raw backup with NT Backup (which has been removed from Windows 2008).

On this plain system, the tape drive backed up 15GB (from local hard drive) in 8 minutes 4 seconds which is 1,917MB/min which a) shows the drive is working fine and b) gives a good indication what speed the LTO-3 drive is capabable of.

I'm just back to put the original media server image back on there (so we can do monthly backup) but just wondered if anyone has any ideas in the meatime why we're only getting 20th of the speed of NT Backup?

Cheers, Rob.

[Ken_Putnam]

Are you using Symantec or Qantum drivers?

From the Devices Tab, check the properties of the drive

Make sure that both "Single Block" options are NOT selected

What are the settings for Block Size, Buffer Size and Number of Buffers?

[robnicholson]

Hi Ken,

If you had asked me that question earlier today, I would have said "Symantec" because I always select the "Use Symantec drivers" when installing Backup Exec but I did check that earlier and both the Quantum SuperLoader 3A library and the LTO-3 drive were using Quantum drivers. I'm not sure when they ended up getting installed. Maybe during Windows update?

So I went through the process in BE of installing the drivers. After a reboot, device manager showed Symantec drivers for the tape drive, but still using Quantum for the Quantum SuperLoader. Not sure if this is how it's supposed to be or should both library & drive be using Symantec drivers? I've lost the opportunity to try some more tests today so will pick up again tomorrow.

"Write single block mode" and "Write SCSI pass-through" were both enabled when I started looking into this problem (see slow2.png). I turned off both these options as recommended in a forum post I found on here. It didn't seem to make much difference (do you need to reboot?). Slow3.png shows how they are now.

I've just had a gander in the logs and there is something that worries me. During start-up, lots of devices and services are showing errors. The log goes back to when this server was built (30 days ago) and they have been there from the start. This could be connected with two other symptoms we're seeing on this pretty-basic Windows 2008 server. Firstly, it takes a long time to get to the point where the user can logon. Normally a W2k8 box comes up in a minute or two. With this server, I have to wait about 10 minutes before it responds to the remote desktop logon. Secondly, I get the "Discovering devices" shown in the devices tab for about 15 minutes. I think the later isn't especially unusual though - something to do with ping of deduplication targets.

So later...

Cheers, Rob.

[Ken_Putnam]

The Quantum drivers for the Loader is correct.  Either Symantec or Quantum can work for the drive itself.  I was going to suggest swapping, but you have already done that

Sounds like rebuilding the media server may be your next step

To save all your current setting see

http://www.symantec.com/business/support/index?page=content&id=TECH62744

[robnicholson]

Hi Ken - thanks for the link. Would that preserve de-duplication storage folder and database as well? I'd read that it was impossible to DR a deduplication storage folder (aside from restoring from other media server).

I've got an early image of this server before BE has gone on there. Tomorrow, I'll restore to that to see if the device errors are still present in the log.

I have a feeling we'll end up trashing this install... not good!

Cheers, Rob.

[robnicholson]

Definately something amis with this installation. That same 15GB that took NT Backup just over 8 minutes to complete took BE 2010 R3 2 hours and 10 minutes - approximately 18 times slower.

There are some intruiging VSS errors in the application log at the same time the backup was running. Intruiging because the test job doesn't have advanced open file enabled...

Tomorrow's test of rolling back to pre: BE and trying a quick test install will be the final test before raising a support call.

Cheers, Rob.

[robnicholson]

It also indeed Windows update that probably put the Quantum driver on there. Just run Windows update and it was shown as an optional update. I thought "What the heck..." so installed it and, not surprisingly, it has changed the device driver.

Cheers, Rob.

[robnicholson]

I've rolled the server back prior to the installation of Backup Exec and the event log is clean. Not of those nasty "service not starting errors". Also, the boot time and first logon are as normal for a server - none of the 15+ minute delays seen when the system had BE (and more Windows updates) on there.

For a laugh, I'm installing (whisper) ArcServer trial to see how it gets on with the test backup to tape.

If all is well there, I'll roll back the server again and try clean installation of BE 2010 R3.

Something obviously went wrong after the installation of BE... finger not pointed totally at BE yet though.

Cheers, Rob.

[robnicholson]

Restore server back to pre-BE 2010 R3, re-install BE 2010 R3 but without live updates or other drivers than the one that came with Windows 2008 for the library (circa 2006) and the Symantec ones for the LTO-3 drive (circa 2008) and the backup speed is a whopping 3,470 MB/min (with compression) which is 30 times faster than the current image is managing.

So the question is what exactly went wrong in the build... all that was done on top of the above was a) install live updates, b) install more Windows updates and c) configure B2D & dedupe storage systems. Not much opportunity there for me, the end user, to make mistakes.

I'll go through one by one, taking lots of images.

Out of interest, on this clean install, both "Write single block mode" and "Write SCSI pass-through mode" are checked so doesn't sound like that was the cause of the original fault.

Cheers, Rob.

[robnicholson]

Another day, another roll-back and detailed installation of "things" on our media server trying to repeat this problem.

Except today, I've got a breakthrough. Install the device driver for an external USB 3.0 host controller (from StarTech) and some weird "WD SES Device" that comes with a WD "My Book Essential" (crap name) external hard drive and BE starts reporting snapshot errors with tape, like we're getting on the live media server. Tape backup speed is still fine so not exactly the same situation but good to isolate the cause of at least one error.

Cheers, Rob.

[robnicholson · Accepted solution]

Found the culprit, well probably as just rebooting before opening the champagne. The culprit is the Quantum driver for the SuperLoader supplied by Windows update:

http://www.picpaste.com/slow5-J6HfeZry.png

On the test system, after allowing Windows update to run, this driver was installed and backup speed drops from a very acceptable 3,500MB/min to around 100MB/min. Way to go Quantum!

I'd let Backup Exec install it's own drivers but Symantec only supply one for the tape drive, they rely upon the existing driver. When installing Windows 2008 from scratch, it says "Unknown media changer" with a Microsoft driver. But that driver works fine.

Cheers, Rob.

[robnicholson]

Yippe!! Tape backup speed back to normal. Once you know the fault, it's easier to Google on the problem so I've just found this thread:

http://www.edugeek.net/forums/windows-server-2008/37701-backup-exec-12-problem-windows-2008-a.html

Exactly the same error and exactly the same fix.

Shame Quantum/Symantec haven't fixed this problem in the TWO years since this thread...

Cheers, Rob.

[pkh]

When installing Windows 2008 from scratch, it says "Unknown media changer" with a Microsoft driver

This is what it should be, i.e. the Library should be listed in the Device Manager as an Unknown Medium Changer with a Microsoft driver.

[pkh]

You cannot really blame Symantec because BE does not have a driver for libraries and works perfectly fine with the default Microsoft driver.  Without a library driver, it cannot change the driver even if you re-install the library in BE.

[robnicholson]

I know that now but it wasn't easy to discover that fact 

I've read the "best practise" saying you should use the Symantec drivers so I did that. But I didn't see any specific mention of what to do with the robotic library so one assumes that the manufacturers own driver (installed via Windows update) would work! This is an obvious bug and my finger is now pointing towards Quantum as they supplied the bad driver. It's not like Windows 2008, Quantum SuperLoader and Backup Exec is an usual combination or that any of the companies involved in these three products are some unknown East Asian manufacturer.

Cheers, Rob.

[robnicholson]

IMO it is a reasonable assumption that when Symantec says "It's highly recommended to use our drivers", then it is indeed using their drivers if you select that option. The fact that this only covers the tape drive and not the library is not obvious. So I can slightly blame Symantec for not making that obvious during install. We write software. When anything goes wrong with our software, we are blamed even if it's nothing to do with us. For example, SQL Server running out of disk space. They say "You should have built in warnings" ;)

But I agree that the villan in this story is Quantum. That post that I found above is over two years old and Quantum has not fixed it. It is highly unusual these days for a driver update to make things so much worse. Unusual, but not unknown.

Especially for a manufacturer of a technology that's dying more and more each day that passes. Maybe they don't care anymore...

Cheers, Rob.

[pkh]

What I find more strange is that the library driver caused the problem.  I would imagine that the library driver takes care of things like loading and unloading the tapes.  If it turns out that the Quantum driver for the tape drive (not the library) is the cause of the lowdown, then I would find it less strange.

[robnicholson]

Yes, I agree is seems very unusual that the driver for the loader would cause slow backup to the drive - which is why I wasn't looking at that driver until later on. I was (as you can tell!) pretty thorough about installing/updating bit by bit and it was definately the changer driver.