Slow backup to tape from remote agent with new server

[David_Walsh_2]

Hi,

I have recently upgraded a customers system, and after the upgrade we are seeing terribly

slow backups to tape from a remote server.  Details of the system before and after the

upgrade, plus backup performance statistics are below:

BEFORE UPGRADE

SERVER1 - HP Proliant ML350 G4 running SBS 2003 Premium with Backup Exec version 10.0 plus

HP Ultrium 1 tape drive

SERVER2 - Old Compaq Proliant ML370 running Windows 2000 Server SP4 plus Backup Exec version

10 remote agent

Both server NIC's running at 100Mbps connected to a 100Mbps switch

Backup to tape from local drives on SERVER1 resulted in a throughput of about 500MB/min
Backup to tape from drives on SERVER2 (via Backup Exec agent) resulted in a throughput of

about 250MB/min

AFTER UPGRADE

SERVER1 - Same as before

SERVER2 - New HP Proliant ML350 G5 running Windows Server 2003 Standard SP2 and Backup Exec

version 10 remote agent

Both server NIC's now running at 1Gbps full duplex (manually configured to this setting on

each server) connected to a new 1Gbps switch

Backup to tape from local drives on SERVER1 are as before the upgrade, i.e. about 500MB/min
Backup to tape from drives on SERVER2 (via Backup Exec agent) now result in a trhoughput of

about 20MB/min

In summary, I have installed a much faster new server and a much faster new managed switch,

which has resulted in a TEN FOLD DECREASE in backup performance to tape from the new server.

The intersting thing is that if I create a backup to disk device on SERVER1 and backup

SERVER2 using the backup to disk device as the target, the throughput is 1GB/min.  To me,

this rules out the new server, the new switch and the rest of the network, and points the

finger at the tape drive, but I can't see any logic to this.

Anyone else seen anything similar and can offer any suggestions for a fix?

Thanks in advance,
Dave.

[Ken_Putnam]

 To me, this rules out the new server, the new switch and the rest of the network, and points the

finger at the tape drive, but I can't see any logic to this.

Sounds about right.

From the Devices Tab, right click the tape drive\Properties

What are the settings for block size, buffer size and number of buffers?  Also, both Single Block settings?

[David_Walsh_2]

Hi Ken,

Thanks for your response.  Settings as requested:

Block Size: 64K
Buffer Size: 64K

Number Of Buffers: 10

Read Single Block Mode - unticked

Write Single Block Mode - ticked

This is an HP Ultrium-1 drive.

Many thanks,

Dave.

[Ken_Putnam]

Both single block settings should be disabled - That completely bypasses all buffering.  Read a block, write a block, read a block  etc

Block and buffer settings look ok

[David_Walsh_2]

Hi Ken,

Have altered the setting you suggested and done a test backup.  It's actually slightly slower.

Please bare in mind that the throughput of backup jobs on the backup server itself is perfectly acceptable.  There is some combination of factors combining when a backup of a remote agent based device is done across the network to tape, but for the life of me I can't make any sense of it!!!

Thanks,

Dave.

[nobby8]

Hi Dave

This is just a random though i had whilst reading your post.  I have checked the model of your new server and the network card fitted comes with TCP Offload Engine (TOE) installed. I work for a network installations and service company and we install Dell servers. They too have TOE enabled and we have had nightmares with it with various switches randomly on lots of sites. The new all powerful servers were REALLY slow on the network and we couldnt work out why. They were also really slow using BE remote agent (because the network speed was badly affected by dodgy TOE) This must have happened to 20 or more servers we have installed recently. If you dont know, TOE utilises other network layers to speed up data transfer (ie levels 1,3,4) instead of just level 2. It doesnt seem to work very well to me although I havent really researched the details of exactly how it works. Disabling it made everything work ok.

I suggest that to test if the new server is having network issues, on a workstation try copying a 50MB file from the new server and then from an existing server that works ok. You should be able to see if the file transfer is much slower. This might indicate that you may have the same problem.

To disable TOE type at a command prompt on the new server

Netsh int ip set chimney DISABLED

To enable it again type

Netsh int ip set chimney ENABLED

It worked for us, hope it helps

Ian

[David_Walsh_2]

Hi Ian,

What can I say, FANTASTIC!!.  You've saved the day and I've still got a couple of strands of hair left!!

Throughput is now about 350MB/min from that remote server which is up on the previous server and is acceptable.

Many thanks for taking the time to respond.  Have a virtual beer on me.... 

Cheers,

Dave.

[Ken_Putnam]

Dave, I'm glad you've still got some hair left 8^)

but I'm still a little confused.  If the problem was that network setting, why did your Disk backups work OK?

[nobby8]

Hi Ken/Dave

The disk backups worked ok because they are local to the Tape device so therefore not affected by network performance. The disks and the tape drive are probably on the same SCSI backbone and throughput should be rapid. Just for reference neither Dell nor Broadcom admit there is problem with TOE. We found out by accident when looking through Linux forums and found that the developers had decided not to write any modules to support TOE on Linux because it was too buggy. (Linux comes native on Dell servers now)

Glad it worked out OK Dave

Regards

Ian

[Ken_Putnam]

The disk backups worked ok because they are local to the Tape device so therefore not affected by network performance

Ian, if you read Dave's   first post, he said that B2D backups of the remote server were acceptably fast, only tape backups of the remote server were slow

[David_Walsh_2]

Hi Ken and Ian,

Just to reiterate:

SERVER1 is the backup server (tape drive and backup to disk folder are both located on this server)

SERVER2 is the backup source

Both servers connected to a Gigabit switch

SERVER1 backs up the files on SERVER2 across the network to the backup to disk folder located on SERVER1- throughput is absolutely fine

SERVER1 backs up the files on SERVER2 across the network to the tape drive located on SERVER1 - throughput is abysmal

I agree this makes absolutely no logical sense, but disabling TOE as Ian suggested resulted in backup to tape performance returning to normal.  Interestingly it is no quicker than previously (even though SERVER2 is a brand new server; much quicker than the server it replaced, and the whole LAN has been upgraded from a 100Mbps hub to a fully managed Gigabit switch).  However, at least the backup now completes in the overnight window, instead of taking over 20 hours!!!

I would like to understand why the above is as it is, but for the moment I think I will steer clear of TOE!!

All the best,

Dave.

[Scott_Chandler]

I too had the same problem remotely backing up (2) new HP servers (BL25p G2, BL20p G4) both running Win 2003 x64, SP2. In addition to the fix listed (netsh int ip set chimney DISABLED) I deselected (disabled) the TCP Offload Engine check box in the HP Network Configuration Utility Program, properties of the adapter, Advanced Settings. Note, both settings are only available after SP2 is installed. Remote backups now running at ~1300MB/min 8-)
BackupExec 11d

GOOD FIND Ian!

[tmountsr]

I understand issue with TOE; however I want to permanently disable TOE.  There is a registry addition form another forum, but that person also said something about disabling TOE in the NIC itself.  How - I've looked at the NIC properties on the server and saw nothing.  Will the registry addition be enough?
Tom Mount
Superpac, Inc.

[Stephen_May]

Ian, what can I tell you bud

I also had ths problem, and just couldn't see where the problem was. I was the football last night (Aberdeen v Motherwell in the SPL) and this was all I could think of. Woke up last night dreaming about it.

I tried you fix and it works. Your a star!

Our backup was taking about 36 hours to complete, coupled with backup exec playing up badly. Installed new tape drive (we had to do this anyway as the organisation had grown so much) but no improvement in the amount of time to do the backup. Loads of backup exec errors etc. A reinstall of Backup exec cleared the errors, and the job kicked off fine at what seemed like a good speed. Come back in the morning to find the job is still running and realise the main server is backing up normal speed, but the remote one is slow as a dog! Found your fix, tried it, it worked. Thanks again.

Stephen