Improving a very slow Job Rate (Read the Technical Documents but no luck)

[Gene_Tang]

Hi there,

We're currently trialing Veritas 11d to replace our current backup software (Brightstor ARCserv 11.5) due to its unreliability. For the most part Veritas is working reliably, but at present it is working ridiculously slowly (i.e. the Job Rate is very low), to a point where it is unusable for timely backups.

Firstly, the current setup we have:
We have about 25 servers that are to be backed up. We have about 7 or 8 Windows servers (Windows Server 2003 SP1) aHi there,

We're currently trialling Veritas 11d to replace our current backup software (Brightstor ARCserv 11.5) due to its unreliability. For the most part Veritas is working reliably, but at present it is working ridiculously slowly (i.e. the Job Rate is very low), to a point where it is unusable for timely backups.

Firstly, the current setup we have:
We have about 20ish servers that are to be backed up. We have about 7 or 8 Windows servers (Windows Server 2003 SP1) and 15 Linux servers. In our current setup each of the Windows servers has its own remote veritas agent, and we have one staging linux server with the remote RALUS agent. All other Linux servers rsync to this Linux staging server. All the servers are connected to a dedicated gigabit network with no other data flowing across it. All machines have been double check to connect at 1Gbps across CAT 6 cables and all backups are performed at night when no other processes are occurring on the machines.
The Veritas server (called bobcat) is installed on a dedicated Windows 2003 SP1 Server (2 x 3.0Ghz Xeon, 4Gb Ram), which is used as a staging area before writing to the Tape libraries (2 x Exabyte VXA2's). So backup data from all remote servers are staged on this Veritas server in a backup-to-disk folders before writing to the tape libraries. The staging area on the Veritas server are 1.8TB of SCSI drives in Raid 1.
For each Windows server we have to backup about 150Gb, and about 1TB of data off the Linux server.

Now the problem we're experiencing:
The job rate when transferring data from the remote server to the veritas server is extremely slow on both Linux and Windows servers. We are currently experiencing between 180mb/min to 400mb/min for Windows servers, and 200mb/min for the Linux server. From this forum and searching on Google, I�m seeing that many people are achieving well into the 1700mb/min to 2000mb/min.

At the job rates we are getting, it takes 101 hours to perform a successful full Linux backup to the backup-to-disk folder, and another 100 hours to write to tape. It takes about 4 days for a successful Windows backup. This is unacceptably slow since this means we can only ever have full backups and not incrementals. Our previous backup software took 12 hours to perform a successful windows backup and 3/4 day to perform a successful Linux backup, including staging to �bobcat� and writing to tape. The setup with the previous backup software is identical to what was described above.

I have been through this forum thoroughly and seen and followed the instructions on several threads and technical papers including those listed in the following:
http://forums.veritas.com/discussions/thread.jspa?messageID=4409958䩦
http://forums.veritas.com/discussions/thread.jspa?messageID=4376568쟸
http://forums.veritas.com/discussions/thread.jspa?messageID=4442467쥣

Despite implementing these changes I have not seen any increase in the job rate. That is for example, I�ve defragged all the Windows machines, turned off pre-scan, ensured no antivirus software is running, enabled hardware compression/software compression etc, and we have yet to see a performance increase.

I have also checked the network performance of bobcat by ftp�ing a file across the dedicated network, and was able to transfer a file across at 60mb/second consistently, thereby ruling out problems with the network and the hardware on bobcat.

We have also tested the backing up of large files only (>1gb ISO images), since backup performance should be fastest with large files. In this case we were only able to achieve a job rate of 333mb/min for Linux and 400mb/min for Windows machines. This is still well under the previously stated 1500mb/min other people are achieving

To rule out issues with the Backup-to-Tape folder I have tried backing up directly to tape, however, this saw no improvement in job rate.

At this point, I am unsure how to tackle the problem to improve the job rate. If anyone could give me some ideas about how I could improve the job rate, they will all be welcome. I really like the veritas backup software, however, cannot proceed to purchase it if backup jobs are so low.

Thanks in advance for you help.

Gene

[Ken_Putnam]

I have also checked the network performance of bobcat by ftp?ing a file across the dedicated network, and was able to transfer a file across at 60mb/second consistentlyI

60mb/sec? that's horrendous on a GB network!

If your NICs and switch ports will permit it, try hardsetting them rather than relying on AutoNegotiate

[Gene_Tang]

Hi Ken, thanks very much for the reply. We have indeed changed the NICs and changed the Gigabit switch to see if that was the issue. All NIC's are also set to 1Gbps Full-duplex. We have also double checked that.

Out of interest sakes, I didn't realise that 60mb/sec was that bad? What is the expected speed? Just in case there is some confusion, when I say 60mb/sec, I mean 60 megabytes per second, not 60 megabits per second. This is ftp'ing a file from a linux server to a windows server.

Ftp'ing the same file from a linux server to another linux server, we managed to get a speed of 100 megabytes per second. So somehow linux seems far more efficient.

I remember doing some research on this a few weeks ago, and over copper, ethernet is 60-70% efficient. So according to my calculations:

1 gigabit/sec = 1000megabits/sec
1000megabits/sec = 125megabytes / sec

Therefore a gigabit network can in theory transfer 125megabytes /sec.

Now in the real world, the network is 60-70% efficient then :
125 * 0.7 = 87.5megabytes / sec.

So that is what I was expected? 60megabytes per second is still a fair amount lower than this, but every single windows machine we tested (about 8 of them) were only able to get 60megabytes per second. On the other hand, an identical spec linux machine was able to get 100megabytes /sec. Go figure...

What speeds do you get?

[josh_Hughes]

Hi Gene,

I am not sure if this will help you, but it may be worth a try.

Click Start then run
Type gpedit.msc
Expand Computer configuration
Expand Administrative Templates
Expand Network
Click QoS Packet Scheduler

Here you will find a setting called Limit reservable bandwidth. Microsoft limits the system to 20% of the bandwidth of a connection. Enable this setting and set the Bandwidth limit to 0. See if this improves your connectivity.

[Gene_Tang]

Hi there,

I tried this, however, it gave no increase in performance. I've been watching the performance monitor on Windows, and have found that this machine is only utilizing 4% of its network during backups. I have double checked that it is connected at 1000Mpbs Full-duplex (across CAT6 cables), and no other applications are running both on the media server and the remote server.

I have seen one breakthrough however, but not a major one. If I backup direct to disk, the transfer rate is a low 200mbps consistently. If I backup to tape directly (i.e. no staging area), I get around 350mbps. However, when starting the backup to tape directly, it reaches a peak of about 700mbps then gradually drops to a consistent 350mbps.

Anyone got any ideas still?

[Robert_Kuhn]

Maybe try setting up a VLAN? What other sort of traffic is on your ethernet?

What is the make and model of your switch? I also assume that there is nothing else in between your servers and the switch (i.e. additional switches/hubs).

As far as the FTP being slower on Windows, are you doing this on the command line or through a FTP program? And are you doing a passive or clear mode FTP?

[shweta_rege]

Hello,



Kindly refer the folowing Document;


Improving Backup Exec performance


http://support.veritas.com/docs/285756




Thank You,

Shweta

[Ken_Putnam]

(thought that I had replied to this already, but don't see anything listed)

Generally 10 mb (small "b") refers to megabits, 10 MB (capital "B") is megabytes

Anyway -

if your network seems to be OK (you can FTP or copy files rapidly) check the device settings

What are the block size, buffer size and number of buffers?

Do NOT enable any of the "Single BLock" settings

[Rucha_Abhyankar]

Hi Gene,

Please let us know if the earlier suggestions given by Ken have helped you.


==============

[Sanjay_Raval]

Hi,

Sorry to say that none of the Technical documents are helpful to resolve this issue. As i am also facing the same problem, suddenly backup rate reduced to 40 to 60 MB/sec, backup rate was good till previous successful backup.

Tried almost all suggested techinical documents,.but still facing same problem...

waiting for solution

[shweta_rege]

Hello Gene,




Please let us know if the earlier suggestions given by Ken have helped you.


==============



Thank You,


Shweta