Slow CPS Transfer Rate

[Mike_Janeiro]

I have recently upgraded to BE11D but have been experiencing very slow transfer rates when using CPS for backups. My transfer rates are about 1-3MB/min and have not found any support documents that have helped improve the performance.

As a test, I also created a Backup to Disk job using Backup Exec (Not the CPS) and when I backup the same data this way, I get an average transfer rate of 800-900MB / Min.

What could be causing CPS to have such a slow transfer rate?

My System Setup

Dell 2900 64-bit Quad Core server

Windows Storage server

4GB Ram

4.3TB Drive space for backups with HW Raid 5

Thanks in advance,

Mike

[D_B_5]

I don't have CPS installed at the moment where I can look at it, but there is a network traffic setting which you may have set by accident.  It may be on job properties...  It allows the user to set a percentage of network traffic volume to use, or some similar terminology.  If you enabled it by accident, you may have throttled yourself down as a result.

[Mike_Janeiro]

Hi D B.

I know exactly which setting you are talking about, however that setting is set to use 100% of my 1GB network connection. I've even tried to scale it down to 50%, 10% but I'm always at the same 1-2MB / Min.

Mike

[D_B_5]

Mike, I emailed a friend who works with CPS more than I do, here was his response:

I would start with CPS Job bandwidth. Make sure that did not change. Also where is the backup destination? If it’s remote that could be the problem. If it’s local, what drive is it on? Was the Backup Exec B2D file created on the same volume as the Backup Destination? If so we can rule out the volume. Can he copy files from the remote machine to the CPS backup Destination volume (same data)? I had a customer have a slow throughput when manually copying files to the same location.

[Mike_Janeiro]

Hi D B,

The B2D folder I created in BE is on the same volume as where I created my backup destination on CPS.  All the servers I'm backing up are all on the same subnet connected to the same 1gb switch. So yes, we can rule out that it's the volume causing the issue.

I've been going back and forth all day with Symantec Support and they are of no help at all. All they keep telling me that this is normal transfer rate, and that the reason why I'm getting the speeds I'm getting is because with CPS it is "replicating" the data and not "backing up" the data. I understand that it's not "backing up" the data and that CPS does other tasks in the background to make sure the data is synchronize properly, but it doesn't explain the drastic difference in transfer rates.

One of the suggestions was to create a temp CPS job with only one file selected to be able to create the folder structure, and then manually copy the data to the destination folder since this way is faster. When I heard this, I was floored. I would be embarrassed to give out a solution like this to any customer if I were Symantec. You pay a huge premium for this software and it doesn't even work properly out of the box.

In any case, I'm trying to get a hold of many a more experienced engineer to see if they can help out, if not I believe this software is headed for the dumpster.

Mike

[D_B_5]

It's not the software per se, since many people are not having throughput issues.  It's just a matter of figuring out what is causing the issue in your environment.

Did you try the suggestion of manually copying one file and dropping it into the destination folder to make sure there is not a throughput problem outside of CPS?

During the install of CPS, there is an environmental checker.  one thing it notes is if you have the correct setting in Control Panel\System\Advanced\Performance\Advanced\   It states that changing these settings (Processor scheduling and Memory usage) from "background services" or "system cache" to "programs" is more efficient.  Have you tried that?

[Mike_Janeiro]

D B,

I did make the change that the environmental checker recommended right after I installed CPS and I have been testing with both the default settings and the recommended change.

Here are some results that tend to be consistent with all the testing I have done. With all my tests I have used the same 500GB folder. Here are the results

BE 11d B2D folder : 44sec

Manual windows copy: 34sec

CPS: 8 minutes

So as  you can see from the results above, I'm leaning more towards CPS as the bottleneck. I can't see how it would be my enviroment if using manual copy or B2D folders work fine.

Mike

[D_B_5]

Mike,

One question...  are there tons of files within your 500GB folder?  It is possible this is just the first time synch which is running slow due to a sheer number of files.  After the initial synch completes, add a bunch of files and see if the throughput is suddenly fast from the original target and in range of the B2D or manual backups.  It could be that you are just seeing the initial synch taking a long time as it prepares to run continuous protection on each file as it looks for new versions of files and such, it needs that initial baseline to start with.

[Mike_Janeiro]

Hi D B,

The 500GB folder I have contains a variety of sizes in files. I did try your suggestion but unfortunately I get the same results.

You mentioned earlier that you know someone that is using CPS. What transfer rates does he/she get when using CPS for normal jobs and baseline jobs?

Mike

[David_Bond]

Hi,

I have CPS installed, it currently runs at between 1 - 4MB/s, (1Gbit lan) on the initial sync (file size dependent). But this is on the second sync that i have done. The first sync, I had slow speeds, less than 100KB/s, but this was with a larger data set, I had to turn off the CPS agent on one of the file servers that had over 500,000 files, as it would stop responding when it was used. Then after all the files are copied, its from a few KB/s to 1-2MB/s depending upon the size of the files that were changed at the time.

David

[lanman1]

I am having the exact same issue.  Throughput with 10d was much better.   

One thing I did notice is that if I do a perfmon on the machine that hosts the backup destination, pages/sec is way high, it bounces aroung 900-1800.  Disk IO and processor are relatively low.


Backup Destination
2003 R2 SP2     2G memory, dual core Opteron 2Ghz processor, 3TB raid5

Backup Target
2003 R2 SP2     2G memory, dual core Opteron 2Ghz processor, 1.5TB raid6

Message Edited by lanman1 on 08-03-200706:41 AM

[lanman1]

I noticed that (sometimes) even when I cancel a job, Pages/sec stays high until I stop the Indexing service.

According to Intel:

0-200 ( > 200 warrants investigation into memory subsystem; define reads (pages in) versus writes (pages out); check for proper paging file and resident disk configuration; May indicate application memory allocation problems, heap management issues)

I am averaging 1200 and peaking at 2400

Hope this helps. 

Message Edited by lanman1 on 08-03-200708:45 AM

[Mike_Janeiro]

David,

You also have a 1GB lan and you are getting really slow transfer rates as I am. Don't you think this is wrong considering that if you just use BE or just copy paste it transfers much faster up to 20x + faster? I understand that CPS "Replicates" the data and might take longer to copy because of synchronization checks, but not this much of a performance hit.

I have given up on the CPS and going back to just regular tape backups with my BE9.1 software until Symantec's competitor comes out with their next version of their D2D2T solution. I have used the current version of the competitor software and I never had transfer rate issues like this and on top of that I will be able to protect Exchange, SQL, File servers, and more from one console. The only reason for going to BE11 with CPS is because I only needed to upgrade my license and not purchase a new backup solution. Hopefully other administrators with not accept the slow transfter rates and give their feedback to Symantec so that just maybe, they will get it right in BE12.

In any case, thanks for all your feedback and suggestions, unfortunately according to Symantec, the 1-2 MB/s is "normal", but for me it's not acceptable at all.

Mike

[David_Bond]

Mike,

Hi, I do think its slow compared to just running a backup, which can also be very slow sometimes. For me, doing a standard backup, speeds can vary on the same server from 0.5MB/s (30MB/m) to >50MB/s (3000MB/m) depending on what its backing up. One of my servers takes 5 days to do a full backup (which it just completed today from a Friday full backup, well it didn't complete, it couldn't backup the system state for some reason so failed), this is one of the servers that I wanted to use CPS on, so that it could sync all the files to the other server and just copy over, when changed, any other files, as it takes so long to do a normal backup on it. So the initial sync time didn't really bother me, as it takes so long anyway to copy files across from this server anyway (averaged this time 41MB/m). But this is the server that becomes unresponsive when using CPS on it, and also the server that has 15,000,000 files.

But what I'm trying to say is, as long as it works, and it keeps the files in sync, it shouldn't matter that it slow on the initial sync, unless you are doing a scheduled sync every few hours and lots of files change. I'm not saying it has nothing wrong with it, I do think it should be improved, as it doesn't work for me, for what I want it to do. But I have yet to look in to the reason why the server becomes unresponsive.

David

[lanman1]

It matters to us for data restore in the event of a failure.  We are using CPS so that we don't lose data in case of a failure.  A couple of days to restore is unacceptable when CPS 10d would do it in ~4-6 hours.