SharePoint 2007 backups getting poor performance

[RGuinn]

I am backing up out SharePoint environment using the NBU SharePoint agent.

Here is my environment:

Master Server AIX 5.3 running NBU 7.0.1

Media Servers SUSE Linux 11 running NBU 7.0.1 - backups of SharePoint environment go to disk using Media Server Deduplication.

SharePoint farm - all Windows Server 2008 R2 running NBU 7.0.1 - front end servers all VM - backend SQL database - 2 Physical Dell 710s clustered running SQL 2008 R1

When I initiate the database backup through a frontend server (the SharePoint Central Admin Server) the parent job kicks off 2 child jobs. The first backs up the index files from the Search index server and gets ~25 to 30 MB/sec throughput. The second job backs up the SQL databases and only gets ~1 to 2 MB/sec throughput.  The entire database is ~ 700 GB and at this rate the backup is going to take 6 to 7 days!!!

I tested these backups in our DEV/QA environment and had no issues at all, using the same domain accounts, same Media Servers, Master Server, etc.

Has anyone seen this or a similar issue?

[watsons]

Some steps to isolate where is bottleneck it is:

Find out the differences between DEV/QA & production environment.

In production env, try testing a normal filesystem backup of about 100GB to msdp pool, is it slow?

Try testing the sharepoint backup (non-GRT) to a tape storage unit, is it slow?

[Omar_Villa]

Try to send your backup to a temp Disk STU just to see if the deduping process is slowing down the backups, running an index doesnt compare with the full SQL DB. 

Other thing that worth to check is the network paths taken, are your dev environment running the same load as production Im sure they are not, so what is the network pipe available that you have, maybe you are running your backup in prod across the production LAN.

Just a couple of thoughts.

Regards.

[RGuinn]

The differences between the PROD and the DEV/QA environment are:

PROD is much larger, (650 GB versus 100 GB), PROD is also on a clustered environment where DEV/QA is a standalone server. PROD is also much more utilized, (it is a global sharepoint environment with thousands of users).

I did make some changes late last week,

1. increased communications buffer size to 64KB on the client
2. created touch files SIZE_DATA_BUFFERS, NET_BUFFER_SZ, and set them both to 65536 to match

After the changes I started the database backup again and the performance improved dramatically to 30MB/sec throughput and it ran like that for about 12 hours and then the performance dropped back to 2MB/sec. It has been running now for 60+ hours and I have backed up 550+ GB.

The network pipe from the client is 2 X 1GB teamed and to the Media server is 4 X 1GB teamed. This does run on our public network as all of the frontend servers are in a VM and the VM team has not configured the hosts in the backup network.