Progressive slowdown?!

[MilesT]

Hi,
I am running Backup exec 12.5 rev 2213 sp3 on sbs2008 sp3 64bit; LiveUpdate tells me that the service packs & hotfixes are current.

I am backing up my SBS, a separate Win2008 File Server and a Win 2003 SQL Server (via agents) as a snapshot to a set of 1TB HDs - A setup that worked fine for months! I had a Job Rate approaching 4,000 MB/Min on a regular basis. Then, starting last April, the Job Rate starting dropping; a little at a time. By mid-May, the speeds had dropped by half; the overnight backup went from 5-6 hours to 10-11 hours. Right now, my Job Rate has fallen to a rather pathetic 600 MB/min and the overnight backup is taking 20+ hours to complete!

I am running ESET's NOD AV - for more than a year w/no changes other than DB updates.
The SBS is running @50-80% of CPU capacity and 90-95% of available memory while the backup runs.
The SBS server's updates are current...

What can I check to determine the cause(s) of this slowdown? Please bear in mind that my background is in windows applicaiton development; not server maint. I may need to be told how to check the inner details. :)

[CraigV]

Hi Miles,

Some Windows updates have been known to cause a slowdown for some or other reason.
However, in this case, I would firstly look at restarting your services. If this doesn't improve, see if you can get a restart of your server.
Some BE updates need a restart...I have personally seen a non-restart hammer my backup times.
Try the above, and post the results here...

Craig

[MilesT]

Thanks for the tip Craig!
I have had one update (for powerpoint of all things) stuck for awhile; I finally has to install powerpoint on this server to get it to apply and clear the way for more windows updates. There are 11 more updates pending on this server; I am applying them now; I'll reboot and I'll post tonight's backup run result in the AM.
-Miles

[MilesT]

Well, the backup has been running for 11+ hours now and the job rate is a little better @ 835 MB/min. But, that's still nothing like the 3,500 to 4,000 MB/min I was getting.

So:
What settings in BE can I examine to potentially enhance the performance?
What are some of the performance "gotchas" to look for? I could really use a checklist! :)

-Miles

[CraigV]

Hi Miles,

A couple of things...

1. Make sure Removable Storage service is stopped and disabled to prevent any conflict with BE controlling your device.
2. Use the Symantec drivers...that is what they recommend.
3. Make sure that your NICs and switch ports are hard-coded to the fastest that the NICs can go to.
4. Under Devices in BE, you can right-click your drive and choose a higher buffer setting. Mine is set to 1MB. It offers SOME speed, but it isn't massive. Any little bit will do.
5. Enable Large Send Offload on your server's NICs. I saw backups complete in 40% less time on 1 of my sites. Just check your server's performance...it might have a bit of a hit.
6. Check and make sure that your antivirus isn't scanning your BE installation directory...I had Symantec support get me to change this on 1 of my servers, due to it constantly scanning the beremote.exe file.

Laters!

[MilesT]

Thanks for the pointers Craig! I do have a few questions though:
1. I can't seem to find a "Removable Storage" service or anything remotely like that in the server's serivces listing... Could it have another name in Windows 2008 (x64)?
2. I will check on the drivers ASAP.
3. As it happens, the NIC and switch are set up to use the fastest settings - I did find that the NIC was set to power off (sleep mode) if usage falls under a threashold... I disabled that!
4. I'm not finding a buffer setting under "devices" in BE at all. Am I looking in the right place? The NIC buffers are set fairly low (512 out, 128 in), perhaps raising them would help?
5. The Large Send Offload server NIC settings were already Enabled.
6.I have disabled ESET's NOD AV's AMON (Active file-system monitor) and Windows Defender.

I've been watching a running BE job off and on for about 3 hours today and the speeds are bouncing between 3,000 and 1,500 MB/min - It seems to start off fast and then really slow down, then pick up the pace a bit, then back to slow, it varies over time...  The pace is better, but not where it used to be.  I have checked and I am the only one on the entire network today... There aren't any server processes going - no defrag or updates, etc... 

What more can I do?

-Miles

[MilesT]

Well, @ 7.5 hours into the backup we are @40% of the job and the job speed has decreased to 750 MB/min...  :( 
It seems to me that there must be some kind of cumulative issue (memory, buffers, ??) as performance is in a progressively slower pattern,,,

[CraigV]

Hi Miles,

RSM isn't present in Win2K8...didn't know that was what you were running.
Try disabling Large Send Offload...this is kind of a hit-and-miss affair...it sometimes helps, and it sometimes doesn't. It has helped on at least 1 of the 2 servers I have enabled this on...
Right-click your tape drive, and go to Configuration. I always enable compression, and have the following settings:

Block size (per device)   -   64KB (changing them to a higher setting may result in you not being able to restore your data...test that change first!!!)
Buffer Size (per device) -   1MB
Buffer Count                   -    10
High water count             -   0

The other settings below that are left off.

Craig

[MilesT]

23 hours later the backup completed showing a job rate of 648.00 MB/min... :(
This same job used to take 6 to 7 hours...

I will try disabling Large Send Offload and see if that helps; thanks Craig - I appreciate you staying with me on this!

[MilesT]

My Windows Application event log shows "SQL Server has encountered 1 occurrence(s) of cachestore flush for the 'Bound Trees' cachestore (part of plan cache) due to some database maintenance or reconfigure operations." - Error codes: 2803 and 17137 - happening A LOT.

This is a potential source of performance issues; or so I am led to believe by Google Search... :)

So, per: http://seer.entsupport.symantec.com/docs/351289.htm I have reset the auto-close setting for the databases to false - I hope this helps... I'll post the results when I know one way or the other...

Cheers!

[CraigV]

Did you come right Miles? If so, can you close off with the solution?