Media server slow to back itself up

[Smitty42]

I have a BUE 2010 R2 media server on Server 08 R2 platform on HP DL385 G7 hardware with a SAS HBA direct connection to an HP Ultrium 720 (LTO-3) SAS single tape drive (small environment).

The backup job for the media server itself (local job) all of sudden dropped from an average of 500MB/min to around 50MB/min, but none of the remote jobs have been affected.  All of the remote jobs run at 500 to 700MB/min over my 1GB network still.  The server has 16GB of RAM with only 7GB used and 24 processor cores.  Yes it is running other roles besides BUE, but those other roles are SharePoint and Terminal Services which are not accessed by users during the backup window (10pm to 4am).

Immediately prior to this change in backup speed, I had installed the BUE 2010 R2 service pack 1 via live update and the Windows Server 2008 R2 service pack 1 via Windows update.  Live update also installed BUE 2010 R2 hotfixes 150096 and 144101 on the same date corresponding to the change in local backup speed.  The server has been restarted several times since.  Windows update also installed the following MS hotfixes at the same time: KB2484033, KB2488113, KB2475792, KB2479628, KB2425227 and KB2485376.

Anyone else seen a similar issue with these updates or have any ideas of things I could check?  It's really strange to me that only the local job is affected and not any of the remote jobs.

[CraigV]

Hi there,

How do you select your local server in the selection list? Using a custom selection via IP address, or selecting the server itself?

[Smitty42]

In the BUE console, in a selection list, the media server itself is the first entry under "all resources", i selected parts of the media server to backup right there.  i don't remember changing this selection list between the last backup that ran "normally" and the first one that ran "slow".

[CraigV]

...OK, then what else does your server do? Is it a new issue that you are facing, or existing?

What is the server's performance like during the backup (both remote server and media server/s)?

[Smitty42]

I thought I explained some of that in my first post, but I will reiterate for clarity:

This server also hosts one Windows SharePoint Services 3.0 website and several virtualized applications through Terminal Services Remote App and Web Access, however, these are only accessed by users during the workday.  The backups all run late at night into the wee hours of the morning so the two roles do not interfere with each other's performance.  The server's antivirus (Symantec Endpoint Protection 1106 MP2) is set to run it's daily scan in the early evening so as not to interfere with backup performance and has the appropriate scan exclusions for the Backup Exec 2010 directories.

This is a new issue that just appeared last week on Monday night's daily incremental jobs.  The Saturday prior to this new issue appearing of running slow on it's own local backup job I had installed several Windows updates as well as Service Pack 1 to Server 08R2. 

Unfortunately, I do not record performance data in general on my server(s).  I know I should but I don't.

I do have a support agreement in place, so I could call them but I prefer to start here instead of spending all that time on the phone.  I should probably start by reversing/uninstalling the Windows updates and Service Pack that I installed right before the problem appeared.

[CraigV]

Uninstalling those updates is a good place to start. Install the SP, and then check what happens. Ideally you're doing that 1 update at a time, and if you can, also update BE. SP1 for BE 2010 is also out, along with a couple of additional patches...

[Timo_Schoenfeld]

Uninstalling those updates is a good place to start

You cannot deinstall SP1 for Windows Server 2008 R2.

Symantec supports Server 2008R2 SP1, so please compare the Backup-Time of each partition, Sharepoint-Datebase, SystemState and so on with the backup time before the problem occured. Than we can go further....

Regards Timo 

[Smitty42]

CraigV:  My BE 2010 does have SP1 and two other hotfixes already, live update reports no more updates available and all remote server agents have been updated to the same level from the media server.

Timo:  Service pack 1 to Windows Server 08R2 uninstalled successfully.  It left behind a slew of hotfixes though, so i guess it uninstalled the new features but left the rolled-up updates.  We'll see how it runs tonight.  In comparing the times of the individual components of the job, the slowest component was the System State.

[CraigV]

...the System State part of your backups will always be slow when the server in question is Server 2008. It is a known problem which is apparently addressed in BE 2010 R3 which is currently in Beta...

[Smitty42]

Yes, I read an article to that effect in the Symantec support KB.  If I am not mistaken it suggested unchecking the boxes for "backup using junction points" and "backup using symbolic links".  I think I tried that first and it seemed to help a little on the remote backups of the domain controller and exchange server.

The media server was taking over 6 hours to back up 12GB of its own system state though, which i thought was excessive, especially since it had been a lot faster before and all of a sudden got that slow.

Anyway, it seems to have fixed itself.  I did uninstall SP1 to Windows Server 08R2 yesterday morning and last night's job ran at 400M/s which is about what it was before.  I can live with that. 

Thanks for the suggestions!

[CraigV]

...alternatively use the built-in Windows Backup software to back up that System State to a file on the server in question, and then use BE to back that up!

[Timo_Schoenfeld]

Sorry i confound  SP1 for Exchange 2010 with SP1 for Server 2008R2 ..... Yes you can deinstall SP1 from Server 2008R2, but is that a solution for us ? Again we need for Symantec to deploy SP1 to the servers.

We had the same subject with Exchange 2010 DAG, where Symatec was also the blocking part in our roll-out.

System-State problem is konw since month, so why did symantec not provide a hotfix for that to their customers ?!

Regards Timo

[Smitty42]

Timo:  I am not 100 percent certain that the SP1 to Windows Server 08R2 was really the problem, but it is interesting that my problem appeared immediately after the installation of it and then disappeared immediately after the deinstallation of it.  Since I have a paid-up support agreement I should have opened a support case with Symantec on it rather than work it out myself so they could in turn confront Microsoft about it.  Oh well, maybe I will reinstall the SP and see what happens.

CraigV:  I never would have thought to do that, thanks for the tip!

[Smitty42]

PS- my server just sent me an email saying two new hotfixes are available for download, 153090 which doesn't show up in the KB search and 148347 which addresses several SharePoint 2010 and Exchange 2010 issues.

[Timo_Schoenfeld]

@ CraigV

I paid for software which has the ability to backup the systemstate, and for me it's no solution to use build in SW on about 100 Server....

@ Smitty42

I seems to be that only the backup of memberserver takes really long. In the moment four server are running with SP1, two memberserver and two DCs.

DCs systemstate is running with abou 300 MB/min (Which also not what i call fast)

On memberserver the backup is much slower, between  30 and 100 MB / min, and what is strange, differential backup are slower than fullbackup, even when on a logical view systemstate backup is always full backup.

Maybe i will also make a case..

Regards Timo

[CraigV]

...oops, sorry. My suggestion was directed to the OP... it was suggested HE try it. That would be the work-around for now. It would be a temporary thing, and I have never advocated it being a permanent fix. It's a trade-off between VERY slow backups, and gettng your backups through in a reasonable amount of time while working on the issue...

BE 2010 R3 apparently sorts this all out, but it appears to be an MS issue...not Symantec's.

[Timo_Schoenfeld]

@CraigV

"BE 2010 R3 apparently sorts this all out, but it appears to be an MS issue...not Symantec's"

Hmh hard to believe... Why does R2 than solve those problems.

Sounds a littlbe bit the chicken and egg principle: "Which Came First, The Chicken or The Egg ?"

Regrads Timo

[CraigV]

...I don't know about R2 resolving the problems. Again, my suggestion was directed at the OP, not your environment.

If you have the same issue, open a case with Symantec and let them sort it out. However, the forums are pretty much full of it. It's a known problem, and we can only hope that it gets addressed in the latest version.

[Timo_Schoenfeld]

Sorry - Why R3 schould solve the problems.

I will get near look on my systmsstate backup now and will open a case as soon as possible.

Regrads Timo