SLOW BACKUP PERFORMANCE after installing SP2 for Backup Exec v11 7170 (with all hot fixes up to #39)

[Kingsley_Mok]

Have anyone notice SLOW backup performances after installing SP2 and all the hotfixes???

For us, the performance was PAINFULLY slow.  For example, backup of the Exchange IS usually average between 950mb to 1000 mb / min.

After installing SP2 and all the hotfixes...backup of the Exchange IS is averaging 85 mb / min. 

We've noticed that other SQL servers are having the same slow backup performance.  What normally took 30mins now took 4 hours?????

That is just UNACCEPTABLE performance.

[Rich_Lichvar]

I've noticed a similar problem on one of our SQL Server machines. All of a sudden performance that was hundreds of MB per minute has dropped off to 50 MB per minute, sometimes even one tenth of that! The funny thing is, this hasn't happened on other SQL Server machines. Really weird!

[Kingsley_Mok]

I've noticed that too.

Some SQL servers took longer.  The worst was backing up the Exchange IS and it's flat files.

I even did a network throughput test as suggested by Symantec technician and the the results come clean at over 1000 mb / min on our giga network.  Obviously, it is not our network.  I kept arguing to them that nothing was changed on our network or server.  Only thing changed was installing the SP2 and it's hotfixes.

For all the backup jobs, I print out the weekly full and daily differentials everytime when a job is completed to monitor backup job performance.  I mainly look at all backup summary showing the size of data backed up, job rate, and time it took.

Since SP2 and hotfixes up to 39, I've done many backup tests and compared it with all the print outs for the past year before SP2 and it is obvious that the performance has downgraded.

[Rich_Lichvar]

We found out what most of the problem was on our side: a corrupted a Symantec Backup Exec System Recovery installation on the media server! Seems like one Symantec product actually killed performance of another. (In reality we weren't using the System Recovery software on that machine anyway and the performance problem simply went away after it was uninstalled.)

Still and all, we have seen slightly degraded backup throughput performance on machines running SQL Server.

RichLich

[JTF]

I started having similar problems right around March 28th, 2008. BE was running fine until we applied the recent hotfixes. Jobs that were running around 3.5 hours now take 11 hours to run. I've uninstalled & reinstalled BE to no avail. I've also disabled AOFO & updated tape drivers in hope that this might help for now.

One other thing is that now our server is showing multiple VSS errors in the event logs. I did some searching and found some related issues but, nothing to currently helps solve our issue.


Message Edited by JTF on 04-07-2008 10:11 AM

[JTF]

Ok, the issue wasn't Symantec's update. It was a certificate issue. We run WSUS and I used SelfSSL (multiple times) to create a certificate to encrypt traffic with the WSUS server. Problem was that I never deleted the old certificate. Basically, SQL doesn't like multiple certificates and/or expired certificates for the server it's being run on.

[Kingsley_Mok]

Rich and JTF,

Thanks for your insights.

JTF,

We don't use encryption as it slows backup peformance a bit for us.  Just straight backup.

Rich,

For us, we use BESR v8.x  with BEFW no problem.  I can guarantee that it is not BESR causing the problem as I had ran another test.  This is what I've done last 2 weeks after noticing the slow backup peformance on Exchange IS after installing BEFW v11 SP2:

1.)  Made a BESR image of our BEFW Server so that I can recover easily onto a spare 500GB USB HDD.

2.)  Completely wiped down the computer clean slate.

3.)  Installed Windows Server 2003 Standard Edition with SP2 and all the latest updates from Microsoft.

4.)  Installed the latest Adaptec SCSI drivers for the SCSI adapter connecting to the LTO-2 Tape Unit.

5.)  Installed Backup Exec Server v11 7170 with SP1 ONLY (and not SP2 and so on).

6.)  I then installed BESR v8.x to make another image of it so it is easier for me to switch between images later on.

7.)  Uninstalled the BESW Remote Agents on our Exchange and restarted the servers on each two nodes to have it clean slate.  We have our Exchange on our backend on cluster.  So the procedure is very straightforward.  (Symantec Technician, please don't tell me it does not work on cluster as it has always worked since v10.x.)

8.)  Push installed BE Remote Agent from BEFW Server to Exchange active node then did a cluster failover to the other node push installed the BE Rmote Agent on that node as well.

9.)  Did a full backp test on the Exchange IS Only during after business hours and weekends.

Guess what?  It was averaging 950mb/min to 1250mb/min over gigalan with unmanaged gigaswitch and each switches are interconnected with fibre rated for 4GB/sec.

During those tests, I've printed encyclopedias of backup logs for proof and comparison.

Nex step, I've made another BESR Image (BEFORE_SP2) of this rebuilt BE Server prior to installing SP2.  After installing SP2, I've installed the hotfixes up to 41 and uninstalled the BE Remote Agent on the Exchange and rebooted the Exchange.

Did a push installed from BEFW (with SP2 and hotfixes up to 41).

Ran another full backup test on the Exchange IS only on after business hours and weekends again.

Guess what?  It was slooooooow.  It was ridiculously slow.  I've printed out the logs and compared it to the logs prior to installing SP2 and hotfixes up to 41.  From 950mb/min ~ 1250mb/min down to 50mb/min ~ 200mb/min.

I've made another BESR image of this server one more time (AFTER_SP2).  Just so I can slap on the image later for comparison.

I did a BESR Recovery of the image with BEFW with SP1 (BEFORE_SP2).  Uninstalled the Remote Agents on the Exchange and rebooted them, then did a push install of the Remote Agent.  Ran another Full backup test on the Exchange IS only.  I was doing this after hours...as in 2 am!!!  As I would suspect, the backup was excellent.  It averaged 1125 mb/min.

As you can see...all these tests and the BESR Images I have just shows proof... there IS a problem after installing SP2 and later hotfixes.  The logs don't friggin lie.  It's the same goddamn tape backup drive, cable and adapter since BE v10.x.  It is the same host computer so it is not the hardware.  I am tired of hearing Symantec Technician blaming it is the hardware or the switches or during high peak traffic time and so on. 

It was a long 2 weeks of testing and really a waste of our company resources when we know the problem is obvious with Symantec.  We have BESR images of BEFORE and AFTER installing SP2.  The backup results on in those logs don't lie.

Again, it is not the hardware issue as it was ruled out.  Not the peak hours as the full backup tests were done in graveyard hours and at the expense of my time away from a good sleep and family.  Yes, I am getting a little annoyed by all this because the Symantec Technician would tell me the same answer I had originally called in about this..."no issue"...well, I have images BE Server of BEFORE and AFTER installing SP 2.

[Kingsley_Mok]

Found the problem.... it is in the SP 2 and all the hotfixes thereafter...

Something was done to the AOFO feature.  I had this confirmed with Symantec's Advance Team.  And worse of all, no more development in Backup Exec v11...

Before, SP2 and all the hotfixes thereafter, we had all the resources (flat file, SQL, Exchange IS and Sharepoint) in one backup job with AOFO setting enabled.  According to the previous Symantec Technician whom I had chatted with, this would be fine and wouldn't matter much.

NOW, with SP2 and after, I have to seperate the backup job.  One for AOFO and the other non-AOFO.  This is a real stinker because we've ALWAYS had all the resources in one backup job with the AOFO featured turned on anyways ( because of the fact that we only have one tape drive and less figuring out the time window between backup jobs ).

After testing the two seperate backup jobs, I can see that non-AOFO resources are backing up properly and fast.  As for the AOFO resources... well...here's a another stinker...let's just just say the use of Symantec's AOFO Agent is USELESS.

If you install the Backup Exec  AOFO agent (which allows you to use Symantec's Volume Snapshot) is MUCH SLOWER than using Microsoft's Volume Snapshot itself. 

In my test on several file server...just plain file server, no SQL, no Sharepoint, no Exchange IS...NICs set to giga and on a giga-lan and giga switch... if you have Backup Exec AOFO agent install, the highest speed or data throughput on to an LTO-4 tape drive is averaging 800mb/min to 1500mb/min...

WITHOUT the Backup Exec AOFO agent and using only Microsoft's Volume Snapshot, that comes with all Windows 2003 servers anways, the speed or data throughput is much faster...almost TWICE as fast averaging 1200 mb/min to 2500mb/min... 

After seeing the results from test after test, I had removed all of Symantec's AOFO agent from all the file servers as they were not effective at all.

Message Edited by Kingsley Mok on 05-06-2008 10:57 AM

[tekzas]

Thanks for this information. We have a backup job running at weekends with about 1,2 TByte which takes about 24h (!) to complete.

After the change to Microsofot Volume Shadow Service it's going down to about 17h. But that's still very slow because the exchange mailbox backup runs with 100 mb/min.

We'll see if theres a solution for that also.

[Ken_Putnam]

that's still very slow because the exchange mailbox backup runs with 100 mb/min

if you are doing legacy mailbox backups, that is actually pretty good through put.  v11d introduced GRT backups - restore of individual items from a stores backup.  you may want to look into that instead