Serious performance problem with Win2003 64 bit, but Win2008 was ok.

[mbuyukkarakas]

Hi everybody.

In my company I'm using BE 12.5 SP3 to backup almost entire IT systems (approx 120 server in 10 different platform). My hardware is ;
- a BE server Sun X4250 (8xCPU (64bit) 6 GB RAM, 320 GB RAID 5 Hard Disk, two teamed gigabit NICs.
- a JBOD 4400 attached to BE server with total 20 TB disk space.
- a Storagetek SL500 Tape library with 4 tapes.
- Most of the servers are running on Vmware Vsphere and I prefere to use agents to take backups.

First time we installed BE we were using Windows 2008 Std. The performance wasnt ok (but not deadly slow as Win2003) and I had serious problems to backup my Exchange2007 servers. After installing some hotfixes I couldt have a affirmative progression about Ex2007 problem so I decided to reinstall everthing with Win2003 64bit.

I installed the OS (Win2003 64bit Std) with Sun's SIA, then the SQL server 2005 SP2 and copied the "data" and "catalog" folders to the original path as described in the migration article. Then I updated the server with all the Microsoft patchs.

I had a brand new server But this time the performance was terrible. Now the backup job rate is 10 times slower then last time. It's hard to believe that.

Let me give you some examples.

- The full backup of my Exchange 2007 server (is a virtual server and the data to be backup is 500 GB)

• 7 hour in Win2008 standart
• 34 hour in Win2003 64bit standart

- The full backup of my SQL server (is a physical server and the data to be backup is 180 GB)

• 2 hour in Win2008
• 21 hours in Win2003

It seems something is wrong in this environment. I will be very happy if somebody has any experience with a configuration similar to mine.

You guys will probably suggest me to go back to my Win2008 configuration. But is it possible to get a very bad performance with a configuration like this ? I wonder to know what is my mistake ?

Thank you in advance.

Regards.
Mehmet

[teiva-boy]

 Typically Win21k8 is faster due to the driver implementation in it.  But you shouldnt see that level of slowdown?

That said, where are the slow downs specifically?  To disk, to tape?  
Dont forget in Win2k3 when configuring disk volumes you NEED to use the align command in diskpart to configure your disks right the first time.
The Disk manager does it wrong.  This is not an issue in Win2k8.  I suggest align=64 in most cases.

The only other thing perhaps is to make sure the latest chipset drivers from Intel are installed. This will ensure that all the PCI add-in cards are using the max bandwidth possible.  The Default Windows installation uses generic drivers.  Intel will usually provide drivers for the north bridge and south bridge to use the hardware more efficiently.  

[mbuyukkarakas]

Thank you for your reply.

I used latest drivers coming with Sun SIA, even for updating the BIOS. But generally Symantec doesnt recommend to update tape drivers and I followed this. But I'm not sure about changer. Should I update it ?

Honestly I dont think it's a tape driver specific problem. As you can see at screenshot even the disk backups are very slow.

Is choosing Win2K3 a bad decision ?

[teiva-boy]

 Personally, I've never been a fan of the Symantec drivers...  You can try either.  My take and opinion is those drivers are made for compatibility, not speed.  But either works, you just need to test to see which is better.  Just remove the devices, do a add hardware wizard, and point to the new driver location, or update the drivers, both drives and changer.  For ADIC, Overland, SpectraLogic, and Sony, I've never had an issue using their drivers.

Win2k8 allows you to use vendor supplied drivers to access the new User Mode Driver features in win2k8.  That means if the driver crashes, the driver crashes, not the entire OS.  :)  No more BSOD's.

Network card drivers, I've also always chose to go straight to broadcom or Intel direct.  HP, I just get HP.  For Sun, I'm not sure if they do anything special like HP does.

That said, I'm thinking that playing with the drivers *could* help, but do it in stages, and see if you can isolate the issue.  Start with the network card, then do the disk alignment if sticking with Win2k3.  Then move on to the SAS/SCSI cards, HBA's, etc.

However, the new Dedupe functions in BE2010 coming out this week, and with Exchange 2010, requiring Win2k8...  You should just go back to Win2k8 64bit and troubleshoot from there.

[mbuyukkarakas]

 When I removed the teaming on NICs the server is quite faster than last time.

But I know it's not enough, it can be faster. I still need your opinions and advices.

For example what do you think about this ?
https://www-secure.symantec.com/connect/forums/optimising-backup-exec-125-performance

[teiva-boy]

 So it sounds like a network issue with teaming.  Either your drivers are at issue, or your switches are not configured properly to support that.  If you break teaming, and the job goes faster, well somethings wrong eh?

The link you posted has some great info, you just need to test and verify it will make a difference for your environment.  Tuning is an art, and you need to change the settings one at a time, and test each to see where you gain and lose performance.  You may find that it works great for some large contiguous files, but chokes smaller files.  

Sorry that there is no definitive answer to your issue, just a lot of testing and re-trying...