Solved
Backup Throughput HALVED Mysteriously....
Hello everyone. I'm hoping to get some resolution on this issue. Here's the situation:
Backup Exec 12 Installed on EFSSVR11 in April. Throughput speeds were 1600-2000 MB/min backing up to an LTO3 external tape drive connected via SCSI.
May 16th we were hit by a worm targetting Symantec Products, specifically Symantec Corporate Antivirus 10. This resulted in only one casualty, the server with BE12 installed. The OS was hosed. Reinstalled the OS and installed remote agent. Installed BE12 on another server, then restored the entire OS via the remote agent to EFSSVR11. Worked fine. Uninstalled Remote Agent, reinstalled BE12. Finally got it installed and patched, patch the OS to latest patches (Including SP2).
Backup Exec had the original jobs and configuration settings still, so I let the job run. It dropped to 770 MB/min.
I have done EVERYTHING I can think of to fix this, including disabling the retarded networking pack included with SP2, even though this is a local backup. I have even installed a brand new SCSI 160 card into a brand new server and installed BE12 onto that, and still have not seen any change in the software's backup speed.
Symantec Tech support has confirmed that our throughput is 1140 GB/S. Yes, that's GBPS, not GBPM, we have overhead enabled on the dual Gig network cards (Intel by the way).
I have ruled out any networking related issue. The backup is as slow remotely as it is locally.
Now for some interesting facts:
Backing up .pst files back up at over 2400 MB/Min. We backed up 3 gig files to local and to tape, same spped.
The file server has three disks in it, seperated onto four logical partitions. Disk 0 has C and D. C backs up at 350 MB/Min, D 1200 MB/min. Disk 1 has F (3% fragmentation) and backs up at 650 MB/min. Disk 2 has G (1% fragmentation) and backs up at 650 MB/min.
Backing up just the test folder (5 Gig) from G gives me 1800 MB/min.
Symantec Tech support has promised call backs and to look into it but as can be expected from this wonderful customer service age, I have heard nothing back, and will likely devote another two hours to hold and one to be told that they'll call me back after reviewing tests tomorrow. So I'm hoping that somebody here will have some experience with this situation. The only thing that changed was the OS was restored from tape, the OS was patched to SP2. That's all. Nothing more. The myriad changes since have done nothing to improve or harm the backup speed.
I have tried running with AOFO, no AOFO, with Symantec's AOFO (never again!)
, BE bare bones, BE fully patched, x64 bit, x86 bit, TOE, no TOE, nothing works. And downgrading the OS to SP1 is not an option, due to security requirements within the company.
Anyone at all have any ideas?
The tech even admits that the throughput of the disks is possible at 2400+ MB/min.
What's unusual is everything is half of what it used to be before that worm and restore. Sections that backed up at 2400 now back up at 1200, sections that backed up at 1400 now backup at 700.
The job used to complete in 3 hours and now takes anywhere from 6 to 9, which really hurts our backup and restore strategy. Hopefully somebody can help me troubleshoot this issue. I will not try to explain to our controller why a brand new LTO 3 drive is backing up slower than our old LTO 2, and I will not try to justify buying another LTO 3 driver like tech support suggested "Splitting the job to multiple tape drives."
This worked before. No hardware changed. Only one piece of software should have changed, if the restore works as advertised. Transfering files directly between servers takes mere seconds (I can transfer 3 gigs in seconds).
I appreciate any feedback that can be offered. If I have left anything out, please make note of it and I'll fill in the blanks. Thank you!
Chris Crabtree
Systems/Network Administrator
Backup Exec 12 Installed on EFSSVR11 in April. Throughput speeds were 1600-2000 MB/min backing up to an LTO3 external tape drive connected via SCSI.
May 16th we were hit by a worm targetting Symantec Products, specifically Symantec Corporate Antivirus 10. This resulted in only one casualty, the server with BE12 installed. The OS was hosed. Reinstalled the OS and installed remote agent. Installed BE12 on another server, then restored the entire OS via the remote agent to EFSSVR11. Worked fine. Uninstalled Remote Agent, reinstalled BE12. Finally got it installed and patched, patch the OS to latest patches (Including SP2).
Backup Exec had the original jobs and configuration settings still, so I let the job run. It dropped to 770 MB/min.
I have done EVERYTHING I can think of to fix this, including disabling the retarded networking pack included with SP2, even though this is a local backup. I have even installed a brand new SCSI 160 card into a brand new server and installed BE12 onto that, and still have not seen any change in the software's backup speed.
Symantec Tech support has confirmed that our throughput is 1140 GB/S. Yes, that's GBPS, not GBPM, we have overhead enabled on the dual Gig network cards (Intel by the way).
I have ruled out any networking related issue. The backup is as slow remotely as it is locally.
Now for some interesting facts:
Backing up .pst files back up at over 2400 MB/Min. We backed up 3 gig files to local and to tape, same spped.
The file server has three disks in it, seperated onto four logical partitions. Disk 0 has C and D. C backs up at 350 MB/Min, D 1200 MB/min. Disk 1 has F (3% fragmentation) and backs up at 650 MB/min. Disk 2 has G (1% fragmentation) and backs up at 650 MB/min.
Backing up just the test folder (5 Gig) from G gives me 1800 MB/min.
Symantec Tech support has promised call backs and to look into it but as can be expected from this wonderful customer service age, I have heard nothing back, and will likely devote another two hours to hold and one to be told that they'll call me back after reviewing tests tomorrow. So I'm hoping that somebody here will have some experience with this situation. The only thing that changed was the OS was restored from tape, the OS was patched to SP2. That's all. Nothing more. The myriad changes since have done nothing to improve or harm the backup speed.
I have tried running with AOFO, no AOFO, with Symantec's AOFO (never again!)
, BE bare bones, BE fully patched, x64 bit, x86 bit, TOE, no TOE, nothing works. And downgrading the OS to SP1 is not an option, due to security requirements within the company.
Anyone at all have any ideas?
The tech even admits that the throughput of the disks is possible at 2400+ MB/min.
What's unusual is everything is half of what it used to be before that worm and restore. Sections that backed up at 2400 now back up at 1200, sections that backed up at 1400 now backup at 700.
The job used to complete in 3 hours and now takes anywhere from 6 to 9, which really hurts our backup and restore strategy. Hopefully somebody can help me troubleshoot this issue. I will not try to explain to our controller why a brand new LTO 3 drive is backing up slower than our old LTO 2, and I will not try to justify buying another LTO 3 driver like tech support suggested "Splitting the job to multiple tape drives."
This worked before. No hardware changed. Only one piece of software should have changed, if the restore works as advertised. Transfering files directly between servers takes mere seconds (I can transfer 3 gigs in seconds).
I appreciate any feedback that can be offered. If I have left anything out, please make note of it and I'll fill in the blanks. Thank you!
Chris Crabtree
Systems/Network Administrator
Finally fixed it. The problem was twofold.
Firstly, if you don't have the files you're backing up as a member of your "local Intranet" in windows explorer under the account you're running backup exec as, it will run extremely slowly. Secondly, the AOFO option, after adding the files to the Local Intranet zone, was slowing things down.
Previously, adding or removing AOFO made no difference, but once we added the Local Intranet entry, it blazes, at 1600 Mb/min.
Settings I ended up tweaking:
NTFS Single Instance: Yes
AOFO: None
Advanced Open File option when AOFO is not used: Without a Lock
Restoring the default buffer settings to our tape drive and applying these settings, along with one last critical thing, fixed it.
Final thing? Job priority: Medium.
Running the job priority: Highest seemed to slow it down by 200 MB/Min or more.
I hope this helps somebody, since Symantec obviously has no clue!
