Extremely Slow Backups

[TedW]

We recently suffered a large-scale power outage within our building, which caused an entire shutdown of our datacenter. Our backup server was shut down gracefully before our UPS ran out of power. When we booted everything back up, our backups went from 800-1200MB/s speeds, down to 8-50MB/s speeds. I have ran consistancy checks on our SAN volumes, local disks, and also ran file transfer tests and network throughput tests with all the servers that we backup, the SAN, and the BackupExec server itself. Everything came back as 100%. I have patched our BackupExec to current levels, yet still the problem exists.

What can I do to figure out the problem and return our backups to functional levels rather than this absurdity that we have now? Thank you.

Ted

[Ken_Putnam]

For networked backups with speeds that low, the first thing I'd check is to verify that AutoNegotiate  is NOT enabled on any switch ports/NICs  between your medai server and the target servers.  Hardset each end of each pipe to the hightest speed that both ends support

[TedW]

Do you have any other ideas? The settings have not changed, and have always been set to Auto/Auto (Speed/Duplex) and the backup speed has been acceptable. The 2 servers in question are a production file server and Exchange server, so I can't very well drop their network to change settings right now.  Is there any other diagnostic that might show me something? Network throughput caps at 980MB/s using iperf. Windows file copy works at expected speeds. Disks and SAN volumes check out clean. Yet BackupExec is unable to backup at a respectable speed.

Thanks.

Ted

[Ken_Putnam]

Are local backups still acceptable?  or are they slow also?

[TedW]

I can't duplicate it exactly, but I did try a local backup of the BE server, and was averaging 1600MB/s, so obviously acceptable. I also tried a backup of a SAN volume mapped to the BE server to another SAN volume mapped to the BE server, and got 500-600MB?s, so not perfect...But backup of the local BE server was sent to a SAN volume. Hopefully this makes some sense...been a busy week.

[TedW]

I also tried a network backup to a local drive (to eliminate SAN connectivity problems), backup ran at slooow speeds. So it seems to be a network issue. Nothing has changed that I know of. Culprits at the moment can be remote agent, networking infrastructure, or some wierd problem with remote backups.

I did do iperf network tests using each target server as "Server" and BE as "client" and the other way around. I hit 920-980MB/s throughput. Which is expected. So not sure why BE can't reach it's normal speeds.

[thespigroup_the]

This is something I've been struggling with too since upgrading to 11d We used to get pretty respectable transfer speeds over gigabit and have been getting 100Mb/s type speeds (slightly slower, slightly faster.) I've tried absolutely everything I can think of from hardwiring the port settings at 1000 Full, jumbo frames on and off with larger frame sizes (9014). netcps posts 40-50MB/s, FTP transfers are in that range too so there's nothing wrong with the switch, network cards, servers.

Some transfers start off in the 1300-1800MB/min range and then rapidly drop off to 100Mb/s range.

:(

[TedW]

We had this before the power outage. It'd start at extremely fast backup and drop down to 600-1000MB/s. Not great, but was something we could live with. Now with nothing changed, no problems with devices that came up in all our testing, and throughput tests saying everything is ok...we're at 80-250MB/s. Very very frustrating. Might cause us to finally go another route as we have too much data to spend 3 days doing a full backup. Users aren't smart enough to live without restoration capabilities.

[Ken_Putnam]

Hmm

take a look towards the bottom  of

https://forums.symantec.com/syment/board/message?board.id=101&message.id=47088&query.id=97515#M47088

Netsh int ip set chimney DISABLE

[TedW]

That's only if you've installed the 2003 Scalable Network Pack. We do not have that installed. Does 11d require the scalable network pack installed?

[TedW]

I only see a blank post. Not sure if it's somethin with my end or not.

Ted

[Ken_Putnam]

Nah, we seem to be having proixy problems today

Anyway -

No that does not need to be enabled/installed

I flocal backups are acceptable, and if all your SwitchPorts\NICs are hard set, and if a large file copy seems acceptable, I'm out of suggestions