32% successful backup rate to network storage server

[Jon_Sumida]

So in my environment I have a a mix of backup destinations that include a network storage server and external USB drives that are attached directly to the PCs.  Running BESR 2010 still and prior to this week was getting OK success rate in backups. 100% success rate to the external USB drives, but I would get a handful every day that would fail for some reason or not to the network storage. Starting sometime this week I noticed a huge % of my PCs would not successfully backup to the network storage. I have a 102 total PCs set up to backup this way and of those only 33 have successfully backed up after last night. The others are all in a state of "backup running" and never finish. They are stuck in all varied %'s of completion, not just the old 95% completion problem of the past. I have 15 folders created for different departments and each represents a policy that runs at different times of the night. There is some crossover in times, but I'm trying to keep it from being all PCs in one policy running at once as this seems to have caused me a lot of issues in the past (RPAM_Store.dat file issue).

Nothing has changed on the network storage server recently outside of installing the latest Windows udpates (running Windows Storage Server 2003). Could it be the Windows updates? I am really getting tired of BESR's 2010 issues but I don't have the time to migrate over to SSR 2011 yet, sigh.

[Jon_Sumida]

Update: I noticed the responsiveness of the storage server was very sluggish. Not sure if that was a result of all the hanging backup jobs being written to it or not. So I restarted the server and now it's responding a lot faster. I ran a single test backup of a PC and it went smoothly. The test will be tonight when all my jobs are set to run again. Crossing fingers.

[Jon_Sumida]

Still more than 50% failure rate. ugh....

[Markus_Koestler]

Have you tried to set the performance registry keys:

http://www.symantec.com/business/support/index?page=content&id=TECH125631&key=53847

[Jon_Sumida]

That's a good article, thanks Markus. I'll try some of the things suggested in there. It still is odd that this large amount of failures/hung jobs all started this week.

Last night was not any more successful, even after uninstalling the Microsoft patches that I thought might be the cause of the problems. Looks like I'm in for a long testing period to see what tweaks gives me the best performance.

[Markus_Koestler]

Hm, one last idea: Try to split the backup file in 640MB pieces.

[Jon_Sumida]

Yeah, I'm going to put that as one of the things to test

[Jon_Sumida]

Ok, I did some testing over the weekend and it looks like what is happening is that when I run too many jobs and have backups overlapping, a lot of them hang and don't ever finish. I basically start small groups of backups at 8PM and go all night until 4AM.  My test this weekend was to run the 8PM and 9PM groups ( around 20 PCs total) and then the 3AM group (8 PCs which have never finished, always get hung up).  All PCs in this testing group successfully completed. So it looks like the jobs I have scheduled after the 9PM groups eventually tie up the server or something and that's why they never finish. Now I have to figure out what combination of things will make them all work. Odd that only recently they started to really experience this type of hanging.

[Markus_Koestler]

So this is (more or less) solved ?

[Jon_Sumida]

Sorry for the delayed response Markus. I've been battling other "fires" at work and then took a much needed vacation. I thought things were going well and then last night pretty much every backup didn't complete again. It's definitely coming down to the number of backups that are getting written to the backup server at once. I just need to find the magic number of clients. My problem is there's only so many hours in the night and to stagger all of these so they don't overlap each other basically pushes me into the morning when users come back. I might have to go to alternate days for backups for different groups.

I have yet to implement those performance registry keys. Will do more testing this week and update. Thanks

[Richard_FDisk]

Random Distribution: will randomize the backup job times based on the time interval you specify

Enable Network Throttling: slows down the data transfer so the server isn't overloaded with all the simultaneous data pouring in

see this thread for a similar problem:

https://www-secure.symantec.com/connect/forums/there-limit-or-threshold-point-using-network-storage-...

[Jon_Sumida]

haha, that's my thread too :)