How much data are you backing up during that time?
Don't know if it would be relevant to your situation, but the one time I saw similar results to this was when we were backing up onto LTO1 tapes, over a Gigabit network. On a couple of occasions the NIC on the backup server decided to drop down to 100Mb, which due to the tape backup speed suddenly introduced a major bottleneck in the backup, increasing the time required for the backup.
In the end I think it turned out that either the NIC or the network cable was faulty, and once replaced we never saw the problem again.
