How tuning the buffer size led to much-improved performance

Sometimes a little bit of tuning takes you a long way.

We recently had a discussion with our sales personnel working with a vendor to have Backup Exec pre-loaded into hardware boxes that they were selling. They were testing with 2 SSD disks for the OS (RAID 1) and 5 HDD 7200 rpm in RAID 5 for the backup to disk storage (they were not using deduplication). However, they were not satisfied with the backup rates they observed and were sure Backup Exec and the hardware could achieve better rates.

Backup speeds depend on various factors. You can think of it as a pipeline with read-transfer-write stages. The bottleneck could be any one of these stages and the net speed will be determined by the slowest among them. Backup Exec uses several mechanisms to ensure high speeds. Examples of this include buffering, asynchronous I/O operation, judicious use of application or system cache, use of single or multiple threads as applicable, and so on.

In the end, however, the performance on a specific system, depends on its hardware and its configuration. In this case, the source of the backup was just a simple file system test data. So naturally, we looked towards the destination side of things.

While one could go deep into the analysis, one of the first things we did, was to adjust the buffer sizes. By opening the Properties page for the backup to disk folder, one can adjust the block size and the buffer size. By default, Backup Exec’s setting is to determine the parameters automatically. That mechanism is somewhat conservative and plays safe to leave spare resource capacity for other possible workloads, including the OS itself. This is to be able to have a reasonably good performance on a wide variety of hardware.

In particular cases, however, it is useful to experiment with these settings. In this case, we found that a buffer size of 64 KB was being used. Given that this was being tested on hardware specifically dedicated to backup servers with sufficient RAM and disk for a typical small and medium business, we recommended testing with a larger buffer size of 1 MB.

BufferSize.png

Within a couple of days, we heard that the speed had more than doubled! The partners in question were entirely satisfied with the performance.

An increase in the buffer size helps because each I/O is made with a larger number of bytes. So the number of I/O operations and associated per-operation overheads are reduced.  When RAID 5 is used for the backup to disk folder like in this case, a rule of thumb would be to set the buffer size using the formula “Stripe Unit * (Number of Disks – 1)”. This allows each I/O to be parallel written out to the RAID-5 configuration in one go so that it occupies exactly one stripe set.

We thought we share this little story with you, in case, some experimentation and tuning at your end, help improve your experience. Note that any change can have unexpected effects as well. For example, I/O may be faster, but this could lead to higher RAM usage due to larger buffer sizes, which may not go well in your particular case. That is why it is important to make controlled changes and measure things over time.

Tags (2)