Multiplexing-Performance settings

[mlacount]

I am looking for tuning or best practices info. regarding Multiplexing settings.
Looked through most .pdf's but not understanding the best way to use Multiplexing. These are the present settings in our environment for one policy
that contains 4 clients:

Storage Unit:

Multiplexing enabled
Maximum concurrent write drives: 12
Maximum streams per drive: 12

Policy:
Media Multiplexing:12

I have many policies pointing to the same storage unit. What is the best way to determine what multiplexing settings to use? Is there a suggested or documented number of clients you should have per policy?


Thanks,
Mike

[Chia_Tan_Beng]

> What is the best way to determine what
> multiplexing settings to use? Is there a suggested or
> documented number of clients you should have per
> policy?
>
Hi Mike,

There's no hard & fast rule on setting the multiplex level nor any document that I know of. General the guideline is base on how fast your backup client can move the data, how fast the drive can write the data and the implication of multiplexing (i.e. longer duration needed for restoration, & again it depends).

With the configuration of your storage unit & policy posted, you're allowing multiplexing of 12 streams into a media/drive at any time. To me, multiplex level 12 is huge.

IMHO, one can define your multiplex level of the storage unit base on drive throughput and how fast each client can move its data .

Example 1, if your drive throughput able to write at 40MB/s and maxmium per stream among the clients is 10MB/s, it is safe to set the storage unit with multiplex level of 4 without creating any bottleneck at drive level.

Example 2, with same environment as example 1, one may set the multiplex level at 8 for storage unit and backup policies for fast clients (runs at 10MB/s) are configured with media multiplex level of 4 while backup policies for slow clients (say, 5MB/s) are configured with media multiplex level of 8.

Above are just some general examples and there're more factors to be considered: restore duration, volume pool definition, backup windows, etc. But at least I hope it helps you to startup something.

[Dennis_Strom]

I agree with Chia 12 seems a bit large. Start with 2 or 3 and go from there.

[Rakesh_Khandelw]

Multiplexing = 12

Sounds really high unless you have very few number of drives and got too many clients. Even in that case you may end up creating bottleneck at drive level for backups.

Following explaination may help -

Multiplexing and multi-streaming
Consider the following factors regarding multiplexing and multi-streaming.
When to use multiplexing and multi-streaming

Multiple data streams can reduce the time for large backups. The reduction is achieved by
splitting the data to be backed up into multiple streams and then using multiplexing,
multiple drives, or a combination of the two for processing the streams concurrently. In
addition, configuring the backup so each physical device on the client is backed up by a
separate data stream that runs concurrently with streams from other devices can
significantly reduce backup times.

Note - For best performance, use only one data stream to back up each physical device on the client. Running multiple concurrent streams from a single physical device can adversely affect the time to back up that device because the drive heads must move back and forth between tracks containing the files for the respective streams.

Multiplexing is not recommended for database backups, when restore speed is of
paramount interest, or when your tape drives are slow.
Backing up across a network, unless the network bandwidth is very broad, can nullify the ability to stream. Typically, a single client can send enough data to saturate a single
100BaseT network connection. A gigabit network has the capacity to support network
streaming for some clients. Keep in mind that multiple streams use more of the client?s
resources than a single stream. We recommend testing to make sure that the client can
handle the multiple data streams and that the users are not affected by the high rate of
data transfer.

Multiplexing and multi-streaming can be powerful tools to ensure that all tape drives are
streaming. With NetBackup, both can be used at the same time. It is important to
distinguish between the two concepts:

◆ Multiplexing writes multiple data streams to a single tape drive.
◆ Multi-streaming writes multiple data streams, each to its own tape drive, unless
multiplexing is used.

Here are some things to consider with regard to multiplexing:
Experiment with different multiplexing factors to find one where the tape drive is just
streaming, that is, where the writes just fill the maximum bandwidth of your drive. This is the optimal multiplexing factor. For instance, if you determine that you can get
5 Megabytes/sec from each of multiple concurrent read streams, then you would use a
multiplexing factor of two to get the maximum throughput to a DLT7000 (that is,
10 Megabytes/sec).
◆ Use a higher multiplexing factor for incremental backups.
◆ Use a lower multiplexing factor for local backups.
◆ Expect the duplication of a multiplexed tape to take a longer period of time if it is
demultiplexed, because multiple read passes of the source tape must be made.
◆ When you duplicate a multiplexed backup, demultiplex it.
By demultiplexing the backups when they are duplicated, the time for recovery is
significantly reduced.
Do not use multi-streaming on single mount points. Multi-streaming takes advantage of
the ability to stream data from several devices at once. This permits backups to take
advantage of Read Ahead on a spindle or set of spindles in RAID environments.
Multi-streaming from a single mount point encourages head thrashing and may result in
degraded performance. Only conduct multistreamed backups against single mount points if they are mirrored (RAID 0). However, this also is likely to result in degraded
performance.

Effects of multiple data streams on backup/restore
◆ Multiplexing
To use multiplexing effectively, you must understand the implications of multiplexing
on restore times. Multiplexing may decrease overall backup time when you are
backing up large numbers of clients over slow networks, but it does so at the cost of
recovery time. Restores from multiplexed tapes must pass over all nonapplicable data.
This action increases restore times. When recovery is required, demultiplexing causes
delays in the restore process. This is because NetBackup must do more tape searching
to accomplish the restore.
Restores should be tested, before the need to do a restore arises, to determine the
impact of multiplexing on restore performance.
When you initially set up a new environment, keep the multiplexing factor low.
Typically, a multiplexing factor of four or less does not highly impact the speed of
restores, depending on the type of drive and the type of system. If the backups do not
finish within their assigned window, multiplexing can be increased to meet the
window. However, increasing the multiplexing factor provides diminishing returns as
the number of multiplexing clients increases. The optimum multiplexing factor is the
number of clients needed to keep the buffers full for a single tape drive.
Set the multiplexing factor to four and do not multistream. Run benchmarks in this
environment. Then, if needed, you can begin to change the values involved until both
the backup and restore window parameters are met.
◆ Multi-streaming
The NEW_STREAM directive is useful for fine-tuning streams so that no disk
subsystem is under-utilized or over-utilized.


Check Backup Planning and Performance Tuning Guide

Veritas NetBackup (tm) Enterprise Server / Server 6.0 Backup Planning and Performance Tuning Guide for UNIX, Windows, and Linux

http://support.veritas.com/docs/281842Message was edited by:
RK

[Chad_Wansing]

My personal approach to multiplexing is to not allow multiple streams from the client side, but up your multiplexing to 10 or 15 streams if you've got a gigabit connection to your master server and a decent infrastructure.  I haven't seen any degredation in my capacity to get data to the master server (each stream generally only runs 2-5 MB/s), and it allows me to get through my Windows hosts (which is most of our environment) very quickly.  Additionally, by not allowing multiple streams on the client side you don't risk bogging down that connection while also accomodating pulling from more hosts at a time to maximize the master servers connection.  I've gotten it up to 50%+ utilization on a 1gig connection at my master server.  The key is really to know where your bottleneck is going to be: client connection, master server connection, library connection or tape drive.  I'd suggest just getting a really good feel for your environment and playing around with how high you can up the multiplexing without getting noticible throughput issues.  If you have large chunks of servers that are all in the same policy, that greatly facilitates multiplexing as well.

Myone caveat is for restores.  If you have to do quite a bit of restoring, cut your multiplexing by 1/2 to 2/3 (my own personal general rule) to lessen the number of tapes you have to track down every time you have to do a restore.  It's all a balancing act.

[Stumpr2]

A lot of good stuff in this post!

don't forget your memory :)  ...sorry, but the pun was intended

Use this formula to calculate the amount of shared memory required by NetBackup:

(number_data_buffers * size_data_buffers) * number_tape_drives * max_multiplexing_setting

For example, assume that the number of shared data buffers is 64, the size of the shared
data buffers is 1048 Kilobytes, there are 14 tape drives, and the maximum multiplexing
setting is 16. Following the formula above, the amount of shared memory required by
NetBackup is:

64 * 1048576 * 14 * 16 = 15032385536

[Chuck_Stevens]

Dumb question Bob - are you referring to a Media server, or the Master server? (assuming an Enterprise installation where the Media servers do the writing to tape)

[Rakesh_Khandelw]

Formula is good for both, master and media servers.

[Stumpr2]

Chuck,

That's a good question.

The NetBackup media server uses shared memory to buffer data between the network and the tape or disk drive (or between the disk and the tape drive if the NetBackup media server and client are the same system). The number and size of these shared data buffers can be configured on the NetBackup media server.

The size and number of the tape and disk buffers may be changed so that NetBackup optimizes its use of shared memory. Changing the default buffer size may result in better throughput for high-performance tape drives. These changes may also improve throughput for other types of drives.

Buffer settings are for media servers only and should not be used on a pure master server or client.

PLEASE: Before going on this adventure, read the "Planning and Performance Tuning Guide" as it tells all....also NEVER be satisfied with a backup UNTIL you are able to do a successful restore :)

After making changes, it is vitally important to verify that the following tests complete successfully:

1. Run a backup.

2. Restore the data from the backup.

3. Restore data from a backup created prior to the changes to SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS.

4. Sleep restfully at night and enjoy your time off.

Message Edited by Bob Stump on 03-27-200706:13 AM