Netbackup multiplexing questions

[posie80]

Hi all, Just would like a confirmation of my understanding of multiplexing : I have an environment where we have plenty of backups,(exchange databases, local drives, system state backups) short window to do them all and not a lot of drives to go around. We are using netbackup 7.1 on windows for all master/media/clients, and we back up directly to lto5 tape drives. I am not seeing good performance on these lto5s and the most i could get out of them is about 40mb per sec. This is without any tuning and no multiplexing involved. I would like to use multiplexing to maximize the potential of these lto5s. My understanding is the following : 1) As an example, i have 4 non mutiplex policies being run at the same time and therefore using 4 drives. If i set the media multiplexing of these 4 policies to 4, then i assume that only one drive will be used when these 4 policies are run at the same time (instead of 4) thus allowing me to have 3 free drives for other backups. Is this correct? 2) Using the same scenario as above, during the simultaneous run of 4 non multiplex policies, i am seeing a thoroughput of 10 mb/s average for each drive. If i multiplex these policies so that only one drive is being used for 4 jobs, then i am still seeing more or less about 10 mb/s for each multiplex job in the activity monitor. However since these are multiplex sessions, is it correct to assume that, after multiplexing, i now have a thoroughput of 10 mb/ sec x 4 = 40 mb/ sec for this one drive? ( instead of seeing about 10 mb/sec per each drive, before multiplexing) My aim at the end of the day, is to free up some drives and to have better t horoughout overall, however i am not sure if my assumptions above afe correct. Kindly corect me if i'm wrong or give your comments! I have yet to do restore tests yet.. I hope multiplex fctor of 4 isnt high enough as to impact my restore performance much.. Thanks in advance for your kind comments! Posie

[Marianne]

1)  Yes - provided STU is also configured with MPX of 4 and all policies use the same volume pool and same retention level. If multiple streams need to be generated per client, check Max Jobs per Client in Master's Global attributes.

2) In theory, yes.

We have always found that MPX = 4 to be a good value for backup and restore performance.

[mph999]

Adding to the excellent posts above from Marianne and Nicolai.

Be aware that you wil need more memory when turning mpx on ...  like this ...

The total amount of shared memory that is used for each tape drive is:

(number_data_buffers * size_data_buffers) * number_tape_drives * max_multiplexing_setting

For two tape drives, each with a multiplexing setting of 4 and with 16 buffers of

256KB, the total shared memory usage would be:  (16 * 262144) * 2 * 4 = 32768 KB (32 MB)

So from this we can see that suddenly increasing MPX to x16 across one of more drives causes a massive increase in the amount of memory required.

Additionally ...

From my bptm log, we see I have x12 data buffers, each with a size of 131072 ...

08:55:45.665 [18200] <2> io_init: using 131072 data buffer size

08:55:45.665 [18200] <2> io_init: CINDEX 0, sched Kbytes for monitoring = 60000

08:55:45.665 [18200] <2> io_init: using 12 data buffers

... therefore, each tape drive, or each stream to a tape drive will require 131072 x 12 = 1572864

Now this example is actually from a MPX backup with 2 streams, so you might think that the amount of shared memory will be 1572864 x2 = 3145728 

No, here is the catch ...

Looking down my bptm log I find these lines ...

08:55:45.748 [18200] <2> mpx_setup_shm: buf control for CINDEX 0 is ffffffff79a00000

08:55:45.748 [18200] <2> mpx_setup_shm: shared memory address for group 0 is ffffffff76800000, size is 6291456, shmid is 117440636

08:55:45.748 [18200] <2> mpx_setup_shm: shared memory address for CINDEX 0 is ffffffff76800000, group 0, num_active 1

So we see the amount of memory is 117440636

Now, 6291456/ 1572864 = 4

So, what has happened, is even though I have one tape drive, the amount of memory NBU will allocate is the amount of memory required by x4 tape drives.  Likewise, if I had say mpx setting of 8 and ran 5 streams, it would use enough memory as if 8 streams were running, so it always rounds up to the nearest 4.

This is not a fault, it actually is designed for reasons of efficiency, one of the BL guys sis explain the exact details, but for the moment they escape me.

Hope this helps.

Martin 

[Mark_Solutions]

Adding to the earlier excellent posts a couple of thinsg to note and a couple from my experience

1. To have multiplexing to work it must be set at the Storage unit, Each Schedule and, if it is all from one client, the maximum jobs per client must also be high enough (Master Server Host Properties) - which ever of these setting is the least will be what multiplexing you get

2. I have found that a MPX value of 6 works well - fast backups and with LTO5 little affect on restores, although i would reccomend a fragement size of 5000MB on the storage units

3. I have also found that 32 as a value for NUMBER_DATA_BUFFERS works well across most Windows Systems and for LTO4 and LTO5 a SIZE_DATA_BUFFERS of 262144 is excellent.

4. The memory used by these stream passes through the Paging Memory so to get the best out of the systems set PagedPoolSize to a hex value of FFFFFFFF and PoolUsageMaximum to a decimal value of 40 (both DWORDS under HKLM\System\CurrentControlSet\Control\SessionManager\MemoryManagement\ - add them if the are not there already - reboot required to take effect)

5. When doing a lot of streams you are also increasing the TCPIP load on the Media Server so add the TcpTimedWaitDelay DWORD with a value of 30 under HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\ - create it if it doesnt exist. On Windows 2003 also add MaxUserPort DWORD with a value of 65534. On Windows 2008 run, from a command line:

netsh int ipv4  set dynamicport tcp start=10000 num=50000

Reboots required for the registry keys

Hope these help

Hope this helps

[Nicolai · Accepted solution]

You must perform buffers tuning first. It's the number one teak for Netbackup. See http://www.symantec.com/docs/DOC4483 page 122 and forward.

You're observations are right (high level). MPX is configured on the storage unit level not policy level. A MPX setting on a policy are used to further limit the setting from the Storage Unit. This way you can say use 5 streams, but if the streams are from the VIP oracle database limit to  2.

But there is more to it. Doing MPX gives a better tape drive usage, but at the cost of restore time. Data written in a MPX stream is interleaved, so the tape drive need to read more data (both the data for restore but also data in the same mpx stream)  compared to a non-mpx restore. Make sure you as well configure a fragment size, do not use infinity. A size of 50GB should do the job.

See technote: How "Maximum Fragment Size" for a Storage Unit affects backups and restores

http://www.symantec.com/docs/TECH15692