cancel
Showing results for 
Search instead for 
Did you mean: 

media multiplexing

raffl
Level 4

i am duplicating images from tape to disk and getting 20-30 GB/Hour. These are LTO 4 tapes and duplicating them to advanced disk.

If these tapes are multiplexed will it imact the duplication ?I see in tape report that they were multiplexed but doesn't show what multiplexing number was used.

Pls help me with my performnce.

I have 1 master and 4 media server. 1 server reading 4 drives.

number data buffers = 32

size data buffer = 262144

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

mph999
Level 6
Employee Accredited

01:55:10.752 [25808.14272] <2> read_data: waited for empty buffer 1608 times, delayed 17063 times

01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times

 
For a duplication, bptm does both the read and the write.

So we have:

Source side  data   --->  Buffers  ---->  Duplication side

The source side fills up the buffer, and was delayed from doing so 17063 times, each delay is I think 15 milli seconds, so in the dupration of the job, there was a (15 x 17063)/ 1000 second delay = about 255 seconds, where the read side process was doing nothing (well it was waiting for a buffer to become free).  Presuming the job length was 'kinda long' , 255 sec isn't worth bothering about (in fact, you usually will get some delays, even in the best tuned system), as it is a small % of the totakl job length.  that said, I don;t know what the job length was, so you have to make a judgement as to how significant the delay is.

The write side is the bigget problem:

01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times

Here bptm, waited 1395403 for data to be duplicated to be written to the buffer - so although the bptm read process was going through the tape, it was only finding relevant data 'every now and then', so the write process bptm had to wait until the buffer was full of data relevant to the backup being duplicated.

(15 x 1395403 )/ 1000 = 20931 sec = 348 minutes = 5.8 hours.

So, the mpx of the backup, may be contributing up to 5.8 hours of delays, in this specific job.

View solution in original post

30 REPLIES 30

sdo
Moderator
Moderator
Partner    VIP    Certified

That's not very fast, 20 GB/hr = 5.5 MB/s, and 30 GB/hr = 8.5 MB/s.

Times four active duplications thats 22.8 MB/s to 34.1 MB/s which looks suspiciously like USB2 disk.  Is your Adv.Disk storage unit on USB2 disk?

And, yes, previously multi-plexed tapes can be slower to read for single threaded duplication/restore simply because the tape can be more sparsely populated with the backup data you are seeking.  But, yes, I would expect better than 5.5 MB/s from an LTO4 tape drive... assuming that the data that you are reading is not extremely sparse.

Did you baseline the performance of the disk being written to?  i.e. do you already know that the disk that you are writing to can actually sustain, over long periods of time, more than 20/30 GB per hour of mixed sequential/random disk IO ?

Are all four duplication reads all writing to the same Adv. Disk storage unit? If so, then your actual disk write patterns will be effectively random.

Is the disk being written to being used for or by anything else... anywhere along the IO path?

Is the Adv.Disk storage unit on a share or UNC path?  Or is it actually real DAS or real SAN (iSCSI/FC) disk?

You may be able to tweak NUMBER_DATA_BUFFERS_RESTORE for tape reading... but I'm not 100% certain whether duplication jobs pay any attention to that tuning file.  Maybe someone else can confirm.

And maybe also tuning NUMBER_DATA_BUFFERS_DISK for the disk writing side.

See tech note:

Notes on number data buffers files

https://www.veritas.com/support/en_US/article.000076412

 

.

How many backups, tapes, GB do you have to duplicate?

Are you intending to ultimetly duplicate these backups back to newer tape?

Advanced disk is created over the unc path and it is DAS to a server and dedicated for duplication. All the duplication goes to this ADV disk at the same time.

There would be lot of images and yes I might duplicate them to LTO7 at later stage

Marianne
Level 6
Partner    VIP    Accredited Certified
As per @sdo - have you tested write speed to the disk outside of NBU? CIFS protocol has it's own overheads and then the other server's write speed to disk...

bptm log will show you where the hold up is in the process - look for 'waited for full / empty buffers'.

i tested this today and copied the data directly, it is 120 mb/sec over the n/w

Marianne
Level 6
Partner    VIP    Accredited Certified

Next step is to check bptm log.

sdo
Moderator
Moderator
Partner    VIP    Certified

Is the tape reader host also Windows, or is the NetBackup Server on Linux/Unix and thus perhaps using Samba client?

Was your network IO test from the same NetBackup Server to the same storage server (share/unc) using the same SMB protocol?  Or was 120MB/s achieved using SCP or FTP or NFS?

Marianne is right, you'll need to check bptm and bpdm logs.   Look for delay and wait counters, and read a few tech notes on how to visualize/theorize the actual numbers... sometimes the way to interpret the numbers is not obvious and you need to think of it in terms of source/producer and target/consumer because, if nothing odd or h/w related is found in the logs, then the bptm and bpdm delay and wait figures may well reveal where a potential bottleneck is.

e.g. the tape drive is the producer and bptm is  the consumer, then blocks get transfered in memory from tape buffers to disk buffers, then bpdm is producer and samba client is consumer... before then being punted across TCP via SMB blocks to a server that has to reconstitute NTFS files from the SMB blocks.

randomly collected this just now

00:02:03.438 [25808.14272] <2> read_data: stopping mpx read because 5027840 Kbytes of 5027840 were read
00:02:03.438 [25808.14272] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 27
00:02:03.438 [25808.14272] <2> read_data: waited for empty buffer 1514 times, delayed 15534 times

00:14:53.346 [3840.23296] <2> read_data: stopping mpx read because 3271168 Kbytes of 3271168 were read
00:14:53.346 [3840.23296] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 2
00:14:53.361 [3840.23296] <2> read_data: waited for empty buffer 98 times, delayed 1590 times

00:59:10.877 [24292.27868] <2> read_data: stopping mpx read because 103527424 Kbytes of 103527424 were read
00:59:11.033 [24292.27868] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 22
00:59:11.033 [24292.27868] <2> read_data: waited for empty buffer 91181 times, delayed 915614 times

01:11:04.323 [25808.14272] <2> read_data: stopping mpx read because 719872 Kbytes of 719872 were read
01:11:04.323 [25808.14272] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 27
01:11:04.338 [25808.14272] <2> read_data: waited for empty buffer 248 times, delayed 2946 times

yes everything is windows. the copy transfer is exactly same the way nbu is configured for the share. i have pasted the output of my bptm and it is some times with low waiting buffer sometimes more.

sdo
Moderator
Moderator
Partner    VIP    Certified

What I mean is... that for any IO... anywhere on any platform for any product for any OS for any software on any hardware... *always* involves something which is writing and something different which is reading.  Any IO is never about one number, e.g. my data transfer is only 50 MB/s is an unclear statement... it is clearer to think of it in terms of either the writer/producer or the reader/consumer is, for some reason, restricted to 50 MB/s... but there is a hidden facet... a nasty gotcha... in that both theproducer and consumer might be restricted (for different reasons) to 50MB/s... in which case no amount of tuning either side will change anything.

sdo
Moderator
Moderator
Partner    VIP    Certified

ok raffl - don't foget to look in the logs for errors and warnings too... look for any unusual time gaps between events/stages too... and so now begins your journey in to rationalising the wait and delay numbers... I mean, have a go yourself... and see if you can deduce what is going on!   good luck.

Marianne
Level 6
Partner    VIP    Accredited Certified

bptm log will also show buffer size and number being used for read and write. 

De-multiplexing will surely slow down the process as the tape constantly needs to reposition to find all the pieces for one image at  a time.

 

mph999
Level 6
Employee Accredited

It's worse than that ....  it doesn't reposition, it reads through everything on the tape ...

A mpx tape, say mpx of 4 looks something like this 

MH BH1 BH2 BH3 BH4 D1 D1 D3 D2 D4 D1 D1 D3 D4 D2 D2 D2 D2 D2 D3 D1 D1 D1 D3 D3 D3 D3 EH EOT

Where BHx is the backup header for client x  /  Dx is a data block from client x

So, all the Backup hears are grouped at the start, so we know which clients are on the tape, but all the data is mixed, as you know.

This means, If we want say a restore from client3, we have to read through all the previous data blocks for the other clients. Athough the data block contains a few bytes (*)  to identify the client it relates towe have no way to tell where in the mix it is, so have to read through everything.  

(*) Something along these lines, it's all a bit secret squirrel stuff ....

IMHO, anything above an MPX4 is madness if you want any real chance of a reasonable restore speed.

 

any logs which shows the mpx while duplicating the data

sdo
Moderator
Moderator
Partner    VIP    Certified

Apologies, I don't know.  Maybe with verbose level 5 logging then the logs might reveal the count or percentage of blocks retrieved (read and duplicated) versus the count of blocks passed over (i.e. of all of the other ignored spliced mpx blocks from other backup images)... but I doubt whether the logs will have this.

If you want to find out which backup images are spliced through which tapes with which other backup images then that meta-data can be pieced together from the fragment numbers in the image catalog.  But the mpx numbers will rise and fall as backup images (in the distant past) "started and joined mpx" and "finished and left mpx"... so, to me neither the question nor the answer really makes any sense.

And, whilst I haven't looked, I can say that I haven't ever noticed the bptm logs for duplications log facts like "mpx stream now has N spliced"... so I think the best that anyone can do is... hopefully you know the mpx level from when the tapes were written... and so divide the typical LTO4 speed by that, and then that figure is perhaps your best possible sustained average duplication speed.

Remember mpx was originally a feature to help reduce hadware costs and to reduce shoe-shining.  If previous admins have pushed mpx to the limits to avoid spending on hardware, then we are at the mercy of those prior decisions.  No-one would or could have ever described mpx as a feature to speed up restores or duplications... except in a few arcane configurations with very well defined requirements - but that's a different story.

Marianne
Level 6
Partner    VIP    Accredited Certified

Duplication logs will not show original MPX number.

All Log Entries and the bptm logs of the backup shows MPX value.
At the end of the backup, something like this is logged in bptm:

successfully wrote <number> of <number> multiplexed backups, total Kbytes <number> at <number> Kbytes/sec

These are very old backups so I don't have logs

so let's assume if the images are multiplexed which they are as I see in backup report, multiplex column shows yes. How much would it slow the duplication from tape to disk if we have the multiplex of 4 or 8. What kind of delay I would see in the bptm logs.


Can i change any buffer settings to run the duplication fast.

sdo
Moderator
Moderator
Partner    VIP    Certified

1) so let's assume if the images are multiplexed which they are as I see in backup report, multiplex column shows yes. How much would it slow the duplication from tape to disk if we have the multiplex of 4 or 8.

If original mpx-4 then duplication read could be 25% of LTO4.

If original mpx-8 then duplication read could be 12.5% of LTO4.

.

2) What kind of delay I would see in the bptm logs.

Where you might see a delay, is in bptm "waited for full buffer", i.e. bptm is not being fed fast enough from the tape drive.

.

3) Can i change any buffer settings to run the duplication fast

Posisbly increase NUMBER_DATA_BUFFERS_RESTORE (for duplication read) but I'm not 100% vouching for that.

Possibly increase NUMBER_DATA_BUFFERS_DISK for the writer - i.e. allow more buffers to be used for the 1-in-4 or 1-in-8 times that data is found by the reader... i.e. cache up as much as possible in the output buffers before  bpdm has to tell bptm "hey bptm, hold on a sec because my buffers are full (because I'm still waiting for SMB across the LAN to catch up)".

sdo
Moderator
Moderator
Partner    VIP    Certified

Cold comfort... but if you sit down and think about it... the feature of multi-plexing is actually very nice indeed... and extremely useful... and the more one thinks about it... the more one realizes how incredibly sophisticated the NetBackup code must be to enable features such as:

- backup stream join an already ongoing multi-plex set

- backup stream leave an ongoing multi-plex set

(...and my favourite mpx feature...)

- multi-plexed tape read... and later... multi-plexed duplication (yes, that is a single tape pass)... to multi-plexed tape write