Forum Discussion

raffl's avatar
raffl
Level 4
7 years ago

media multiplexing

i am duplicating images from tape to disk and getting 20-30 GB/Hour. These are LTO 4 tapes and duplicating them to advanced disk. If these tapes are multiplexed will it imact the duplication ?I see ...
  • mph999's avatar
    mph999
    7 years ago

    01:55:10.752 [25808.14272] <2> read_data: waited for empty buffer 1608 times, delayed 17063 times

    01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times

     
    For a duplication, bptm does both the read and the write.

    So we have:

    Source side  data   --->  Buffers  ---->  Duplication side

    The source side fills up the buffer, and was delayed from doing so 17063 times, each delay is I think 15 milli seconds, so in the dupration of the job, there was a (15 x 17063)/ 1000 second delay = about 255 seconds, where the read side process was doing nothing (well it was waiting for a buffer to become free).  Presuming the job length was 'kinda long' , 255 sec isn't worth bothering about (in fact, you usually will get some delays, even in the best tuned system), as it is a small % of the totakl job length.  that said, I don;t know what the job length was, so you have to make a judgement as to how significant the delay is.

    The write side is the bigget problem:

    01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times

    Here bptm, waited 1395403 for data to be duplicated to be written to the buffer - so although the bptm read process was going through the tape, it was only finding relevant data 'every now and then', so the write process bptm had to wait until the buffer was full of data relevant to the backup being duplicated.

    (15 x 1395403 )/ 1000 = 20931 sec = 348 minutes = 5.8 hours.

    So, the mpx of the backup, may be contributing up to 5.8 hours of delays, in this specific job.

mph999's avatar
mph999
Level 6

It's worse than that ....  it doesn't reposition, it reads through everything on the tape ...

A mpx tape, say mpx of 4 looks something like this 

MH BH1 BH2 BH3 BH4 D1 D1 D3 D2 D4 D1 D1 D3 D4 D2 D2 D2 D2 D2 D3 D1 D1 D1 D3 D3 D3 D3 EH EOT

Where BHx is the backup header for client x  /  Dx is a data block from client x

So, all the Backup hears are grouped at the start, so we know which clients are on the tape, but all the data is mixed, as you know.

This means, If we want say a restore from client3, we have to read through all the previous data blocks for the other clients. Athough the data block contains a few bytes (*)  to identify the client it relates towe have no way to tell where in the mix it is, so have to read through everything.  

(*) Something along these lines, it's all a bit secret squirrel stuff ....

IMHO, anything above an MPX4 is madness if you want any real chance of a reasonable restore speed.

 

raffl's avatar
raffl
Level 4
7 years ago

any logs which shows the mpx while duplicating the data

  • mph999's avatar
    mph999
    Level 6
    7 years ago

    01:55:10.752 [25808.14272] <2> read_data: waited for empty buffer 1608 times, delayed 17063 times

    01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times

     
    For a duplication, bptm does both the read and the write.

    So we have:

    Source side  data   --->  Buffers  ---->  Duplication side

    The source side fills up the buffer, and was delayed from doing so 17063 times, each delay is I think 15 milli seconds, so in the dupration of the job, there was a (15 x 17063)/ 1000 second delay = about 255 seconds, where the read side process was doing nothing (well it was waiting for a buffer to become free).  Presuming the job length was 'kinda long' , 255 sec isn't worth bothering about (in fact, you usually will get some delays, even in the best tuned system), as it is a small % of the totakl job length.  that said, I don;t know what the job length was, so you have to make a judgement as to how significant the delay is.

    The write side is the bigget problem:

    01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times

    Here bptm, waited 1395403 for data to be duplicated to be written to the buffer - so although the bptm read process was going through the tape, it was only finding relevant data 'every now and then', so the write process bptm had to wait until the buffer was full of data relevant to the backup being duplicated.

    (15 x 1395403 )/ 1000 = 20931 sec = 348 minutes = 5.8 hours.

    So, the mpx of the backup, may be contributing up to 5.8 hours of delays, in this specific job.

  • mph999's avatar
    mph999
    Level 6
    7 years ago

    If you're not filling buffers quickly enough, increasing the number of them makes no difference, apart from using more memory ....

  • mph999's avatar
    mph999
    Level 6
    7 years ago

    It is confusing, I agree.

    Think of water as being data, and buffers being buckets.

    We can adjust the size of the buckets, and the number.

    Data flows from Client to Media server (water in this case) but to get there has to go via a bucket.

    The rules

    You can only start to fill an empty bucket (cleint sending data)

    You can only empty a full bucket (media server writing data to storage)

    If you have poor read from client due to network or whatever, the water will only fill the bucket slowlty - hence bptm saying 'waiting for full' as it can only 'empty' a full bucket. (the exception being the end of backup where there may be less data than a full buffer).  You can also se how increseing the number of buffers (buckets) makes no difference, as you still can't fill them quick enough.

    Of course, the idea is we have many buckets (typically 128 or 256 works well with LTO drives) and there are of size 256k, so there is a whole load of data flowing, into various buckets.

    Does that help ?

  • sdo's avatar
    sdo
    Moderator
    7 years ago

    1) so let's assume if the images are multiplexed which they are as I see in backup report, multiplex column shows yes. How much would it slow the duplication from tape to disk if we have the multiplex of 4 or 8.

    If original mpx-4 then duplication read could be 25% of LTO4.

    If original mpx-8 then duplication read could be 12.5% of LTO4.

    .

    2) What kind of delay I would see in the bptm logs.

    Where you might see a delay, is in bptm "waited for full buffer", i.e. bptm is not being fed fast enough from the tape drive.

    .

    3) Can i change any buffer settings to run the duplication fast

    Posisbly increase NUMBER_DATA_BUFFERS_RESTORE (for duplication read) but I'm not 100% vouching for that.

    Possibly increase NUMBER_DATA_BUFFERS_DISK for the writer - i.e. allow more buffers to be used for the 1-in-4 or 1-in-8 times that data is found by the reader... i.e. cache up as much as possible in the output buffers before  bpdm has to tell bptm "hey bptm, hold on a sec because my buffers are full (because I'm still waiting for SMB across the LAN to catch up)".

  • mph999's avatar
    mph999
    Level 6
    7 years ago

    There should be a line that reads "read_buffer: waited for empty buffer xx times, delayed yy times"

    You do not show this, altough I expect the numbers will be low (this would show that the job is reading from the tape)

    There will be anther line that you didn't show "write_data: waited for full buffer xx ...." - this is the one we really want to see, though I suspect the numbers may be higher that what we see in the fill_buffer line.

  • sdo's avatar
    sdo
    Moderator
    7 years ago

    Apologies, I don't know.  Maybe with verbose level 5 logging then the logs might reveal the count or percentage of blocks retrieved (read and duplicated) versus the count of blocks passed over (i.e. of all of the other ignored spliced mpx blocks from other backup images)... but I doubt whether the logs will have this.

    If you want to find out which backup images are spliced through which tapes with which other backup images then that meta-data can be pieced together from the fragment numbers in the image catalog.  But the mpx numbers will rise and fall as backup images (in the distant past) "started and joined mpx" and "finished and left mpx"... so, to me neither the question nor the answer really makes any sense.

    And, whilst I haven't looked, I can say that I haven't ever noticed the bptm logs for duplications log facts like "mpx stream now has N spliced"... so I think the best that anyone can do is... hopefully you know the mpx level from when the tapes were written... and so divide the typical LTO4 speed by that, and then that figure is perhaps your best possible sustained average duplication speed.

    Remember mpx was originally a feature to help reduce hadware costs and to reduce shoe-shining.  If previous admins have pushed mpx to the limits to avoid spending on hardware, then we are at the mercy of those prior decisions.  No-one would or could have ever described mpx as a feature to speed up restores or duplications... except in a few arcane configurations with very well defined requirements - but that's a different story.

  • Marianne's avatar
    Marianne
    Level 6
    7 years ago

    Duplication logs will not show original MPX number.

    All Log Entries and the bptm logs of the backup shows MPX value.
    At the end of the backup, something like this is logged in bptm:

    successfully wrote <number> of <number> multiplexed backups, total Kbytes <number> at <number> Kbytes/sec

  • raffl's avatar
    raffl
    Level 4
    7 years ago

    These are very old backups so I don't have logs

  • raffl's avatar
    raffl
    Level 4
    7 years ago

    so let's assume if the images are multiplexed which they are as I see in backup report, multiplex column shows yes. How much would it slow the duplication from tape to disk if we have the multiplex of 4 or 8. What kind of delay I would see in the bptm logs.


    Can i change any buffer settings to run the duplication fast.