07-24-2017 08:59 AM
i am duplicating images from tape to disk and getting 20-30 GB/Hour. These are LTO 4 tapes and duplicating them to advanced disk.
If these tapes are multiplexed will it imact the duplication ?I see in tape report that they were multiplexed but doesn't show what multiplexing number was used.
Pls help me with my performnce.
I have 1 master and 4 media server. 1 server reading 4 drives.
number data buffers = 32
size data buffer = 262144
Solved! Go to Solution.
07-26-2017 05:30 AM
01:55:10.752 [25808.14272] <2> read_data: waited for empty buffer 1608 times, delayed 17063 times
01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times
For a duplication, bptm does both the read and the write.
So we have:
Source side data ---> Buffers ----> Duplication side
The source side fills up the buffer, and was delayed from doing so 17063 times, each delay is I think 15 milli seconds, so in the dupration of the job, there was a (15 x 17063)/ 1000 second delay = about 255 seconds, where the read side process was doing nothing (well it was waiting for a buffer to become free). Presuming the job length was 'kinda long' , 255 sec isn't worth bothering about (in fact, you usually will get some delays, even in the best tuned system), as it is a small % of the totakl job length. that said, I don;t know what the job length was, so you have to make a judgement as to how significant the delay is.
The write side is the bigget problem:
01:55:24.908 [21596.7104] <2> write_data: waited for full buffer 365295 times, delayed 1395403 times
Here bptm, waited 1395403 for data to be duplicated to be written to the buffer - so although the bptm read process was going through the tape, it was only finding relevant data 'every now and then', so the write process bptm had to wait until the buffer was full of data relevant to the backup being duplicated.
(15 x 1395403 )/ 1000 = 20931 sec = 348 minutes = 5.8 hours.
So, the mpx of the backup, may be contributing up to 5.8 hours of delays, in this specific job.
07-24-2017 11:24 AM - edited 07-24-2017 11:27 AM
That's not very fast, 20 GB/hr = 5.5 MB/s, and 30 GB/hr = 8.5 MB/s.
Times four active duplications thats 22.8 MB/s to 34.1 MB/s which looks suspiciously like USB2 disk. Is your Adv.Disk storage unit on USB2 disk?
And, yes, previously multi-plexed tapes can be slower to read for single threaded duplication/restore simply because the tape can be more sparsely populated with the backup data you are seeking. But, yes, I would expect better than 5.5 MB/s from an LTO4 tape drive... assuming that the data that you are reading is not extremely sparse.
Did you baseline the performance of the disk being written to? i.e. do you already know that the disk that you are writing to can actually sustain, over long periods of time, more than 20/30 GB per hour of mixed sequential/random disk IO ?
Are all four duplication reads all writing to the same Adv. Disk storage unit? If so, then your actual disk write patterns will be effectively random.
Is the disk being written to being used for or by anything else... anywhere along the IO path?
Is the Adv.Disk storage unit on a share or UNC path? Or is it actually real DAS or real SAN (iSCSI/FC) disk?
You may be able to tweak NUMBER_DATA_BUFFERS_RESTORE for tape reading... but I'm not 100% certain whether duplication jobs pay any attention to that tuning file. Maybe someone else can confirm.
And maybe also tuning NUMBER_DATA_BUFFERS_DISK for the disk writing side.
See tech note:
Notes on number data buffers files
https://www.veritas.com/support/en_US/article.000076412
.
How many backups, tapes, GB do you have to duplicate?
Are you intending to ultimetly duplicate these backups back to newer tape?
07-24-2017 08:10 PM
Advanced disk is created over the unc path and it is DAS to a server and dedicated for duplication. All the duplication goes to this ADV disk at the same time.
There would be lot of images and yes I might duplicate them to LTO7 at later stage
07-24-2017 09:11 PM
07-25-2017 01:19 AM
i tested this today and copied the data directly, it is 120 mb/sec over the n/w
07-25-2017 01:21 AM
Next step is to check bptm log.
07-25-2017 02:24 AM - edited 07-25-2017 02:25 AM
Is the tape reader host also Windows, or is the NetBackup Server on Linux/Unix and thus perhaps using Samba client?
Was your network IO test from the same NetBackup Server to the same storage server (share/unc) using the same SMB protocol? Or was 120MB/s achieved using SCP or FTP or NFS?
Marianne is right, you'll need to check bptm and bpdm logs. Look for delay and wait counters, and read a few tech notes on how to visualize/theorize the actual numbers... sometimes the way to interpret the numbers is not obvious and you need to think of it in terms of source/producer and target/consumer because, if nothing odd or h/w related is found in the logs, then the bptm and bpdm delay and wait figures may well reveal where a potential bottleneck is.
e.g. the tape drive is the producer and bptm is the consumer, then blocks get transfered in memory from tape buffers to disk buffers, then bpdm is producer and samba client is consumer... before then being punted across TCP via SMB blocks to a server that has to reconstitute NTFS files from the SMB blocks.
07-25-2017 02:29 AM
randomly collected this just now
00:02:03.438 [25808.14272] <2> read_data: stopping mpx read because 5027840 Kbytes of 5027840 were read
00:02:03.438 [25808.14272] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 27
00:02:03.438 [25808.14272] <2> read_data: waited for empty buffer 1514 times, delayed 15534 times
00:14:53.346 [3840.23296] <2> read_data: stopping mpx read because 3271168 Kbytes of 3271168 were read
00:14:53.346 [3840.23296] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 2
00:14:53.361 [3840.23296] <2> read_data: waited for empty buffer 98 times, delayed 1590 times
00:59:10.877 [24292.27868] <2> read_data: stopping mpx read because 103527424 Kbytes of 103527424 were read
00:59:11.033 [24292.27868] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 22
00:59:11.033 [24292.27868] <2> read_data: waited for empty buffer 91181 times, delayed 915614 times
01:11:04.323 [25808.14272] <2> read_data: stopping mpx read because 719872 Kbytes of 719872 were read
01:11:04.323 [25808.14272] <2> io_ioctl: command (1)MTFSF 1 0x0 from (bptm.c.14997) on drive index 27
01:11:04.338 [25808.14272] <2> read_data: waited for empty buffer 248 times, delayed 2946 times
07-25-2017 02:32 AM
yes everything is windows. the copy transfer is exactly same the way nbu is configured for the share. i have pasted the output of my bptm and it is some times with low waiting buffer sometimes more.
07-25-2017 02:36 AM
What I mean is... that for any IO... anywhere on any platform for any product for any OS for any software on any hardware... *always* involves something which is writing and something different which is reading. Any IO is never about one number, e.g. my data transfer is only 50 MB/s is an unclear statement... it is clearer to think of it in terms of either the writer/producer or the reader/consumer is, for some reason, restricted to 50 MB/s... but there is a hidden facet... a nasty gotcha... in that both theproducer and consumer might be restricted (for different reasons) to 50MB/s... in which case no amount of tuning either side will change anything.
07-25-2017 02:40 AM
ok raffl - don't foget to look in the logs for errors and warnings too... look for any unusual time gaps between events/stages too... and so now begins your journey in to rationalising the wait and delay numbers... I mean, have a go yourself... and see if you can deduce what is going on! good luck.
07-25-2017 02:44 AM
bptm log will also show buffer size and number being used for read and write.
De-multiplexing will surely slow down the process as the tape constantly needs to reposition to find all the pieces for one image at a time.
07-25-2017 03:00 AM
It's worse than that .... it doesn't reposition, it reads through everything on the tape ...
A mpx tape, say mpx of 4 looks something like this
MH BH1 BH2 BH3 BH4 D1 D1 D3 D2 D4 D1 D1 D3 D4 D2 D2 D2 D2 D2 D3 D1 D1 D1 D3 D3 D3 D3 EH EOT
Where BHx is the backup header for client x / Dx is a data block from client x
So, all the Backup hears are grouped at the start, so we know which clients are on the tape, but all the data is mixed, as you know.
This means, If we want say a restore from client3, we have to read through all the previous data blocks for the other clients. Athough the data block contains a few bytes (*) to identify the client it relates towe have no way to tell where in the mix it is, so have to read through everything.
(*) Something along these lines, it's all a bit secret squirrel stuff ....
IMHO, anything above an MPX4 is madness if you want any real chance of a reasonable restore speed.
07-25-2017 03:32 AM
any logs which shows the mpx while duplicating the data
07-25-2017 04:14 AM - edited 07-25-2017 04:18 AM
Apologies, I don't know. Maybe with verbose level 5 logging then the logs might reveal the count or percentage of blocks retrieved (read and duplicated) versus the count of blocks passed over (i.e. of all of the other ignored spliced mpx blocks from other backup images)... but I doubt whether the logs will have this.
If you want to find out which backup images are spliced through which tapes with which other backup images then that meta-data can be pieced together from the fragment numbers in the image catalog. But the mpx numbers will rise and fall as backup images (in the distant past) "started and joined mpx" and "finished and left mpx"... so, to me neither the question nor the answer really makes any sense.
And, whilst I haven't looked, I can say that I haven't ever noticed the bptm logs for duplications log facts like "mpx stream now has N spliced"... so I think the best that anyone can do is... hopefully you know the mpx level from when the tapes were written... and so divide the typical LTO4 speed by that, and then that figure is perhaps your best possible sustained average duplication speed.
Remember mpx was originally a feature to help reduce hadware costs and to reduce shoe-shining. If previous admins have pushed mpx to the limits to avoid spending on hardware, then we are at the mercy of those prior decisions. No-one would or could have ever described mpx as a feature to speed up restores or duplications... except in a few arcane configurations with very well defined requirements - but that's a different story.
07-25-2017 04:45 AM
Duplication logs will not show original MPX number.
All Log Entries and the bptm logs of the backup shows MPX value.
At the end of the backup, something like this is logged in bptm:
successfully wrote <number> of <number> multiplexed backups, total Kbytes <number> at <number> Kbytes/sec
07-25-2017 05:12 AM
These are very old backups so I don't have logs
07-25-2017 05:54 AM
so let's assume if the images are multiplexed which they are as I see in backup report, multiplex column shows yes. How much would it slow the duplication from tape to disk if we have the multiplex of 4 or 8. What kind of delay I would see in the bptm logs.
Can i change any buffer settings to run the duplication fast.
07-25-2017 10:34 AM - edited 07-25-2017 10:41 AM
1) so let's assume if the images are multiplexed which they are as I see in backup report, multiplex column shows yes. How much would it slow the duplication from tape to disk if we have the multiplex of 4 or 8.
If original mpx-4 then duplication read could be 25% of LTO4.
If original mpx-8 then duplication read could be 12.5% of LTO4.
.
2) What kind of delay I would see in the bptm logs.
Where you might see a delay, is in bptm "waited for full buffer", i.e. bptm is not being fed fast enough from the tape drive.
.
3) Can i change any buffer settings to run the duplication fast
Posisbly increase NUMBER_DATA_BUFFERS_RESTORE (for duplication read) but I'm not 100% vouching for that.
Possibly increase NUMBER_DATA_BUFFERS_DISK for the writer - i.e. allow more buffers to be used for the 1-in-4 or 1-in-8 times that data is found by the reader... i.e. cache up as much as possible in the output buffers before bpdm has to tell bptm "hey bptm, hold on a sec because my buffers are full (because I'm still waiting for SMB across the LAN to catch up)".
07-25-2017 10:48 AM - edited 07-25-2017 10:49 AM
Cold comfort... but if you sit down and think about it... the feature of multi-plexing is actually very nice indeed... and extremely useful... and the more one thinks about it... the more one realizes how incredibly sophisticated the NetBackup code must be to enable features such as:
- backup stream join an already ongoing multi-plex set
- backup stream leave an ongoing multi-plex set
(...and my favourite mpx feature...)
- multi-plexed tape read... and later... multi-plexed duplication (yes, that is a single tape pass)... to multi-plexed tape write