LTO9 tape performance poor - bottlenecks?
Hi all We've recently completed a staged total replacement of our environment from NBU8.2, writing first to HPE MSA2040 storage and then out to LTO5 drives; to NBU9.1 writing to HPE MSA2060 storage ...
Forum Discussion
There's no settings files / registry entries for any of the settings so presumably all are default. I'm creating the NUMBER_DATA_BUFFERS files - the doc only mentions a reboot required for registry changes, will that be required for creating the files?
Not every job has waited for messages within it, indeed most don't - but I've found a few in some duplication jobs - some of them quite large in numbers:
11-Oct-2023 12:04:26 - Info bptm (pid=10408) waited for full buffer 2556 times, delayed 4739 times
By comparison, there is a duplication job that is writing out at an insanely slow pace - 363GB in 22 hours - that has no waited for messages at all. This 'just happens' sometimes, generally from one specific media server and killing the job will often result in another duplication starting from the same media server at ~200GB/hr speed.
The answer to do I need to reboot the servers is no, they've taken the new buffer size on new jobs started since the change.
LTO9 performance is markedly better on the first duplications started since - between 600GB/hr to 800GB/hr - but the waited for numbers are stratospheric on one of them:
waited for full buffer 42564 times, delayed 75144 times
Coincidentally they're nearly all on the same media server, which is currently under immense load - two duplications running with 512 buffers, and three with the old default (there is a backlog on the LTO5 SLPs that we decided to let run to LTO5 tape, hence more jobs than LTO9 drives) - so that might now be a very temporary limiting factor.
NUMBER_DATA_BUFFERS will be read by the BPTM process when starting up. Long running jobs before the change will continue to use the old value.
To me the "waited for" doesn't look to bad. You can't avoid them, but keeping the numbers low should be keep a eye on
What OS are the media servers ?
If Linux then try the GEN_DATA directive. And no, this feature isn't available on Windows.
https://www.veritas.com/support/en_US/article.100030600
GEN data will generate data in memory and write the data to eiter disk or tape. This is a brilliant way to test if the tape subsystem has the intended performance. You can also "restore" the data, but no data will really be written to any location.
Duplication: If the duplication is from disk to tape, pls ensure you don't mix write and read workloads. It is really kill performance, especially on spinning disk
Entire environment is Windows. Duplications are disk to disk to tape; storage array is spinning disk I believe (I don't admin it thankfully). Generally there would only be write operations to the secondary disk storage in the early morning; tape windows are currently 24/7 but if I can get out of a backlog situation I have been considering controlling the windows to avoid overlap.
I won't have any particularly large jobs to write to the LTO9 environment until the weeklies run on Friday night; but the old SLPs that are writing to the LTO5 drives have accelerated significantly now too - one completed at nearly native drive speed (job was a little above 500GB/hr) so I'm hopeful that this has solved the problem.
