Almost certainly, you have an issue outside NBU. I appreciate this happened after the upgrade, however NetBackup doesn't actually write to tape itself (the OS does it).
The block size is 64k
(16:08:21.418 [78489] <2> write_data: received first buffer (65536 bytes), begin writing data)
This is a bit small for modern drives but should work without issue, however, I'd suggest:
Create file called /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS on every media server
Add this number into the file : 262144
This will be picked up on the next backup. Please also test a restore of a few files.
I don't think this will fix the issue, but it's a recommended tuning step and should give better speed.
What has happened:
16:12:54.303 [78489] <2> io_terminate_tape: absolute block position prior to writing empty header is 1885518, copy 1
16:12:54.303 [78489] <2> io_terminate_tape: block position check: actual 1885518, expected 1731186
The tape started at this position
16:08:15.325 [78489] <2> write_data: absolute block position prior to writing backup header(s) is 1576851, copy 1
Netbackup, then sent some data to the operating system, it sent in total 154335 blocks.
After the write finished, NBU knows the tape started at position 1576851, it sent 154335 blocks and so the tape should have finished at position 1576851+ 154335 = 1731186
It then asks the tape drive, 'What position are you at', the tape drives sends back the answer 'I'm at position 1885518' - this does not match where the tape should be, hence the error.
Could NBU cause it - well, if we mis-counted yes, but if that was the case it would fail on every backup because the same code would be used, and it would always mis-count.
Lets look:
Started at position 1576851
16:12:46.235 [78489] <2> write_data: Total Kbytes transferred 9877280
9877280/64 = 154332.5 - so the data backed up was about 154332 64k blocks (this will be a tiny bit out as there are headers as well), but it's close enough.
Lets say then - NBU sent 154332 blocks of data.
start position + number blocks sent = end position
1576851 + 154332 = 1731183
From this line:
16:12:54.303 [78489] <2> io_terminate_tape: block position check: actual 1885518, expected 1731186
NBU expected 1731186 (I'm close enough with my value of 1731183) but the drive is reporting = 1885518 - miles out ...
1885518 - 1731186 = 154332 blocks
Or in other words, the drive is positioned 154332 blocks further down the tape than it should be (or at least, this is what it reports).
The issue in not NBU, I'd suggest upgrading the firmware / drives (HBA and tape drives ) and see if that helps. If they are on the latest firmware, downgrade to the previous (the lateset is not always the best).
As a first look, I'm happy that I have proved this is not a NBU issue.
