04-14-2016 09:00 AM
Hi team
I'm trying to import 3 physical tapes (LTO3), master is NBU 7.5 over linux redhat, using IBM ts3100.
The tapes was created using this configuration
Backup was made with Netbackup 7.0, using VTL DL4100.
EDL4100 duplicate the tapes into Adic i500 (drive IBM Ultrium)
Now, I need import these tapes.
When i run the 1st fase for import this error appears
04/14/2016 12:28:03 - Info bptm (pid=17748) Waiting for mount of media id CA1065 (copy 1) on server lacarglxap006.
04/14/2016 12:28:03 - started process bptm (pid=17748)
04/14/2016 12:28:03 - mounting CA1065
04/14/2016 12:28:03 - Info bptm (pid=17748) INF - Waiting for mount of media id CA1065 on server lacarglxap006 for reading.
04/14/2016 12:28:03 - granted resource CA1065
04/14/2016 12:28:03 - granted resource IBM.ULT3580-HH5.003
04/14/2016 12:29:32 - mounted CA1065; mount time: 0:01:29
04/14/2016 12:29:32 - Error bptm (pid=17748) block read is not a NetBackup or Backup Exec media header, len = 65536, media id CA1065, drive index 41, data is unknown
04/14/2016 12:29:32 - Info bptm (pid=17748) EXITING with status 172 <----------
04/14/2016 12:29:32 - Error bpimport (pid=11392) Status = cannot read media header, may not be NetBackup media or is corrupted.
04/14/2016 12:29:32 - end Import; elapsed time 0:08:37
cannot read media header, may not be NetBackup media or is corrupted (172)
but if use the command, previosuly tape monted
/opt/openv/volmgr/bin/scsi_command -map -f /dev/nst0
I can see some dates and some imagenes for the tape
00000000: file 1: record 1: size 65536
01555929: file 1: eof after 1555929 records: 101969362944 bytes
01555930: file 2: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 2: file 2: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1065
01555931: file 2: record 2: size 65536
As I can understand, the tape is good .
What more I need to do for import these tapes?
Thanks
04-20-2016 07:55 AM
Stumbled on to this tech note https://www.veritas.com/support/en_US/article.000024136 about 172 caused by wrong density, while searching for something else on the support site
04-20-2016 01:21 PM
I don't know for sure, but file one record one, if a VTL header would be followed by a tape mark.
Good find on the TN, however again I think not the case here. QCART is a special case and writes in fixed, I think 512 byte blocks - here the tape is not written in fixed length blocks as the backup header we see is 1k.
I'll ask a colleague to take a look at this, at the moment, :
1. Try tar - can we confirm what the data is at the beginning of the tape
2. Is the original tape available can we check that.
As mentioned, my concern is that we are blaming the VTL unfairly, the issue 'might' have happened on the original tape (hence step 2) and because it was discovered after the VTL duplicate teh tape 'outside NBU' - which is a little unusually and technically 'not supported' it is easy to jump to the wrong conclusion.
Rule one for troubleshooting - keep an open mind, it might not be what you think and 'anything' is possible until proven otherwise.
Even though duplicated outsoide NBU, the tape should still import into a differntent NBU environment providing it is a perfect copy AND the original tape has the backup header.
Also, another thought, would a VTL header not be on an area of the 'tape' that isn't read by NBU ... and as this is a copy to physical, there should be no VTL header at all.
04-20-2016 04:27 PM
The physical tape format is wrong... I'm assuming the original VTL media is no longer available ?
If the original VTL media is still available, try copying the VTL tape to physical tape using 'tcopy' - lots of copies out there in the wild, it's a native BSD utility. (or use NetBackup to duplicate the tape)
A data recovery company may be able to help if the source VTL media isn't available.
Or you could try to 'copy' the tape again, but this time make sure the first block is 1k in size.. otherwise NetBackup will reject the tape as a non-NetBackup format.
Can you 'dd' the first block off tape into a file ?
e.g. 'dd if=/dev/nstX of=/tmp/header bs=64k count=1'
Then perform a 'strings /tmp/header' and confirm the media label looks sane.
If the media header is there, grab source of tcopy.c, modify so the first block only writes out 1k and recompile. tcopy the tape to another tape and try importing the new copy.
Until that media header is 1k, NetBackup logic to detect valid media format will reject this tape.
It looks like a bug with the VTL native copy utility and I would suspect all such copied media will be corrupt.
04-21-2016 12:53 AM
Thanks Mark - much appreciated.
04-21-2016 08:08 AM
Michael, yes, was written by EDl4100, using native exportations. On the tape, mouse right buttom and export. The density is correct.
Markh794, virtual tape media was erased long time ago, also EDL4100 was destroyed.
I will try to do what you recommend, here take time for complete the permission for execute the procedure.
For all, I execute SCSI_COMMAND -MAP and this was the result for the 3 tapes with issue
cat /tmp/CA1065
00000000: file 1: record 1: size 65536
01555929: file 1: eof after 1555929 records: 101969362944 bytes
01555930: file 2: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 2: file 2: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1065
01555931: file 2: record 2: size 65536
02840465: file 2: eof after 1284535 records: 84183221248 bytes
02840466: file 3: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 3: file 3: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1065
02840467: file 3: record 2: size 65536
04347023: file 3: eof after 1506557 records: 98733655040 bytes
04347024: file 4: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 4: file 4: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1065
04347025: file 4: record 2: size 65536
05771717: file 4: eof after 1424693 records: 93368615936 bytes
05771718: file 5: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 5: file 5: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1065
05771719: file 5: record 2: size 65536
07509136: file 5: eof after 1737418 records: 113863361536 bytes
07509137: file 6: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 6: file 6: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1065
07509138: file 6: record 2: size 65536
07725381: file 6: eof after 216244 records: 14171702272 bytes
eot
contenido tape CA1066
cat /tmp/CA1066
00000000: file 1: record 1: size 65536
01273645: file 1: eof after 1273645 records: 83469598720 bytes
01273646: file 2: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 8: file 2: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1066
01273647: file 2: record 2: size 65536
03052345: file 2: eof after 1778699 records: 116568753152 bytes
03052346: file 3: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 9: file 3: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1066
03052347: file 3: record 2: size 65536
04541481: file 3: eof after 1489135 records: 97591886848 bytes
04541482: file 4: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 10: file 4: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1066
04541483: file 4: record 2: size 65536
05939257: file 4: eof after 1397775 records: 91604517888 bytes
05939258: file 5: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 11: file 5: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1066
05939259: file 5: record 2: size 65536
07258097: file 5: eof after 1318839 records: 86431368192 bytes
07258098: file 6: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 12: file 6: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media CA1066
07258099: file 6: record 2: size 65536
07737380: file 6: eof after 479282 records: 31410160640 bytes
eot
Contenido tape A1068 cat /tmp/CA1068
00000000: file 1: record 1: size 65536
00951648: file 1: eof after 951648 records: 62367203328 bytes
00951649: file 2: record 1: size 1024: NBU BACKUP header
backup_id lacargbaas099_1396049286: frag 14: file 2: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media E00207
00951650: file 2: record 2: size 65536
00984618: file 2: record 32970: size 55296
00984619: file 2: eof after 32970 records: 2160647168 bytes
00984620: file 3: record 1: size 1024: NBU TIR header
backup_id lacargbaas099_1396049286: frag -1: file 3: copy 1
expiration 2147483647: retention 9: block_size 65536
flags 0x4: mpx_headers 0: resume_count 0: media E00207
expected size 129233109
00984621: file 3: record 2: size 65536
00986592: file 3: record 1973: size 62464
00986593: file 3: eof after 1973 records: 129234944 bytes
00986594: file 4: record 1: size 1024: NBU EMPTY header (file 4)
00986595: file 4: eof after 1 records: 1024 bytes
eot
Something is not good here ...
I saw 00951649: file 2: record 1: size 1024: NBU BACKUP header
Is it NBU Header or not?
Is I use tar I saw
/opt/openv/netbackup/bin/tar tvf /dev/nst0
Blocksize = 128 records
Hmm, this doesn't look like a tar archive.
Skipping to next file header...
Is a inconsistency?
Maybe my skills are not enauf for understand this issue ...
Any Comments?
Thanks to all !!!!
04-21-2016 10:18 AM
Think you could read the 2nd file and forward on these tapes by setting the block size (b) on tar to 128 (65536/512)
I have an idea that the Netbackup tape header and the first Netbackup Image header is inside the first 64KB record.
I would try use dd with ibs=64K and different obs values starting with 1K on the first 64K record, too see if I could find something looking like a Netbackup header
Thinking something like dd if=/dev/rmt/<tape dev> ibs=64K count=1 obs=1K | od -cx
04-21-2016 04:40 PM
I saw 00951649: file 2: record 1: size 1024: NBU BACKUP header
This is correct, there is a backup header at the start of each image on the tape (see my diagram above) - therefore a tape will have 1 or more backup headers
There should be a media header at the beginning, this is 'missing' - a tape will only ever have 1 media header.
eg.
MH BH DATA DATA DATA BH DATA BH DATA DATA DATA DATA DATA DATA BH DATA EH
(More DATAs mean a bigger backup, EH = Empty header at the end)
The EH is used to locate the end of the last backup, so we can append more backups later. A tape that spans to another (backup was too big for the tape) will not have an EH, but we don't need it as we'll never append to a full tape.
Without the MH (media header) NBU cannot mount the tape.
Try Marks suggestion (he wrote MHVTL and is one of, if not the, most knowledgeable people in the company regarding VTLs/ tapes (and other things as well, he's awesome)..
I have to admit, that if Mark is correct, I never thought the VTL copy could screw things up that much.
04-24-2016 03:20 PM
In front of a computer instead of a mobile device..
Here is what the process should look like.
Mount tape with media ID J00008 of hcart density
# tpreq -m J00008 -d hcart -p NetBackup -f /tmp/fred
Read in the first block of data (Media Header) - Attempt to read in 64k but we all know it should only read in 1k
# dd if=/tmp/fred bs=64k count=1 of=/tmp/media_header 0+1 records in 0+1 records out 1024 bytes (1.0 kB) copied, 0.00071469 s, 1.4 MB/s
Now use hexdump to view the contents
# hexdump -C /tmp/media_header 00000000 56 4f 4c 31 4a 30 30 30 30 38 00 00 00 00 00 00 |VOL1J00008......| 00000010 00 00 00 00 00 00 00 01 00 00 00 06 57 1c 73 a1 |............W.s.| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 |................| 00000040 00 00 00 00 00 00 00 00 00 00 04 00 00 00 00 00 |................| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000070 54 68 49 73 20 49 73 20 41 20 42 50 20 74 41 70 |ThIs Is A BP tAp| 00000080 45 20 68 45 61 44 65 72 00 00 00 00 00 00 00 00 |E hEaDer........| 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000400
And clean up afterwards - unmount the tape
# tpunmount /tmp/fred
04-24-2016 04:08 PM
I have managed to 'reproduce' the problem by some use of 'dd' and 'tcopy' (if you have problems finding a copy - download the latest copy of 'mhVTL' - www.mhvtl.com. There is a copy of the tcopy utility included with the source).
Mount the source media - I'm choosing /tmp/corrupt (but any filename that does *not* exist will do)
# tpreq -m J00001 -d hcart -p NetBackup -f /tmp/corrupt # scsi_command -d /tmp/corrupt -map 00000000: file 1: record 1: size 65536 <==== 64k not a valid NetBackup media header 00000001: file 1: eof after 1 records: 65536 bytes 00000002: file 2: record 1: size 1024: NBU BACKUP header backup_id mhnbu75_1461482401: frag 1: file 1: copy 1 expiration 1462087201: retention 0: block_size 524288 flags 0x0: mpx_headers 0: resume_count 0: media J00008 00000003: file 2: record 2: size 524288 00000123: file 2: record 122: size 65536 00000124: file 2: eof after 122 records: 62981120 bytes 00000125: file 3: record 1: size 1024: NBU EMPTY header (file 2) 00000126: file 3: eof after 1 records: 1024 bytes 00000127: file 4: eof after 0 records: 0 bytes eot
Move the new tape into a pool (so it won't be chosen by NetBackup for fresh backups) - I'm using the 'None' pool in this example:
# vmchange -m J00002 -p 0
Mount media J00002 - I'm choosing /tmp/fix (but any filename that does *not* exist will do)
# tpreq -d hcart -m J00002 -p None -f /tmp/fix
Just to be sure - rewind both tapes:
# mt -f /tmp/corrupt rewind # mt -f /tmp/fix rewind
Read in the first block of data - we already know it is 64k
# dd if=/tmp/corrupt of=/tmp/blk_header bs=64k count=1 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00105776 s, 62.0 MB/s
Confirm the first 1k looks OK (if not, it's time to have a chat with a data recovery house - sorry).
This exampe I had 'corrupted' the 1k block with additional 63k of NULLs - but it's the data between 00000000 to 00000090 that we're concerned with.
# hexdump -C /tmp/blk_header 00000000 56 4f 4c 31 4a 30 30 30 30 38 00 00 00 00 00 00 |VOL1J00008......| 00000010 00 00 00 00 00 00 00 01 00 00 00 06 57 1c 73 a1 |............W.s.| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 |................| 00000040 00 00 00 00 00 00 00 00 00 00 04 00 00 00 00 00 |................| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000070 54 68 49 73 20 49 73 20 41 20 42 50 20 74 41 70 |ThIs Is A BP tAp| 00000080 45 20 68 45 61 44 65 72 00 00 00 00 00 00 00 00 |E hEaDer........| 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00010000
Now read the first 1k of data from /tmp/blk_header to the tape mounted as /tmp/fix
# dd if=/tmp/blk_header of=/tmp/fix bs=1k count=1 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00668387 s, 153 kB/s
The above 'dd' command will write the 1k of data plus the filemark and will be positioned at the EOT (End of Tape) side of the filemark.
Lets position the source media in the same spot (Note: I'm using the 'mt' command from the optional mt-st RPM package):
# mt -f /tmp/corrupt rewind # mt -f /tmp/corrupt fsf # mt -f /tmp/corrupt tell At block 2.
Now use the 'tcopy' utility to copy from source (/tmp/corrupt) to destination (/tmp/fix)
# tcopy /tmp/corrupt /tmp/fix file 0: block size 1024: record 0 file 0: block size 524288: records 1 to 121 file 0: block size 65536: record 121 file 0: eof after 122 records: 62981120 bytes file 1: block size 1024: 1 records file 1: eof after 1 records: 1024 bytes eot total length: 62982144 bytes
Time for a cup of coffee while the tape contents is copied - it is typical to take about 2hrs to read a tape from end-to-end ;)
Now lets check the final result:
# mt -f /tmp/fix rewind # scsi_command -d /tmp/fix -map 00000000: file 1: record 1: size 1024: NBU MEDIA header (J00008) <=== With luck, this is now reported as a NBU Media header 00000001: file 1: eof after 1 records: 1024 bytes 00000002: file 2: record 1: size 1024: NBU BACKUP header backup_id mhnbu75_1461482401: frag 1: file 1: copy 1 expiration 1462087201: retention 0: block_size 524288 flags 0x0: mpx_headers 0: resume_count 0: media J00008 00000003: file 2: record 2: size 524288 00000123: file 2: record 122: size 65536 00000124: file 2: eof after 122 records: 62981120 bytes 00000125: file 3: record 1: size 1024: NBU EMPTY header (file 2) 00000126: file 3: eof after 1 records: 1024 bytes 00000127: file 4: eof after 0 records: 0 bytes eot
Unmount source and destination tapes:
# tpunmount -f /tmp/corrupt # tpunmount -f /tmp/fix
And repeat the process for your other tapes.. This email thread shows you have at least three tapes for backup image - lacargbaas099_1396049286
Please post an update so we know how you get on.
04-24-2016 04:19 PM
One follow up comment - Reading thru the thread again, I noticed you needed change control to have some (mt) packages installed.
My recommended minimum set of RPM packages to troubleshoot SCSI devices (including disk, tape and libraries) on Linux are:
- mt-st (for tape drives) - mtx (for tape libraries) - sg3_utils (for all SCSI devices) - lsscsi (for all SCSI devices)
04-27-2016 08:25 AM
Wow ! a lot new information for me.
I will try to do what are you suggested. I havn't free time in this moments.
I'll keep informed to all.