Forum Discussion

DBWR's avatar
DBWR
Level 3
10 years ago

Recover from partial backups

We had several backups where the tapes got full.  I know this is a situation you want to avoid, but there was a period of time where there wasn't much we could do about it, and depending on the amount of data there on the day sometimes it'd finish sometimes it wouldn't.

 

When it failed, it has still backed up 95% of the files.  I've gone to restore from one of the old tapes, it was quite a while ago so I had to import it, and the import fails saying "...tar did not find all the files to be restored".  I know it can't find all the files, but what about the ones that are there?  Just because a few files were missed off the end I can't believe it would invalidate the entire backup.

 

Does anyone know a way around this?

 

Thanks

Matt

  • The NBU is virtualy the same, same options ...  The main difference from memory is it frops the leading / so retores are relative to where you, not absolute (making althernate restore locations possible).

    This is the posh way to do it  ...  It was aimed at images that span tapes, but works fine for a single tape.  You actually get the data off the tape with dd, then untar the files later.  

    How to use dd to read data off a tape (Unix/ Linux)

    On occassion it may be necessary to try and read data off a tape if there is a failure in NetBackup.  Usually this will be only to demonstrate that the tape is unreadable due to an issue outside of NetBackup.

    It is easier to explain this via example.

    We can see from bpimagelist that the image nbmedia00_1418641103 spans 2 tapes, E01001 and E01002

    IMAGE nbmedia00 0 0 8 nbmedia00_1418641103 nbmedia_big 0 *NULL* root Full 0 1 1418641103 130 1419850703 0 0 2545152 1 1 2 0 nbmedia_big_1418641103_FULL.f *NULL* *NULL* 0 1 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 269 0 0 740 0 0 *NULL* *NULL* 0 0 0 0 *NULL* *NULL*
    HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
    FRAG 1 1 1566208 0 2 6 1 E01001 nbmedia00 262144 2 1418641103 2 0 *NULL* 1419850703 0 65537 0 0 0 1 0 1418641233 0 *NULL* *NULL*
    FRAG 1 2 978944 0 2 6 1 E01002 nbmedia00 262144 2 1418641103 2 0 *NULL* 0 0 0 0 0 0 1 0 0 0 *NULL* *NULL*

    It does not actually matter if the image spans multiple tapes, or is contained on a single media, the procedure is exactly the same.

    First, use scsi_command -map to obtain an image of what is on the tape, this needs to be done in fragment order, so E01001 is the first tape as it contains fragment 1, E01002 is the second tape as it contains fragement 2.

    [root@nbmedia00 testdata]# scsi_command -map -f /dev/nst1
    00000000: file 1: record 1: size 1024: NBU MEDIA header (E01001)
    00000001: file 1: eof after 1 records: 1024 bytes
    00000002: file 2: record 1: size 1024: NBU BACKUP header
              backup_id nbmedia00_1418641103: frag 1: file 1: copy 1
              expiration 1419850703: retention 1: block_size 262144
              flags 0x0: mpx_headers 0: resume_count 0: media E01001
    00000003: file 2: record 2: size 262144
    00006121: file 2: eof after 6119 records: 1603798016 bytes
    eot

    We see that the header for the backup image nbmedia00_1418641103 is located at file 2, and the data fragment is immediate after the header as expected.

    We need to first read this fragment using dd, into a file on disk, followed by the actual data in fragment 1.

    We can build the commands required by referencing the output of scsi_command -map output, consideration must be given to filenames and blocksize used to allow the read via dd to be successful, and correct identification of the files later.  In this example we have used the backupid, follwed by a description of what part of the image the file relates to.

    NOTE:
    We want to position the tape to file 2, fsf 1 is correct as it positions after file 1, which of course is just before file 2.
    mkdir /tmp/recover

    mt -f /dev/nst1 rewind
    mt -f /dev/nst1 fsf 1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag1header bs=1k   count=1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag1 bs=256k


    A quick look at the files created :

    -rw-r--r-- 1 root root 1603796992 Dec 15 03:16 bu-nbmedia00_1418641103_frag1
    -rw-r--r-- 1 root root       1024 Dec 15 03:15 bu-nbmedia00_1418641103_frag1header


    The frag1header (which is the backup header) is 1K in size which is correct for a header.


    We now need to recover any remaining fragments for the image from the next tape, so as before, we repeat the process and run scsi_command -map -d <device> on the next tape, E01002.

    00000000: file 1: record 1: size 1024: NBU MEDIA header (E01002)
    00000001: file 1: eof after 1 records: 1024 bytes
    00000002: file 2: record 1: size 1024: NBU BACKUP header
              backup_id nbmedia00_1418641103: frag 2: file 1: copy 1
              expiration 1419850703: retention 1: block_size 262144
              flags 0x0: mpx_headers 0: resume_count 0: media E01002
    00000003: file 2: record 2: size 262144
    00003827: file 2: eof after 3825 records: 1002439680 bytes
    00003828: file 3: record 1: size 1024: NBU EMPTY header (file 2)
    00003829: file 3: eof after 1 records: 1024 bytes
    eot


    Here we see that we have frag 2 of backup header for frag 2 of image nbmedia00_1418641103 at the position file 2, so we can again put the required commands together.
    In this case, because the fragements we want are both the first fragments on the tapes, the commands we need are the same, so we just change the filenames.

    mt -f /dev/nst1 rewind
    mt -f /dev/nst1 fsf 1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag2header bs=1k   count=1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag2 bs=256k

    Looking now at the files we have created :

    -rw-r--r-- 1 root root 1603796992 Dec 15 03:16 bu-nbmedia00_1418641103_frag1
    -rw-r--r-- 1 root root       1024 Dec 15 03:15 bu-nbmedia00_1418641103_frag1header
    -rw-r--r-- 1 root root 1002438656 Dec 15 03:22 bu-nbmedia00_1418641103_frag2
    -rw-r--r-- 1 root root       1024 Dec 15 03:22 bu-nbmedia00_1418641103_frag2header


    We have the two backup headers (one for each fragment) for reference if required, and the two files holding the actual data.

    We can then 'cat' the two files containing each fragment of the image into one file:

    cat bu-nbmedia00_1418641103_frag* >bu-backup

    And read this single tar file back, which contains the data from the backup.

    [root@nbmedia00 recover]# /usr/openv/netbackup/bin/tar tvf bu-backup
    Erwxr-xr-x root/root 103 Dec 15 02:58 2014 .AtTrIbUtEs.0
    --Extra header --
    drwxr-xr-x root/root   0 Dec 15 02:41 2014 /
    drwxr-xr-x root/root   0 Dec 15 02:49 2014 /testdata/
    -rw-r--r-- root/root 1532471296 Dec 15 02:50 2014 /testdata/tarfile1

  • Yes this sounds very feasible.  It wasn't multiplexed no.

  • Any ideas how to recover the incomplete tar tape?  While I don't use regular tar in linux much I've got a funny feeling if it was there would be a way, but Netbackup's inplementation of tar is different.

  • You have to use the tar in Netbackup as far as I know, tar --help will give all the options as I remember it.

  • The NBU is virtualy the same, same options ...  The main difference from memory is it frops the leading / so retores are relative to where you, not absolute (making althernate restore locations possible).

    This is the posh way to do it  ...  It was aimed at images that span tapes, but works fine for a single tape.  You actually get the data off the tape with dd, then untar the files later.  

    How to use dd to read data off a tape (Unix/ Linux)

    On occassion it may be necessary to try and read data off a tape if there is a failure in NetBackup.  Usually this will be only to demonstrate that the tape is unreadable due to an issue outside of NetBackup.

    It is easier to explain this via example.

    We can see from bpimagelist that the image nbmedia00_1418641103 spans 2 tapes, E01001 and E01002

    IMAGE nbmedia00 0 0 8 nbmedia00_1418641103 nbmedia_big 0 *NULL* root Full 0 1 1418641103 130 1419850703 0 0 2545152 1 1 2 0 nbmedia_big_1418641103_FULL.f *NULL* *NULL* 0 1 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 269 0 0 740 0 0 *NULL* *NULL* 0 0 0 0 *NULL* *NULL*
    HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
    FRAG 1 1 1566208 0 2 6 1 E01001 nbmedia00 262144 2 1418641103 2 0 *NULL* 1419850703 0 65537 0 0 0 1 0 1418641233 0 *NULL* *NULL*
    FRAG 1 2 978944 0 2 6 1 E01002 nbmedia00 262144 2 1418641103 2 0 *NULL* 0 0 0 0 0 0 1 0 0 0 *NULL* *NULL*

    It does not actually matter if the image spans multiple tapes, or is contained on a single media, the procedure is exactly the same.

    First, use scsi_command -map to obtain an image of what is on the tape, this needs to be done in fragment order, so E01001 is the first tape as it contains fragment 1, E01002 is the second tape as it contains fragement 2.

    [root@nbmedia00 testdata]# scsi_command -map -f /dev/nst1
    00000000: file 1: record 1: size 1024: NBU MEDIA header (E01001)
    00000001: file 1: eof after 1 records: 1024 bytes
    00000002: file 2: record 1: size 1024: NBU BACKUP header
              backup_id nbmedia00_1418641103: frag 1: file 1: copy 1
              expiration 1419850703: retention 1: block_size 262144
              flags 0x0: mpx_headers 0: resume_count 0: media E01001
    00000003: file 2: record 2: size 262144
    00006121: file 2: eof after 6119 records: 1603798016 bytes
    eot

    We see that the header for the backup image nbmedia00_1418641103 is located at file 2, and the data fragment is immediate after the header as expected.

    We need to first read this fragment using dd, into a file on disk, followed by the actual data in fragment 1.

    We can build the commands required by referencing the output of scsi_command -map output, consideration must be given to filenames and blocksize used to allow the read via dd to be successful, and correct identification of the files later.  In this example we have used the backupid, follwed by a description of what part of the image the file relates to.

    NOTE:
    We want to position the tape to file 2, fsf 1 is correct as it positions after file 1, which of course is just before file 2.
    mkdir /tmp/recover

    mt -f /dev/nst1 rewind
    mt -f /dev/nst1 fsf 1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag1header bs=1k   count=1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag1 bs=256k


    A quick look at the files created :

    -rw-r--r-- 1 root root 1603796992 Dec 15 03:16 bu-nbmedia00_1418641103_frag1
    -rw-r--r-- 1 root root       1024 Dec 15 03:15 bu-nbmedia00_1418641103_frag1header


    The frag1header (which is the backup header) is 1K in size which is correct for a header.


    We now need to recover any remaining fragments for the image from the next tape, so as before, we repeat the process and run scsi_command -map -d <device> on the next tape, E01002.

    00000000: file 1: record 1: size 1024: NBU MEDIA header (E01002)
    00000001: file 1: eof after 1 records: 1024 bytes
    00000002: file 2: record 1: size 1024: NBU BACKUP header
              backup_id nbmedia00_1418641103: frag 2: file 1: copy 1
              expiration 1419850703: retention 1: block_size 262144
              flags 0x0: mpx_headers 0: resume_count 0: media E01002
    00000003: file 2: record 2: size 262144
    00003827: file 2: eof after 3825 records: 1002439680 bytes
    00003828: file 3: record 1: size 1024: NBU EMPTY header (file 2)
    00003829: file 3: eof after 1 records: 1024 bytes
    eot


    Here we see that we have frag 2 of backup header for frag 2 of image nbmedia00_1418641103 at the position file 2, so we can again put the required commands together.
    In this case, because the fragements we want are both the first fragments on the tapes, the commands we need are the same, so we just change the filenames.

    mt -f /dev/nst1 rewind
    mt -f /dev/nst1 fsf 1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag2header bs=1k   count=1
    dd if=/dev/nst1 of=/tmp/recover/bu-nbmedia00_1418641103_frag2 bs=256k

    Looking now at the files we have created :

    -rw-r--r-- 1 root root 1603796992 Dec 15 03:16 bu-nbmedia00_1418641103_frag1
    -rw-r--r-- 1 root root       1024 Dec 15 03:15 bu-nbmedia00_1418641103_frag1header
    -rw-r--r-- 1 root root 1002438656 Dec 15 03:22 bu-nbmedia00_1418641103_frag2
    -rw-r--r-- 1 root root       1024 Dec 15 03:22 bu-nbmedia00_1418641103_frag2header


    We have the two backup headers (one for each fragment) for reference if required, and the two files holding the actual data.

    We can then 'cat' the two files containing each fragment of the image into one file:

    cat bu-nbmedia00_1418641103_frag* >bu-backup

    And read this single tar file back, which contains the data from the backup.

    [root@nbmedia00 recover]# /usr/openv/netbackup/bin/tar tvf bu-backup
    Erwxr-xr-x root/root 103 Dec 15 02:58 2014 .AtTrIbUtEs.0
    --Extra header --
    drwxr-xr-x root/root   0 Dec 15 02:41 2014 /
    drwxr-xr-x root/root   0 Dec 15 02:49 2014 /testdata/
    -rw-r--r-- root/root 1532471296 Dec 15 02:50 2014 /testdata/tarfile1

  • Many thanks for the detailed instructions.

    Unfortunately it is connected to a Windows box, and we don't have a spare Linux box, or the hardware required to connect it up to the autoloader.  If anyone knows a way of doing it from Windows that'd be really helpful.  If not I'll look into getting something setup to follow the above instructions when I get chance but it's going to take a while.