Forum Discussion

smith_jones's avatar
smith_jones
Level 6
9 years ago

difference in image expiration, erase, quick erase, long erase and format

Hi,

bpexpdate - changes the expiration date of the images in a tape. As per me if an image gets expired (or if -d 0 was selected in bpexpdate then all the images in that tape expires), that means data is still there and physically it is not deleted from the tape. Say if all the images are expired then when next backup will run for that pool the data in this tape will get overwritten. Am I right? And if not then what exactly expiration is doing with the data?

erase - If you do not need contents stored on tape, you can erase tapes. Erase will be enabled if the tape has no valid image and the tape is deassigned.

Short erase (fast) — use this option to speed up the erase process. The short erase operation does not physically erase data written on the tape. It simply loads the tape to the drive and wipes the tape header.

Quick erase writes an indicator at the beginning of the media and the contents of the media are made inaccessible. I did  not get this.

Long erase (slow) – use this option to clear all data written to tape. The long erase operation loads the tape to the drive, rewinds the tape and physically erases all data written to the tape. I understood this.

Is formatting and erase same here? Or they are different?

Or can somebody explain the difference in leyman. 

Thanks.

7.7
Backup and Recovery
NetBackup
Windows Server (2003-2008)
  • mph999's avatar
    mph999
    9 years ago

    Say if all the images are expired then when next backup will run for that pool the data in this tape will get overwritten. Am I right? And if not then what exactly expiration is doing with the data?

    Yes - this is 100% correct

  • mph999's avatar
    mph999
    9 years ago

    I should add.

    NBDB is one big database ... but we talk about 'volume database' and 'media database' as well (but this info is all in NBDB).

    vmquery -a command - shows volume DB stuff, mainly physical, robot, slot number in robot, density etc ...

    bpmedialist  command shows 'media DB stuff' and this relates to images (eg. number of images on tape).

    If a tape shows in the media section of the GUI (or vmquery output) it is in the volume DB and the only way to remove it would be to 'delete' the tape from NBU (only possible if there are no valid images on the tape.

    When a tape is selected for backup, it is given an assign time and then appears in the bpmedialist command - and we then say 'it's in the media database).

    A tape should only have one assign time, it keeps this, until the last image expires - at which point the tape is marked to expire, but remains in the media DB for a short time, until the 12 hr image cleanup runs (bpimage -delete_all_expired), or bptm -deassignempty runs (every 10 mins I think - but does not show in activity monitor, but does in the bprm log).  (Hopefully I got those command options the right way round, but you get the idea).  THe tape then loses its assign time, and will no longer appear in the bpmedialist output.

    The tape expire time should be equal or maybe just after the expire time of the last image, image expire and tape expire are in fact seperate, although as they are usually in sync, they look the same, they are not.  If a tape expires before the images, NBU thinks the images still exsist, but in fact the tape could be reused.

    This should not happen, but can - and is one of the things nbcc looks for and warns about - though is fairly rare and can have various different causes.

    If a tape should get overwritten, then the old images will still exsist in the catalog.  The new image(s) will also exsist, but will have a more recent assign time.  The tape media id, and assign time are part of the image fragement table, and is how NBCC spots overwritten tapes, if a tape has two assign times in NBDB, then the images relating to the earliest assign time have, for 100% certainty, been overwritten.

    Generally speaking, it's safe to say that if a tape has an assign time it has valid images.

     

  • mph999's avatar
    mph999
    9 years ago

    OK, if you have this ...  | = filemark

    <PHYSICAL BOT> <MEDIA HEADER> |<BACKUP HEADER> <DATA> <DATA>| <BACKUP HEADER> <DATA> <EMPTY HEADER>  <LOGICAL EOT>            <PHY EOT>

    After the relabel ...

    <PHYSICAL BOT> <NEW MEDIA HEADER> | <LOGICAL EOT> <MEDIA HEADER> |<BACKUP HEADER> <DATA> <DATA>| <BACKUP HEADER> <DATA> <EMPTY HEADER> <LOG EOT>   <PHY EOT>

    ...the new media header overwrites the old, the a LOG EOT and then whatever was there before is still there ... but the drive won't go past the first LOG EOT.

  • mph999's avatar
    mph999
    Level 6
    9 years ago

    image expiration, erase, quick erase, long erase and format

    Image Expiration - NBU removes the image information from the catalog.   If disk image an entry is made in the deleted frag table in NBDB and nbdelete uses this to know which fragments to delete off the disk.  If the image is the last image to expire on tape, the tape is marked to be expired and the next image cleanup will 'expire' the tape / or bptm -deleteallexpired will which runs 'invisibly' every 10 mins.

    But - Image expiration only removes the entries from the catalog.  The removal of fragments from disk / tape expire are seperate steps, but all the steps are part of the 12hr cleanup.  It is in fact possible to expire an image and leave the fragments on the disk (bpexpdate -nodelete) - here, no entries are made in the deleted frag table, and therefore when nbdelete is next run, it doesn't delete anything, and therefore your disk image could be reimported.  Running bpexpdate (without -nodelete) causes nbdelete to run as well, so although it looks like image expiration deletes images of media (well, at least disk) they are in fact seperate steps, as I have hopefully explained above).

    Quick erase - exactly the same as a re-label (bplabel command).  This rewrites the 1k media header to the tape and then writes a new  end-of-data mark.  A 'normal' tape drive will not go past the e-o-d mark making the data on the tape unrecoverable.  The data however, is still on the tape  (though there is a chance the re-label wrote into the first few bytes of the image - no way to say that a relabel will be in the exect same part of the tape).   If you have a tape drive with 'modified firmware' and some serious expertise, you could get the data back - this is how a data recovery firm would recover data.

    Long erase - well this is one big question mark.  A long erase is started from NBU by sending the 'erase' scsi command to the drive.  Exactly what then happens is upto the drive manufacturer.

    I have it on very good authority, that some brands of LTO drive do not erase the complete tape on a long erase, only some of the tracks.  This was a few years back, so maybe things have changed.

    We don't 'format' tapes in NBU - and in fact I don't hear that term very often, but would personally take it to mean a re-label.

  • mph999's avatar
    mph999
    Level 6
    9 years ago

    Say if all the images are expired then when next backup will run for that pool the data in this tape will get overwritten. Am I right? And if not then what exactly expiration is doing with the data?

    Yes - this is 100% correct

  • mph999's avatar
    mph999
    Level 6
    9 years ago

    I should add.

    NBDB is one big database ... but we talk about 'volume database' and 'media database' as well (but this info is all in NBDB).

    vmquery -a command - shows volume DB stuff, mainly physical, robot, slot number in robot, density etc ...

    bpmedialist  command shows 'media DB stuff' and this relates to images (eg. number of images on tape).

    If a tape shows in the media section of the GUI (or vmquery output) it is in the volume DB and the only way to remove it would be to 'delete' the tape from NBU (only possible if there are no valid images on the tape.

    When a tape is selected for backup, it is given an assign time and then appears in the bpmedialist command - and we then say 'it's in the media database).

    A tape should only have one assign time, it keeps this, until the last image expires - at which point the tape is marked to expire, but remains in the media DB for a short time, until the 12 hr image cleanup runs (bpimage -delete_all_expired), or bptm -deassignempty runs (every 10 mins I think - but does not show in activity monitor, but does in the bprm log).  (Hopefully I got those command options the right way round, but you get the idea).  THe tape then loses its assign time, and will no longer appear in the bpmedialist output.

    The tape expire time should be equal or maybe just after the expire time of the last image, image expire and tape expire are in fact seperate, although as they are usually in sync, they look the same, they are not.  If a tape expires before the images, NBU thinks the images still exsist, but in fact the tape could be reused.

    This should not happen, but can - and is one of the things nbcc looks for and warns about - though is fairly rare and can have various different causes.

    If a tape should get overwritten, then the old images will still exsist in the catalog.  The new image(s) will also exsist, but will have a more recent assign time.  The tape media id, and assign time are part of the image fragement table, and is how NBCC spots overwritten tapes, if a tape has two assign times in NBDB, then the images relating to the earliest assign time have, for 100% certainty, been overwritten.

    Generally speaking, it's safe to say that if a tape has an assign time it has valid images.

     

  • smith_jones's avatar
    smith_jones
    Level 6
    9 years ago

    Hi Martin,

    Really thanks for this great explanation. I have one question here. You wrote -

    Quick erase - exactly the same as a re-label (bplabel command).  This rewrites the 1k media header to the tape and then writes a new  end-of-data mark.  A 'normal' tape drive will not go past the e-o-d mark making the data on the tape unrecoverable.  The data however, is still on the tape  (though there is a chance the re-label wrote into the first few bytes of the image - no way to say that a relabel will be in the exect same part of the tape).   If you have a tape drive with 'modified firmware' and some serious expertise, you could get the data back - this is how a data recovery firm would recover data.

    Say this is the tape before quick erase :

    BOT | 1K HEADER | DATA | DATA | DATA | DATA | EMPTY SPACE | EOT

    Till now that I understood is, after quick erase this will happen -

    BOT | NEW 1K HEADER | DATA | DATA | EOT | DATA ( NOW UNACCESSIBLE SINCE EOT HAS BEEN MARKED BEFORE IT) | EMPTY SPACE 

     

    ?

  • mph999's avatar
    mph999
    Level 6
    9 years ago

    OK, if you have this ...  | = filemark

    <PHYSICAL BOT> <MEDIA HEADER> |<BACKUP HEADER> <DATA> <DATA>| <BACKUP HEADER> <DATA> <EMPTY HEADER>  <LOGICAL EOT>            <PHY EOT>

    After the relabel ...

    <PHYSICAL BOT> <NEW MEDIA HEADER> | <LOGICAL EOT> <MEDIA HEADER> |<BACKUP HEADER> <DATA> <DATA>| <BACKUP HEADER> <DATA> <EMPTY HEADER> <LOG EOT>   <PHY EOT>

    ...the new media header overwrites the old, the a LOG EOT and then whatever was there before is still there ... but the drive won't go past the first LOG EOT.