Forum Discussion

Biolli's avatar
Biolli
Level 2
6 years ago

Automated Backup Integrity check for Data stored on LTO Tapes

Hi everybody!

Does anybody has a solution for automated , e.g. script based, third party software ..., solution allowing to check stored data on LTO tapes e.g. by comparisson of md5 checksums.

I could not find any information/discussion on this topic and I am wondering if nobody does has this requirement?!

How do you proof that your backups made 10 years ago are still 100% consistent?

Would be nice to get some support from experienced backup specialists.

Regards

 

  • sdo's avatar
    sdo
    Moderator
    6 years ago

    The only way to prove that the backups are readable is to read them.  The "verify" function of NetBackup does not read the backups, as the verify function only verifies the catalog meta-data.

    So, there are only two ways to read the backup data:   1) restore everything to somewhere  or  2) duplicate everything to somewhere.

    option 1) is the most comprehensive because it:  a) confirms that the NetBackup custom tar format is intact and that the tape is intact, and also:   b) confirms that a current version of client and current version of OS and current version of file-system are altogether all able to receive the old data from old versions of client+OS+file-system and write that data to new versions of client+OS+file-system.

    option 2) just proves that the bits-n-bytes of the NetBackup tar format are still readable and that the LTO tape is intact - and does NOT prove that a restore is actually possible for client+OS+file-system interop compatibility of old systems (the backup) versus new systems (the restore).

    N.B:  Your options narrow even further if you are considering for example restoring MS Exchange mail-store database from 10 years ago - if so then you will need the same client+Exchange+OS+file-system version interop for your restore target.  So, you do still have an MS Exchange 2003 running on Windows 2003 running NetBackup Client v5.1 - you do right ?

    • Biolli's avatar
      Biolli
      Level 2
      6 years ago
      thanks for your answer.
      We are running netbackup to backup various production systems.
      In a first step we want to proof if our tapes are still ok (option 2). We store always 2 tapes. So in case of detected data failure, we want to duplicate the duplicate if this is hopefully still good.
      My question was how can I automatically perform this check. I guess only by comparison of initial md5 with md5 from a duplicate that we have to make periodically, as you mentioned? An how to do this? Does netbackup has this function implemented?
      • sdo's avatar
        sdo
        Moderator
        6 years ago

        Ok - so the question is now expanded or enhanced.  I have to ask, is there yet more detail, more requirements, that you could add - more that you haven't yet been able to describe to us?

        My first reaction is to ask... why is this "double tape double check" only now a business requirement, after a whole ten years of keeping dual copies?  To me it seems that the original decision to keep two copies of all tapes was for exactly the specific reason of avoiding having to confirm tape media readability and tape media reliability.  But now, even having in the past taken two copies to tape, this is now not seen as potentially safe enough.  Maybe now really is the time to leave tape and move everything back to a reliable disk platform.  I mean, you already have two copies of all backups on tape, and yet for some new reason you now need to check the consistency of both of those copies, but I can't help but think that if the backup data was already that important then surely it would already be on replicated disk.

        Anyway, AFAIK, there are are no automatic tools for complete and all encompassing yet still tunable and selective fully automated selective and adaptable tape media reliability testing.  And I suspect that none of the enterprise backup products have such a feature either.  So, you need to write your own tools, or engage a third party who can adapt or retool their software.

        AFAIK NetBackup doesn't use MD5 on tape.  I believe that LTO has it's own embedded CRC (or other checksum) type checks, but this is abstracted at the tape head and hidden from all enterprise backup applications.  i.e. if the tape is readable then it has passed its own embedded LTO based reliability check.

        NetBackup can definitely help you achieve your goals but it's quite clear that this won't be in the way that you currently imagine.  I think you need to engage an accomplished channel partner to help you translate your new (never previosuly designed into deliverable system) legal and business requirements in to actual product capabilities combined with some custom coding (custom script code and/or custom compiled code).

  • sdo's avatar
    sdo
    Moderator
    6 years ago

    Automated restore testing for everything would be quite hard to do.  Thinking about just file-system data, let alone database agent data, then you'd need to identify the backup image contents, the sizes, and complex embedded inline (i.e. backup image content) changes of volume names, volume letters, paths, folder structures.  i.e. you would need to walk the backup image listing looking for change in root / MFD / top-folder path name, and total up via those "path trees" and use this to feed your scripted automated restore tests.  But, this wouldn't prove system state or BMR, it would only prove the restore of NTFS or EXT2/3/4 folder structures.

    I doubt very much whether you could automate BMR physical server recovery testing.  Instead you could script perhaps B2V to VMware - this might be possible in scripting.

    Instead of restore testing, you could write some scripts to duplicate say the oldest 'n' years starting at for example year 6 then year 7 then 8 etc... or perhaps by LTO generation, e.g. duplicate all your LTO4 to LTO8 - assuming you still have the old tape drives - this at least proves that the old backup images are intact and also protects the old data by placing a copy on new media which is less likely to degrade so quickly - and also reduces your tape count significantly.  So, you could write your own duplication engine, which schedules itself, and re-tries itself, and handles being stopped for server reboots, and simply keeps running until the job is done.  I can vouch from experience that this won't be a small simple script, instead you will end up writing what will effectively become a small custom stand-alone application.

    • sdo's avatar
      sdo
      Moderator
      6 years ago

      Not that I've ever looked that hard, but do any of the other enterprise backup platforms have embedded automated restore testing?