Why verify?

[sdo]

Hi Forum,

Why does page 222 of the BE 2012 Admin Guide recommend that verify be enabled for all backup jobs?

AFAIK, documentation for NetBackup does not make any similar recommendation.

I could use either BE or NBU to save backups to the same HA disk targets, or the same LTO drives and media - yet one product recommends that verify is used.

Thanks.

[VJware]

"Verify" ensures the tape or the disk can be read and/or restored from the media. During a "Verify" a CRC verification is done to ensure that the data was written properly and can be read as well during restore.

Hence, we recommend to run verify for all backups, but then again, best way to test your backups is by running restores.

[sdo]

Thanks for that. But I still don't understand why verify is recommended for one product (BE), but the other product (NBU) doesn't even have the feature of verify as part of backup jobs (verifies can only be run manually). Could this be construed as there being less faith in the quality of backups performed by BE?

[VJware · Accepted solution]

I am unsure as to (if) and why is not recommended for NBU. Though If i glance at the NBU documentation vs the BE one, NBU is pretty vast and I don't see any such recommendations for other things either :)

In BE, verify can be set to run separately as well. It does not need to run always immediately after a backup job completion. This ways, BE is offering you flexibility as to when you would like the verify to run. Secondly, by adding it as an option in the backup job properties, it reminds the administrator to "verify" his backups. If the "verify" isn't included as an option in the backup job properties & has to be configured, run manually (like NBU), I wonder how many administrators would even know of this "verify" and how many would even remember running it ?

And not just BE, Almost all major backup vendors do have some of data integrity, verification check. I personally wouldn't construe as these backup vendors providing poor quality of backups.

[sdo]

Ok, I'm aware where verify can run in BE, and the options.

I'm still not getting why verify is recommended.  I could use either NetBackup or BE to backup through the same HA server over the same HA SAN to the same HA disk storage and duplicate to the same HA tape - yet one product steers admins towards always running a verify, and the other does not.  I've worked with NetBackup for a long time, and most shops simply don't bother with running verifies - it's just too time consuming.

With NetBackup it's not a case of admin's having to remember to run verifies, it's more a case of they just don't - because when you're running backups for 5,000+ servers and VMs , then no one bothers with verifies (be it for data saved to disk or tape).  You might do a few occaisionally as part of a QC excercise, but in my experience, NetBackup admins usually don't run verifies for everything (not in the shops that I've seen).

Also, it's worth remembering that LTO tape drives have a built-in-read-after-wite-check at the tape head, and if the immediate-read-after-write fails, then the tape simply carries on moving on and the data is written again.

IMO, the situation should be that both BE and NBU recommend verify (and the NBU developers get busy amending the product to make this a feature of backup/duplication/SLP scheduling - and update docs and TNs) - or neither documentation should recommend it.

I get that flakey old tape technology needed a verify, or data saved to tape in dirty/dusty environments probably needs a verify - but I don't really see a pressing need to verify backups/duplications written to tape in clean data centres - and I certainly don't see a need to ever perform a verify of any data that has been saved to HA disk (be it local RAID disk, SAN FC/iSCSI disk arrays, VTL, or de-dupe, or OST, or cloud).

Anyone else care to comment?

[pkh · Accepted solution]

I remember this topic came up before and it was stated that Symantec would not be responsible if you are unable to restore from an unverified backup set.  This needs to be verified if it still holds true.

[sdo]

VJware, pkh - thanks for taking the time to respond.

That last comment re Symantec not being responsible for customers being unable to restore unverified BE backup data seems completely reasonable to me.  IMO, as a NetBackup admin, the same holds true for data backed-up by NetBackup - I mean, it would seem unreasoanle of me to hold Symantec responsble if I were unable to restore some data (using NetBackup) due to faulty media.

But then it would also seem unreasonable to hold Symantec responsble if customers were unable to restore backed-up data that had been previously verified for which root cause later proved to be due to faulty/aged media.  Again, IMO, I see both products (BE and NBU) as stable and mature, and so notwithstanding bugs that prevent restore, I see no reason to have less faith in one product over any other, in terms of readability of backups if both products were installed in the same site/location.

I guess the continued stance of recommending verify for BE must be down to perhaps typical installed location.  BE has tended, and to some degree continues, to be installed in smaller sites, offices, departmental areas and comms rooms that are sometimes dirtier/dustier than the typically cleaner large enterprise data centre machine rooms.

My take on this all would be to expect BE, which is installed in cleaner environments, to perform just as well as NetBackup - in terms of readability of backups, in the same manner as NetBackup - and to have the same expectations and reservations.

And so - as long as I have faith in the environmental aspects - i.e. quality of hardware/software, sound installations, well engineered and managed solutions, and good quality matched media and good handling/storage practices - basically all of the characteristics that one woudl expect in an enterprise environment - then I would be of the opinion that verifying all backups for BE would seem un-ncessary.

Seeing as how it is much easier to verify backed-up data using BE (automatically) than in NetBackup (manually), then as a BE admin I would probably verify just a few of the more important, but smaller, backups on a regular basis.

[Colin_Weaver · Accepted solution]

Interesting reasoning. I believe this recommendation was brought in when the reliability of tape technology and how clean the environments the servers ran in was questionable (for instance I can remember working on tape drives located under desks next to open windows that Diesel Locomotives passed by, IT equipment in such environments often ended up 'sticky' and unreliable)

In theory servers are usually stored in better / cleaner environments and tape techology  has also improved in that time as such (and this is a personal opinion and not an official Symantec view) it's possible it may no longer be as critical a suggestion

Saying this however that recommendation remains in the product, even to the point of the product warning you when you disable it, which does mean it has to be your decision on whether to run verifies or not.

Of course industry best practice is that you should also do periodic test restores which is all part of maintaining confidence in your backup strategy

[Colin_Weaver · Accepted solution]

BTW NBU writes to tape with TAR format, BE writes to tape with MTF which might be part of the reasoning for differences in operation and recommended practice.

[sdo]

Thank you Colin.