Advise on long term archival solution

[RaymT___]

Hello All,

Need you advise on long term archival solution. currently my new client is manufacturing company, having 25 years retention of some daily manufacturing ndmp data and goes directly to LTO 5. monthly around 40-50TB data is archived to tape from ndmp filer and keep for 25 years retention. they started this retention since 10 years before with LTO 2, 4 now 5 and planning to for LTO7. since these many years they have huge numbers tapes were used and simply laydown. By keep on upgrading the library with higher capacity may help to reduce in count but its not good forecaste design(my opinion, please correct if wrong). migrating data from older tape to newer will also its heavy work as tons of tapes. I have proposed them to reconsider the retention period.still coustmer sticks to longer year retention. any other proposal can be share to them? any longer archival solutions?

Kindly share yours advise.

Thanks in advance.

[Marianne]

Firstly, backup should not be used as an archiving tool. 
Backup is meant for recovery from fairly recent backups.

We've seen here on this forum where someone tried to restore from the only backup copy made just over a year ago, but the tape turned out to be faulty.
There are more modern solutions for long term retention (LTR) such as build-your-own Veritas Access or the Access Appliance.  Access has built-in S3 connector where older data can be automatically migrated to private or public cloud.
NetBackup integrates seamlessly with Access.

A recognized archiving product such as Veritas Enterprise Vault should ideally be used to remove older data off primary storage, but still available for easy retrieval (unlike restore from backup...)

NetApp archiving is supported via a 3rd-party product.
Extract from EV Compatibility Guide: 
Enterprise Vault File System Archiving supports archiving of Linux, Unix and
NetApp NFS volumes using Vault Solutions Archive Accelerator for NFS. See
http://www.vault-solutions.com/products/archive-accelerator-nfs/.
EV too can use Access for back-end storage. 

Oh, I just realized that you did not say what type of NDMP.... Not sure if more types of NAS/NDMP storage is supported.

You may want to reach out to your local Veritas office or reseller to discuss possible solutions. 

[Michal_Mikulik1]

Hello,

here are my points:

- retention period is usually a combination of legal and business requirements, so it is probably undisputable. But maybe try to raise this question again, for example via your boss

- I dont consider NDMP-based backup as a good idea for a long-term archive. Any archive should be in format which is likely to be readable and restorable after many years, and this type of backup is not between them. It is tightly linked to a certain NDMP vendor and format version. Use NDMP for short-term backups only, but for archive, present and archive data as flat files.

- every month you archive new data, or you archive monthly new data plus everything which was archived before? 50TB monthly seem to be too much. So especially when there is a large ratio of duplicate data every month, I'd recommend you a real archiving solution. like Veritas Enterprise Vault  (which ,in the end, is able to archive data to tapes, too).

Regards

Michal

[Nicolai]

I agree with Marianne.

For archiving the customer should use a archive solution like Veritas enterprise vault (not Netbackup enterprise vault - two diffrent products). Archive solution provides functionality for indexing and searching - backup doesn't. Else it will be the famous needle in the haystack.

Plus NDMP backups are product dependant - meaning vendors of NDMP box A can't read vendors B tapes and vendor B can't read vendors A tapes*. So if customer wants to read tape 25 years old they need a NDMP box from the current vendor in 25 years time. And even if current vendor exist in 25 years they still need to support current format in 25 years time. 

* If NDMP devices write direct to tape.

Best Regards

Nicolai

[mph999]

Michael makes good points ....

Will the product you archive with be here in 25 years time ?

Lets say you have some Word documents  - will Word be here in 25 years time, probably, but are you sure.

Will it be able to open a word dox written 25 years ago - yes/ no ???  Would you bet on it ???

So, is there not an argumrnt, of convertimg everything to a format that will be available eg. word doc --> rich text format ?

Arguable, the best media for archiving is optical - DVD or similar.  They have a life exceeding i think, anything else.  2nd to this would have to be tape, written, stored (temp/ humidity) and handled correctly, it's very reliable.

Not a simple topic ...

[DPeaco]

I can't get into detailed specifics here but very careful thought and consideration should be strongly applied to carrying protected data forward for very long periods of time. From my own experiences, how do I try and carry forward over 60 PB's of data to the new solution. We aren't just talking about "backups" here, it's the "data recovery" piece that you truly need to be concerned with. Many times, we may be under U.S. federal regulations to make us keep data for up to 7 years for purposes of potential "litigation" should it ever arise. What about what it truly cost the company in resources, man-hours, and technical services and time to take data from LTO-4 tapes to a disk LZ and then backing it up again to LTO-7 tapes? How long would it take to do that across 22,000 tapes? What I can say is that it wasn't long ago that we were rotating over 65,000 tapes every month. 

What about a way to read those tapes if you didn't have NetBackup? Is the catalog recoverable? When's the last time you performed a D.R. on your own master / media and performed a restore from the DR solution? How long does the tape media remain readable? Way too many things to take into consideration in carrying "protected data" forward for 25 years.

I'm sorry, but this raises more questions than I would have answers for, but @Marianne and @Nicolai and @Michal_Mikulik1 all have very valid points that should very seriously be considered.

Remember this, it's not the backup that is important....it's the ability to recover the protected data from a given backup. :) 

[RaymT___]

Thanks Marrianne,

Last friday got chance to participate  Veritas Vision 2020. its very informative and new announcments.

There i heard about Veritas Access and veritas cloudpoints. Need to check with mgmt and sales person more on this.

---

Type of NDMP:

Netapp filer- NDMP Local -  Biweekly - differintail incr &  bi monthly - full  --> all to tape media.

[Genericus]

Can you check to see if the oldest backup you have is still readable? If it was created on a different filer type, it may not be!

That would be your best argument for making the change.

IMHO, you should argue that the data retention should be ON THE FILER, or on an active archive, so the data is readable.

You have set yourself up to fail in so many ways, OS, application and hardware changes all combine to make older data almost guarenteed to be unusable.

NetBackup is for disaster recovery, to restore your systems if they crash or are destroyed; or for dumb-asster recovery, when your user deletes the wrong data; or maybe recovery after malware or software attack. It should not be used for archiving.

[RaymT___]

Thanks Genericus,

I havent tested those older tapes as its very huge,(one man army in my client side :) ). its mixed of LTO 2, 4, 5

However i have passed the Veritas ACCESS archival solution for mgmt consideration for new data.

Migrating older tape data still under discussion.

Thanks everybody who advised here and its really helpful information.