01-29-2020 12:25 PM - edited 01-29-2020 12:33 PM
My library is 400 slots so its not small. However, there is this policy that infinity archives data from a directory tree that has been going for over 10 years.
We move batches of live data to this directory tree. We then backup the directory tree to tape and delete the data. Then we move another batch of live data into the directory tree. We then back that up to tape. Been doing this for a really long time. All of this directory tree lives on tape only.
The directory tree is structured like this:
The 3 letters are customer codes. We have a couple hundred customer codes Then within each customer code is a date folder(5-10 years per customer), then within the year folders are several sub folder layers of data
When we go to do a restore we see a view of the DirectoryTree and all the customer codes and all the years in that catalog restore browser.
The problem is that I have tried to do a test restore of the entire directory tree and the restore asked for several thousand tapes. Our library is 400 slots. So this seems actually totally infeasible. With my older version of Netbackup a restore job fails if every tape it requires is not available to load when it needs it.
What was also odd was even just selecting a single customer code or even a single year under a customer code...Netbackup still was askig for the same amount of tapes to do the restore (several thousand).
Is that because all of these backups share the same directory tree over many years?
That is actually what is useful for us. In that we can't have all this content media on disk, it only lives offline on tape. So we rely on the Netbackup UI to browse this large content media library that covers all customers over all years. If we started using seperate policies for every 3 letter customer code, then we couldnt browse the restore catalog...we would have to select and browse a couple hundred different policies to view what content is under each policy. Whether it be by customer code or by year. Still breaking it up by customer code doesnt solve the problem. Even that customer code will eventually grow to requiring just too many tapes just like the current DirectoryTree has
So i am stepping back realising there must be some kind of design flaw here. Alot of places toss data after 7 years, but by design and for insurance purposes we 100% have to inifinity keep on tape this never ending growth of data on tape for this client content media
Have other people encountered this sort of thing where a really long term archive that lives only on tape media can simply outgrow the practicality of restore?
I see 2 parts to this
1) Restoring the entire archive from tape because of a site disaster. Yup its going to be thousands of tapes and even with 400 slot library it will take several reloads. But will that even work where we could serially load 400 tapes at a time, let Netbackup do its thing for a few days until its done with the first 400 and load 400 more, rinse repeat?. The potential problem I see is the restore itself requires all of the thousands of tape. I have no idea if the list of tapes it gives me are in order of what it will try to restore? They just seem to be in ascending tape number order. I suspect the library will need to jump around in what it is restoring potentially requiring us to re-insert the same tapes over and over again because we can only support 400 concurrent tapes?
2) Restoring one off files or ten off files where we know what we want we just have to hunt down and select the files we want restored from within the restore gui. This "should" be more realistic and only require a handful of tapes. Is there any explanation why even this may result in Netbackup thinking I need ALL the tapes for that directory tree to even do a handful of files?
Any help or shared experiences others have had at this scale would be appreciated
01-29-2020 01:40 PM - edited 01-29-2020 01:41 PM
Is the restore actually asking for all those tapes, or is it just the 'preview' button that shows them?
In my experience the 'preview' button can show me the dozen+ tapes involved in a backup, yet when I submit a restore for a filesystem or set of files it then only mounts the tapes it really needs to position to the data requested.
01-29-2020 01:54 PM
If you're actually processing a restore and it requests 100 tapes, it's not going to use all those tapes simultaneously, it will ask for them in sequence (restore will only use 1 drive at a time).
So, if you can only fit 10 in your library, you insert the first 10, wait for the restore to process them and then it will WAIT when it gets to needing the next volume. At which point you can remove the first 10 and add the next 10 and the restore will resume where it left off.
Your post says you're on 7.x; as far as I recall, 7.x will not fail if all the tapes required are not available.
Additionally, how were your backups run? Fulls? Differentials?
What are you selecting for date range/image selection in the Restore console? If you're spanning years of backups and they were all Differential, then yeah, it's gonna ask for a LOT of tapes.
01-30-2020 06:59 AM
Please share what schedules you have configured for the directory tree
e.g.: full, incremental or diff - and what are the frequencies.
As being mentioned before - if tapes are missing upon a restore request , you will have a "pending action" in the device monitor with the type of "Read".See documentation link:
01-30-2020 10:22 AM
Yes it was based off the Preview where it (I thought) tells me what tapes I need for the restore I configured. You are saying that is not accurate? What is it showing in that case?
Is there a way to get the actual restore tapes without triggering a restore?
01-30-2020 10:31 AM
All fulls with infinity retention. Every time we do a backup of that Directory Tree its with all new data files but the same tree folder structure "Customer code", "year", etc
The version is actually 6.x. Does that explain why the restore job fails if it tries to access a tape it doesnt have in the library? I was shocked when this happened, I thought for sure that it would go into waiting state and ask for that media id.
The first level of the directory tree is three letter customer codes, the second level is the year, with several levels below it.
So if it restores the tree from top down, its going to hit the first customer code and the first year of that customer that is perhaps 2020. Then it will hit 2019 and 2018 and down many many more years. Then it will have finished that first customer code and it will go to the second customer code and again hit years 2020 and 2019 and 2018 etc
The tapes were written chronologically so the same set of tapes likey encompasses one year.
Do you see my concern that the restore could ask for the same tapes over and over again as it hit older or newer years for each diferent customer code?
I dont think the restore is going to be super smart in how it does the restore or will it so that we know for sure we will only have to insert each tape it wants once?
If its just going to restore the tree from top down...then I think this goes back to my initial design flaw in our strategy. It's onerous enough to have to deal with thousands of tapes to do a full restore of the Directory Tree, but if its this endless cycle or ejecting and re-inserting the same thousands of tapes I think this quickly falls into the not feasible category and we may need to redesign how we store this data in the Directory Tree
I guess the question is what is the restore order on Netbackup when doing a total Directory Tree restore? Is there a way t control how it decides to iterate over a massive Directory Tree?
01-30-2020 10:34 AM
It is all fulls and there is no schedule. Every backup is a manual backup
We copy live data that we have recieved into this Directory Tree where it is sorted into the Directory Tree based on three letter customer code/Year/Other media related field/Other media related field/etc
We then backup that whole Directory Tree to tape with an infinity full. Then we wipe out the data in the directory and copy a different new batch of data from the live data and populate te Directory Again, do another inifinity full....rinse repeat
01-30-2020 12:28 PM
I have seen cases where a STANDARD backup of a server - lets say - spanned 6 tapes. If I select one directory to restore and then click PREVIEW if will show *all* the tapes that make up that backup.
But when I actually submitted the restore it only mounted the tape(s) that it needed to for that directory. Apparently Netbackup keeps track of which tape and which file number of the tape to position to.
I dont know of a way to show only what will *really* be needed.
01-30-2020 03:22 PM
Playing the devil's advocate here - what is the practical use of attempting to restore the entire directory tree. Do you have sufficient storage to enable the restore to complete?
A more useful solution would be to target one or more selected customers or years which would reduce the number of tapes required for starters, and you would also be able to complete the restore without filling the storage.
02-03-2020 06:32 AM - edited 02-03-2020 06:35 AM
You might be able to see which tapes are *really* used by starting off a job (even if it fails). The start of the job process will list all the media it will request. I believe this is a different output than the 'preview' output?
I'd still like to know exactly what you're selecting on the Backup, Archive & Restore screen?
If you're only picking 1 backup, then the restore should only request the tapes used for that specific backup. If you're selecting multiple backups, then it will likely ask for a lot more tapes. Unless you're deleting files between backups, then you should only need to restore from 1 backup.
EDIT: Ah, I see, all new files in each structure for every backup but the structure is the same. That should still be fine; I would say to shorten your restore selections. You may have to run multiple restores, but that will be fine.