Can NBU auto erase tape after data expires?

[FlyMountain]

Can NBU auto erase tape after data expires?

I couldn't find this option after searched around. It will help us greatly if this is possible. We are using third party virutal tape library and tight on space, will have to manually erase data on scratch tapes to reclaim space on VTL after some time.

[RonCaplinger]

If your device is acting as a VTL, I would either expect it to automatically delete the data on disk, or require a disk space reclamation job (such as on Data Domains, where you run the "Clean" process), which can be run manually or automatically at specified intervals.  Otherwise, it isn't really acting as a VTL, because tape data doesn't normally get physically erased; the backup application's database simply removes the entries related to that particular file.

[FlyMountain]

RonCaplinger,

Thanks for your reply. I remember TSM has an option to erase data when move tapes into scratch pool. My VTL will only reclaim space when index tape (after de-duplicated) erased. This is why we have to have manual erase step involved.

On NBU scratch tape, i did see "long erase" and "quick erase" options available. But i am not sure if we have similar feature as TSM has.

[Marianne]

Your problem is not with NBU - NBU knows the tapes can be overwritten and does not need tapes to be cleared in order to overwrite. Because this not a NBU requirement, it does not have any built-in automation for this process.

There is no need to erase the tapes - a quick 'label' will do it.

I have seen customers either writing scripts (basically running a list of media-id's through a 'for' loop doing a bplabel on them) or just select the expired tapes in the GUI, right-click and label them.

[FlyMountain]

That is exactly what we are manually doing now.

If NBU can "auto-label" the scratched tapes. It will be a great feature for us. For physical tapes, it is not necessary to "delete" data before we can overwrite it. But on disk, it will be an issue if we don't delete data after it expires. Disk will become full and not allow more wirte. I believe there is no way VTL can do the "auto-clean" work since they don't perform and trace backups, at least on FalconStor. 

[Mark_Solutions]

I think you will find that is common to all VTLs

You need to plan your retnetion period of the backups carefully so that you do not fill it up

By choosing how many tapes are available and of what capacity you can restrict the used capacity of the VTL

Without this planning it may well fill up

If you are in such a position then you doo need to ensure your duplications (assuming you are duplicating to "real" tape from the VTL) are up to date and then expire your oldest images.

This tends to be the safest way of doing it.

So search the catalog for images more than say a week old - dump this list (Highlight them all and CTL-C then paste to a text file) and run a bpexpdate against those images (remembering to specify copy 1 (if the VTL is the first backup copy) or if the query in the catalog was against Copy1 just right click and select "Expire" (though this would be best in batches of no more than 100 at a time from the Catalog section of the GUI

I have used a basic script in the past that works through the text file to run the bpexpdate command against the list - with a -d 0 for immediate expiry

Next run a bpexpdate -deassignempty -force

This will "empty" all tapes that have been expired so that NetBackup can start using them again

The real key to this though is not to get in that position in the first place - it is the same as if you were using disk (without any staging and had to set the retention period to be just right so that things expired before the disk filled up so that backups could always run

Hope this helps

[Marianne]

I fully agree with Mark.

I have always had a problem with VTL - why make disk look like tape?

NBU knows how to use disk storage - basic and advanced. Image files on disk are deleted when they expire.

NBU know how to use tape - no need to delete, label or erase. It can simply overwrite from scratch. Since VTL presents itself as tape - that is how NBU will treat it.

The compatibility guide (http://www.symantec.com/docs/TECH76495 ) contains the following notes in the VTL section:

9. Filling up the available disk space on a virtual device may cause unpredictable behavior of the device. Working with the device vendor may be required to recover from errors that result from a full disk.

12. When NetBackup expires a virtual volume, the virtual device is not aware that the data is no longer needed. Because of this, the virtual device may appear to be much closer to capacity than it really is. When NetBackup subsequently begins writing to that virtual volume at the beginning of the volume, the virtual library will then be aware that the rest of the data is no longer needed and can free up the disk space.

[FlyMountain]

Actually we are doing fine with VTL, we have 50:1 block leve de-duplication ratio, it means we do save a lot of space, we don't need to recall tapes from IronMountain if restore will be needed, no need for a team to do tape rotation in our remote sites ..... : )

only one thing brother us is the space reclaimation on VTL, we will need a manual run of "quick erase" on scratch tapes to make the VTL reclaimation really works. Another way we figured out is we can strictly limit the scratch tape amount. but it is hard to do as the data growing....

Thanks for all your sharing. If there is no easy way on NBU can do "auto-erase". we will talk to our VTL vendor or keep our weekly manual work to make us have something to do....

happy new year

[Mark_Solutions]

You should not need to do any auto erase

As we have said if you have configured the number of tapes correctly then once they have expired in NetBackup they will become available for use again.

When this happens NetBackup can overwrite them which has the same effect as an erase.

As long as you retention periods are correct and the number and capacity of tapes available is correct then NetBackup will do the rest for you.

If you need to erase tapes then something is wrong as NetBAckup should be overwriting tapes once they have expired

[Will_Restore]

We had that product some years ago and it did not play well with NetBackup.  Don't miss it at all.

[FlyMountain]

Guys, thanks for more input.

We have the latest FalconStor VTL 7.0.

Sorry i didn't explain detail enough in the first time. The VTL allows "thin provision" and runs de-duplication automatically. If we can have 50:1 de-duplication ratio, we can store 100TB data into 2TB capacity VTL. But it will depend on the successful de-duplication and reclamation jobs to free up space on time. Most of time, we will need to "erase" tape data on scratch tapes before re-write, otherwise we may have space issue. If we use physical tapes, we will never have this issue.

[Mark_Solutions]

So does NetBackup overwriting the tape actually do the same thing as an erase for you?

If it does not then I guess you need to do the erases

If it does and you are erasing tapes in the scratch pool (indicating that they have expired) then as i said earlier your design is not quite right as you have too many tapes - you need to keep the ones in scratch to a minimum so that they are regularly used

[Will_Restore]

"FalconStor VTL OpenStorage Option bridges the gap between disk and NetBackup, letting you mix storage arrays in the same environment, allowing you to use whatever devices meet your performance and budget needs. For example, you can use FC disks at your primary site, while using lower-cost SATA drives in your DR site.

The OpenStorage system further enhances disk utilization by making it easy to reclaim disk space that is no longer needed. Unlike a tape model, where all images on a tape must be expired before the tape can be deleted or scratched, OpenStorage lets you target individual backup images for deletion independently. Disk space utilization is readily displayed in the NetBackup console. "

http://www.falconstor.com/dmdocuments/VTLOpenStorageSB.pdf

[FlyMountain]

Tape erase on NBU will free up occupied space on VTL immeadately, overwrite will do the same only when it happens. The problem is we will to keep certain free space on VTL, either to limit the scratch tape number or run erase manually.

FalconStor Open Storge Option will be our next step. As i know, FalconStor will still emulate virtual tapes on VTL even on the big disk will be seen on NBU.

BTW, what is disk behavor on NBU side, will expired data be deleted right way?

[Will_Restore]

NetBackup image cleanup runs every 12 hours

[Mark_Solutions · Accepted solution]

I guess the easy way then is to have a script that runs on the Master and schedule it using Windows scheduler to run every day - try this one to see what you think - name it autoerase.bat and drop it on the C drive (or anywhere else as long as you edit this appropriately) add the correct paths etc, (in italics) according to your setup - if on unix i am sure you can amend it to suit:

<installpath>\volmgr\bin\vmquery -pn scratchpool -l>c:\scratchlist.txt

for /f %%a in (scratchlist.txt) do <installpath>\netbackup\bin\admincmd\bplabel -m %%a -d hcart -p scratch_pool -host mediaserver -erase

Hope this helps

[RonCaplinger]

If you are using standard non-dedupe disk appliances that don't have to run a "cleanup" or reclaimation job, then as soon as NBU expires the image, the space is available.

However, if you use a dedupe appliance, such as Data Domain (and I assume NetApp), the exipration in NetBackup doesn't actually free up the physical space, much like what is seen with the Falsonstor mentioned here.

In order for the disk space to be made available again after images expire, Data Domains have a Disk Cleanup process scheduled on each appliance and can run many times a week, if necessary.  We have ours set to run weekly.  This means that on the day after cleaning, we may have 40% of the space used.  The next day, even after images expire and new images are written, the disk usage goes up to, say, 47%.  Third day, images expire and new ones are written, but disk usage goes up to 62%.  And so on, until the next Disk Cleanup is run on the Data Domain the following week, and it typically goes down to 40-42% again.

We have times where we have hit 98% usage on a Data Domain because we are storing too much data on it; we then cut back how long we retain data on that device from, say, 1 month to just one week.