Mash with HP Storeonce VTL and Netbackup 7.1 not deleting oldt expired tapes

[sasa_m]

Hi there

I have a customer who has a mash of things in backup environment and needs me to resolve it. Several different sites, but on this one I have following configuration.

Master server is on Solaris 10, Media server is on windows 2008 and uses VTL HP Storeonce with 2000 Tapes (yes you read well, 2000). A lot of Media pools exist and only 3 are used. The rest are residuum from some old configuration, I suppose. VTL D2D is FC connected. Zoning is correct as this VTL and netbackup are operational.

I am facing with an issue, since there is no way other than by hand to delete expired tapes and not face with space issue (D2D is at 99% usage almost every day), and since Netbackup sees that more tapes do exist, he does not overwrite the old ones.

Question is, can I delete those tapes (and media pools) that are not used, and by deleting I mean, completely delete, remove from netbackup configuration, and at later point remove them from VTL configuration, thus giving netbackup enough tapes to rotate backups and reqrite old tapes that have expired and not face space issues again?

Should I also delete tapes that are in pool used for backups but are empty and not used at all, thus providing netbackup information that he doesnot have any more tapes in this pool and he should overwrite old ones (expired)?

Is there a procedure you can point me to, that will provide me with this kind of solution in most painless way.

Regards

Sasa

[Marianne]

Please bear in mind that NBU knows no difference between physical tape libraries and VTL, as the VTL is presented at OS level the same as physical devices.

NBU has no reason to delete or erase expired tapes.
Where all images have expired, NBU removes the Assign Time and media is then available to be overwritten from NBU point of view. Same as with physical tape.

To know why NBU are not using expired tapes, we need more info, as there may be a difference in type/density or in the wrong pool.

If density is different to VTL tape drives, you may delete them from NBU and VTL and create new ones.

[RiaanBadenhorst]

I'm sure the media is not being used because no policy is refercing their pool. They were probably manually placed there and are therefore not returning to scratch either. If you move them to scratch NetBackup will reuse them.

The bigger problem how ever is over subscription (configuring to many media). How big is the VTL?

If your VTL is X TB you should apply some logic in the amount of tapes you create. You should also still have realistic retention periods. If it has dedupe then sure, apply that to your calculation.

Assume that you get a dedupe ration of 90%. That would mean you're going end up to storing 10% (think physical world).  Assume you have 10TB of data and you back it up daily and keep it for 30 days. That would give you a requirement of 300TB (should you use non-dedupe disk / or tape and ignoring compression). If you were to backup to a dedupe device you would then need 30TB in your devices  (10% of 300TB).

When working with dedupe disk devices its fairly easy cause you just buy a 30TB device and off you go. The problem comes when you get a VTL and nobody does this calculation. The engineers who sets up the VTL ask how many media do you want, the client says "Hmmm, how many can I have?", engineer says "As many as you want, its virtual". And boom, he goes and configures 2000 media with a 100GB cartridge size (200TB).

Now NetBackup sees 2000 (empty medias) and it starts using them. After it gets to media number 300 its going to run out of space. Your dedude-VTL is full, its as simple as that, deleting media will not help (at least deleting it from NetBackup, you'll have to delete them from the VTL).

Make sense?

[sasa_m]

yes I am aware of this, so I am not expecting for netbackup to delete old tapes :).

From his point of view there are more than enough tapes for him to write to, so I believe this is not netbackup configuration error, but D2D device configuration error (too much Tape drives created and not enough storage to go around), so I need to delete those drives from his database and make him not use those empty ones any more.

question is should I first delete drives on D2D and then reread library from netbackup point of view (Inventory robot), or should I delete tapes from netbackup first and then delete them from D2D ?

I am not allowed to delete all and start from beginning, thus I can only delete the ones that are not used and have 0 content on them.

Regards

Sasa

[sasa_m]

Yes, I agree and this is my conclusion also. 

Someone who configured this D2D device probably did not use sizing tool and since usable space size is approx 16TB, by creating 2000 cartdiges with 1TB size, made a lot of issues for me now when I tried to bring this to life, one year after :).

So the same question: in which order should I delete drives; netbackup and D2D, or D2D and then netbackup rescan?

Sasa

[RiaanBadenhorst]

Hi,

Currently even though those tapes are unassigned, the VTL is keeping the data on them. Think of it as having free / expired physical media in an unused pool. They're just sitting there, technically the data is still on them as NetBackup doesn't erase the data, its just overwrites. Same goes for the VTL media, the bits are still stitting on the HDDs.

What you need to do is look at all full (not in those pools) and "eject" them by moving them to NONE. Then move those expired media to scratch so NetBackup can use them. By doing that NetBackup can overwrite those used disk space on the VTL and put images on the expired media.

Then you'll need to let the ones that are outside the robot expire and start deleting some of the excess media bit by bit. Will be a management process and will take a while.
 

[Marianne]

When you say " 2000 Tapes" did you mean media/cartridges or tape drives?

I'm asking this because in your previous post you are saying "too much Tape drives created ".

If you want to recreate media/cartridges with bigger capacity, you can delete all Unassigned media from NBU and then delete them from the VTL. 

After you have recreated media in the VTL, you can Inventory the robot in NBU to see the new cartridges.

PLEASE be sure that you create media with the same media type as the VTL tape drives. You may want to create a barcode rule for all new media to go into Scratch pool (create one if there is not a scratch pool already). 
Pools that do not have media will draw from Scratch and expired media will go back to Scratch.
 

[sasa_m]

Ok

To cut long story short

Is there a command that I can issue on master server (Solaris 10) that will erase all expired tapes.

In this way I will at this moment (when I have 0GB free and available space) free some space and then will be able to take one, full backup that is valid and then I can playaround.

Regards

Sasa

[sasa_m]

Hi, thank you for your answer.

I was not so precise, and I do apologize for this

VTL has 16 tape drives and 2000 tape cartridges.

I do not want to recreate cartridges, just to remove the ones I do not use at all, and make meaningful number of tapes available to netbackup.

This was not created by me, but I expected to fix it.

Regards

Sasa

[sasa_m]

To make a long story short:

Is there a command to delete all expired tape cartridges from master server (Solaris 10)?

without checking for expiration date and everything else and stating it as a list of tapes...

Thanks

Sasa

[wojo]

I think one more step is needed here. Run a bplabel on all expired tapes before moving them back to scratch.

Once a new BOT marker is written to these tapes, both the VTL and NBU will see these tapes as empty and this space will then be released within the VTL.

Should this post be a duplicate - my appologies.

[RiaanBadenhorst]

You should not need to relabel media in a VTL constantly. NetBackup will just start writing at the beginning of the tape.

[wojo]

Part of the issue is related to a lack of overall space within the VTL. In order to releave this condition and buy time for the FE so that additional work can be performed to correct tape count and size and pools, forcing a relabel of expired tapes immediately releases the space back to the VTL.

You are correct in what you say, but why wait for a tape to be used when you need the extra space now.

[RiaanBadenhorst]

That makes sense as a once of yes, well spotted.

[sasa_m]

I've found some sort of workaround for this issue of mine.

I am posting here since it might bring someone help.

1. Using putty and logging all output I've logged onto D2D device (HP Storeonce)

2. I've listed all drives in library using command vtl show library list . This gave me the name of the library. Then I listed cartridges using command vtl show cartridges LIB1 where LIB1 is the name of the library.

3. I've extracted list of cartridges and set it in one file (just media ID). This file I named tape_list and used in 2 scripts I created on master server (Solaris 10).

4. Scripts gave me expiration date and media state. Scripts are:

#!/bin/bash
for x in `cat /home/sasa/tape_list` ;
 do
echo `/usr/openv/netbackup/bin/admincmd/nbemmcmd -listmedia -mediaid $x |grep "Data Expiration: " `>>output.txt

done

#!/bin/bash
for x in `cat /home/sasa/tape_list` ;
 do

echo `/usr/openv/netbackup/bin/admincmd/nbemmcmd -listmedia -mediaid $x |grep "Media Status:" `  >>output2.txt

done

I put all data from D2D ( that gave me media ID, and used data size) and data from those 2 files (Expiration date - if any, and status - if frozenoor full or active) in an excell spreadsheet.

5. Now I sorted all this data in spreadsheet so that cartridges with largest size used data, that have expired, show up on the top. Simply going one by one, checking in netbackup gui for data on this media, and deleting (quick erase) and then changing pool to "none" gave me more space on D2D device and full control over tapes.

6. Also media status showed those frozen cartridges that I found no use of, so I unfroze them and deleted also.

7. Plan is in future to change pool to almost all cartridges other than those that are used by valid backups and few more and to reduce usable cartridges to meaningfull number. ATM I am not approved to delete those cartridges from D2D or netbackup, so this is a good workaround, since at some point in time, I'll remove all unused cartridges.

8. Also to add, I've changed pool to cartridges that are empty, so that I can lower the number of cartridges before netbackup even starts writting to them.

I hope this brings help to someone.

Regards

Sasa