No Scratch Tapes

[Moeman]

Every few months I run into a problem where all my scratch tapes are gone.  One day I'll have a dozen scratch tapes and the next they will all be gone and and backups will exit with status of 96.  Using the bpexpdate command I will have to manually expire tapes that haven't reached their expiration date yet just to get more scratch tapes in the library.  I have 70 policies defined and 37 clients.  Maybe it's bad or old media.

I'm just putting a shout out to see if anyone else has run into this.  Maybe my policies are not optimally configured.

NetBackup 6.5.6 running on Windows Server 2003 SP2

StorageTek L80 Tape Library

[RiaanBadenhorst]

Hi,

Take a look at bpmedialist, it will provide you details on the expiration dates on groups of media i.e. a bunch of tapes will expire tomorrow, a bunch will expire at the end of the week, and so forth. If you do this daily you should see that a bunch of media should consistently be expiring as the days progress. This should give you an indication if you actually have enough media based on the dataset and retention periods.

Also look our for media that is frozen.

[Ed_Wilts]

I hear a lot of people running into scratch tape issues if they have too many pools defined because a tape can only be in 1 pool at a time.  You can also run into issues if you have too many retentions, retentions that are too long, and if you mix retentions on the same tape.

Lastly, don't forget the possibility that you simply don't have enough scratch tapes in your library.  Sometimes things aren't busted but you simply don't have enough of what you need.

[J_H_Is_gone]

that was my first thought.

        "Sometimes things aren't busted but you simply don't have enough of what you need. """

and if you don't mix retention periods, then if all week you have been appending to tapes with 2 week retention, then on the weekend your jobs have retention of 2 months .......  well that is going to required a whole separate set of tapes.

Buy more tapes.

[Marianne]

Just my 2c to underline what everybody else has said:

Look at media usage on a daily basis. Useful standard NBU reports:

Tape summary report (remember to select the Verbose tick-box)
Tape written report (it will default to last 24 hours)
Tape logs report (Filter out Info type messages, leaving you with Warnings and Errors, giving you an idea of possible media/hardware problems that will cause frozen tapes)

If you have more than one media server, consider media sharing. This will greatly reduce media usage.

And of course everything else said in the previous posts...

[Moeman]

Thanks.  Looks like my problem is mixing retention periods on the same Volume Pool so buying more tapes won't help.  To keep things clean I have two Volume Pools I use for every backup: Full and Cumulative.  Unfortunately I have 1 week, 2 week, 1 month and 2 month retention periods going to the same tape.

[Ed_Wilts]

You can use multiple retentions in the same pool but just don't mix them on the same tape.

You can (and should) combine Full and Cumulative backups into the same pool.  

[J_H_Is_gone]

https://www-secure.symantec.com/connect/blogs/understanding-how-netbackup-writes-tape 

Once you read that you will see how mixing retentions on the same tapes is hurting you

you could have images with retentions like this on a tape

1week|1week|1month|2month\end of tape

now when the 1weeks expire you have

expired|expired|1month|2month|end of tape

you CANNOT write to this tape yet because the tape is still considered full, because backing up to tape it can only append to the end of the tape.

when 1month expires you get

expired|expired|expired|2month:end of tape

still cannot write to it

now the 2month expires

tape becomes a scratch tape and you can write to it again.

So you issue can still be you don't have enough tapes.

if your tapes are full you have to wait for ALL images on that tape to expire before you can write to it again.

if you don't mix retentions and you have a tape with just 1 week retentions on it when they all expire you can write to it again.

[Stumpr2]

What?

are your scratch tapes getting frozen?

FAQ DOCUMENTATION: How to troubleshoot frozen media on UNIX and Windows

https://www-secure.symantec.com/connect/forums/faq-documentation-how-troubleshoot-frozen-media-unix-...

[Moeman]

Exactly!

When this system was put in I thought I would keep it simple by having Full and Cumulative volume pools since those are the only two types of backups I perform.  Now it seems I should have volume pools that correspond to retention periods.  What do others do to not run into having multiple expiration dates on the same tape?

[Ed_Wilts]

I think you're confused by the purpose of a volume pool.  For your backups, you need ONE pool.  

Unless you've configured it to do otherwise, NetBackup will fill your tapes with backups that have the same retention.  So let's say your cumulative backups have a 2-week retention and your fulls have a 3-month retention.  NBU will take a tape from your scratch pool, move it to your now SINGLE pool, and start writing cumultative backups to it.  When the tape fills, it will grab another tape.  In roughly 2 weeks, the images on the first tape will expire and the tape will be moved back into the scratch pool.  Since NBU will not mix retentions on the same tape by default, your first FULL backup will grab a different tape.  NBU will write to it until it's full, and then grab another one.  

Pools are for PHYSICAL separation of images, not LOGICAL separation of images.

[Marianne]

"What do others do to not run into having multiple expiration dates on the same tape?"

Allow multiple retentions per media is disabled by default. We prefer to leave it like that.

It is enabled/disabled in Host Properties -> Media server -> Media -> Allow multiple retentions per media

Extract from Admin Guide:

This property allows NetBackup to mix retention levels on tape volumes. It applies to media in both robotic drives and nonrobotic drives. The default is that the check box is clear and each volume can contain backups of only a single retention level.

[Stumpr2]

hmmm. I got a negative thumbs down.

Does that mean that the scratch tapes are not being frozen?

[Moeman]

It wasn't me who gave you a thumbs down.  I don't even mess with the thumbs up and down, I just reply.

I'm not sure if scratch tapes are being frozen.  I checked for frozen media yesterday and found a bunch.  None of them had reached their expiration date yet, probably why they were frozen?  I unfroze all of them but none of them returned to the scratch pool.

[Stumpr2]

Just a little bit of confusion here. there are two expiration dates.

1. physical expiration of the media

2. images expiration on the media

A tape only goes to the scratch pool when all of the images on the media have reached the expiration date. Are there any  images on the tapes that you expired? Did you try to change the volume pool to scratch and it failed becuse it is assigned? The whole reason that I jumped on frozen tapes is that you could put a couple dozen tapes in the library on Monday and they could all be found to be frozen on Tuesday morning. Some reasons for frozen scratch tapes are write protect on, non Netbackup data on the tape, catalog images on the tapes, etc. If frozen scratch tapes is the problem then check out the link to the technote above.

[Moeman]

Ok, so that's not it.

I checked Host Properties -> Media server -> Media -> Allow multiple retentions per media and it is indeed not checked.

So by not having that checked, NetBackup by default is not mixing different retention levels on the same tape, but that is not what I am seeing.  I ran a Images on Tape report for one piece of media assigned to the Full volume pool.  In the Retention Period column it shows 3 weeks and 2 months, obviouslly different retention periods.  I checked a few other pieces of media and they too had multiple retention periods on them.

[Moeman]

I have it configured to do otherwise.  I have a Full volume pool and a Cumulative one.  I thought I needed these to keep thing separate.  Keep in mind this was installed back in 2005 so I don't recall the reason for the current configuration of volume pools.  So if I need only one volume pool, I just use the default one of NetBackup then and let the software manage filling the tapes with similar retention?

[Ed_Wilts]

>  So if I need only one volume pool, I just use the default one of NetBackup then and let the software manage filling the tapes with similar retention?

Exactly.

[Marianne]

Is it possible that someone else has enabled this at some point in time and disabled it again in the meantime? Also, if you have more than one media server, check this setting for all of them.

NBU will only mix retentions when this setting is enabled.

[Moeman]

Sounds logical but how does that play with vault?  I have vault jobs that specifically look for cumulative backups.  If I change to one volume pool how with my vault jobs know to vault cumulative versus full backups?