Netbackup 5.1 going threw scratch tapes very fast

[hstankovic]

We run backups 24/7 and have recently received 80 brand new LTO1 tapes 2 weeks ago.  Now we have 10 new tapes left out of that original 80.  We have had problems with not enough space in our library but 3 weeks ago we expanded our library so space is no longer a problem.  We use to have to remove tapes to make room for scratch tapes which would cause our vault process to fail.  I'm wondering if the vault process has ANY errors, such as media read/write error, or tapes not being in the library how does that effect vault and the scratch tapes it was using to duplicate the images to?  Will those use to be scatch tapes still be written to or will NBU see them as full?  Any help with this would be great.  We are running NBU 5.1 no MP, on a HPUX master and 3 HPUX media servers, 6 LTO1 drives.

Thanks in advance!
Helen

[John_Stockard]

You're running NBU 5.1 with no maintenance packs?  That's daring.  At a minimum, I would put Maintenance Pack 7 on your NetBackup master server, media servers, and clients to squash the bugs that are known to exist in NBU 5.1 GA.

The NBU 5.1 MP7 pack for the core HP-UX master server and media server binaries is here:  http://seer.entsupport.symantec.com/docs/302681.htm

The NBU 5.1 MP7 pack for the Unix client binaries is here:  http://seer.entsupport.symantec.com/docs/302661.htm  (you will need this for the master server and media servers, too)

The NBU 5.1 MP7 pack for the Java admin GUI is here:  http://seer.entsupport.symantec.com/docs/302652.htm

The NBU 5.1 MP7 pack for the Vault binaries on your HP-UX master server is here:  http://seer.entsupport.symantec.com/docs/302642.htm

Also, you should be aware that NBU 5.1 is now an end-of-lifed product as of June 30, 2009.  Symantec Tech Support won't provide much help with this version, and will probably just tell you to upgrade to NBU 6.0 or NBU 6.5.

Given that NBU 5.1 GA needs to be patched and is at the end of it's supportable life, try looking at these things:

* Run the available_media script to see if your new tapes are all frozen.  If they're frozen, you should manually unfreeze them with the bpmedia command and then manually label them with the bplabel command.  NBU will not use a frozen tape for any backup or duplication jobs, and it won't automatically unfreeze the tape on it's own.

* Check the Vault progress logs (/usr/openv/netbackup/vault/sessions/sid*/*) to see if Vault encountered any problems during the duplication jobs.  If it encounters read/write errors or tape mounting problems, it will freeze the tapes.  (This is the same behavior that you will see on backups, too.)

[Abesama]

Scratch tape usage really depends on the configurations and settings.

Given the data growth rate these days, it will not surprise me some LTO1 environment running out of scratch tapes fast, especially if that environment runs backups 24/7.

Even with LTO4 tapes and handful of clients and non-24/7 environment, to say, if retention period is all set differently and volume pools are all separate, then many tapes will be used over night and probably the same number of tapes again, when vault tries to duplicate.

Check your retention period and volume pool configurations, verify if you are doing enough multiplexing (not too much, not too little) for your backup system's capacity.

And before doing all that, like John Stockard said (to which I clicked one vote) upgrade your NetBackup ... you don't want to find yourself in a big trouble which Symantec Tech Support can't help.

Abe

[Marianne · Accepted solution]

NetBackup will use a different tape for each of these scenario's:
Different pool
Different retention level
Different media server

Depending on your vault configuration, your media will probably suspended, meaning that no further backups can be written to it and new tapes will be needed every day.
Run Media Written report and Media Logs report on a daily basis to see what is happening to your tapes.

[hstankovic]

Hi Everyone,

   Believe me if I could upgrade I would have.  I did find out the hard way that NBU 5.1 is no longer supported when I tried to call for assistance.  However the team that needs to be involved in this upgrade is VERY busy with other projects and I can't upgrade without them so I'm stuck. 

Marianne - what command do you use for the Media Written and Media Log reports? 

Abesama - I tried to tell me boss that we will use a lot of tapes because the data we backup becomes more and more but because this hasn't been a problem until recently (within 6 months of me taking of the department) "there must be another reason because this wasn't going on before." Find it!  So that's what I'm trying to do.

John - Thanks for the suggestions, I have checked for frozen media and that's not the case because it is usually an error on a different tape or different drive each time so there are not 3 errors on the same tape or drive in a 12 hour period so NBU is not freezing the tapes.  I will check the logs though.

All thank you again for the suggestions.  Please keep them coming.

[ctate]

You mentioned you expanded your library...Did you add more tape drives also?

If you added tape drives you have potentially less queueing so more tapes are used at the same time and you get less data per tape.   I actually downed some drives in one of my robots because it was using too many tapes.  I downed 5 tape drives out of 20 and went from using 70 tapes per day to around 50 in that robot.   I had more queuing and job ran longer, but they still finished before my catalog backup and vault so it wasn't a problem.

[hstankovic]

No we only added about 60 more slots for tapes because we would have to eject tapes multiple times a day to make room for scratch tapes.

[hstankovic]

Yeah we have majorly long run times.  Our vault has been running for over 100 hours at this point.  it was running so smoothly for the first time in a long time last week but this week it has just been taking a very very long time to run.  But that's another issue I'm sure that would be cleared with a better library, tapes, and more recent software.

[hstankovic]

I looked at the logs in /usr/openv/netbackup/vault/sessions/V1/sid* (depending on the day) and the ones for today are duplicating just fine.  I did kill one because it had been running for 18 hours but in the gui it didn't look like it was actually doing anything.  The only failure I see are when it's a 150 error, we killed it.