Full tape problem with Netbackup V4.5

[Tom_VK]

Hi,

I'm still using Veritas Netbackup V4.5

Recently I have the problem that my full backup at night is not finished. During the last task Netbackup tries to load another tape which is not possible because the server has only one tape drive.

When I check the media with bpmedialist I see that it is marked FULL and that the used space on the tape is +/- 158 GB. The tape is a 200/400GB LTO.

So at least another 42GB should be available. Before the tape was used it was expired, so I presume it is totally empty.

Any help will be appreciated.

rgds,

Tom

[Andy_Welburn]

Unfortunately tape capacity is not always "as advertised" - it can depend upon the type of data being saved & the compression that can be applied to it.

e.g. we use LTO3 (400/800Gb) & currently have FULL media that NB has recorded at anywhere from 250Gb to 1.2Tb!

[Marianne]

NetBackup will only mark a tape as FULL when the hardware has returned EOF. The product does not have a 'mind of it's own'  wrt to tape capacity. It will simply keep on writing until the media physically reaches end of media.

If you have bptm log on your media server, you will be able to find the evidence.

[mph999 · Accepted solution]

Adding to mariannes excellent post ...

NetBackup will only mark a tape as FULL when the hardware has returned EOF. The product does not have a 'mind of it's own'  wrt to tape capacity. It will simply keep on writing until the media physically reaches end of media

or ...

until the tape drive incorrectly returns end-of-logical media (we should never actually hit the very very end of media, that should be impossible).

I have just had a case - exactly the same data was sent to two identcal drives, it was exactly the same because it was in-line-tape copy.  (bptm sends exactly the same 'buffer-full' of information, one after the other - not quite the same time as bptm is single threaded).

One set of tapes was filling up way way before the others, not even reaching the native capacity.

Reason - faulty drive (or drives).

NBU is totally unaware of tape capacity - we send a block of data, the os / tape driver puts this on the tape, when the tape passes 'logical end of tape' we are able to finish that block (there is a bit more tape left) and the tape drive firmware sets a flag in the tape driver.  When we try and write ethe next block of data, the tape driver says, hold on, tape full.  So NBU goes, yep, got that, I'll load a new tape.

So, we only load a tape when we are told to.  Data amounts on tape have 0% to do with NBU.

Martin

[Tom_VK]

The most important I need to know is how I can find out if a tape is full and if the old backups are overwritten. I checked bptm.log but that is not that clear.

I cycle my daily tapes every two weeks, but I do not know if the new data is added or if the new backup starts backing up data from the beginning of the tape.

Somewhere between 250GB and 280GB I get a tape full warning and the backup stops (obviously).

The data being backed up is most source code (lot of text) so I still think the 280GB should fit compressed on a 200/400GB tapewithout any problem. If I check with BPMEDIALIST  I see the backup of last Friday takes 153.239.797KB. The native size of this data on the disk is +/- 250GB. So the compression is OK. But the last 30GB does not fit on that tape anymore and that is very very strange. Native there is still +/- 45GB space available on the tape.

[Marianne]

Please bear in mind that NBU will only overwrite when ALL images on a tape have expired. Before then, NBU will try to append.

Use the following reports on a Daily basis:

Media List report (look for expiration date)

NetBackup Media Summary report (click on Verbose)

from cmd: available_media (in netbackup/bin/goodies folder)

Please also see this excellent blog: https://www-secure.symantec.com/connect/blogs/understanding-how-netbackup-writes-tape

[Tom_VK]

Marianne, many thanks for the list of reports I should use.

Below you see some results of the reports:

MEDIA LIST REPORT:
IMAGES VALID IMAGES  ALLOCATED DATE EXPIRATION DATE  KILOBYTES  STATUS
4  4 04/07/2011  11/07/2011  153.239.234 MPX

---

MEDIA SUMMARY REPORT
*******************************************************************************
MEDIA SUMMARY FOR SERVER bebruaclr02, POOL MONDAY ON Tue Jul 05 2011 18:06:57
*******************************************************************************
ACTIVE FULL SUSPENDED FROZEN IMPORTED
3 0 0  0 0
Number of NON-ACTIVE media that:
    There are no NON-ACTIVE media present in the media database
Number of ACTIVE media that, as of now:
       1 - will expire within 1 week
       2 - will expire in greater than three months
Summary by retention level of ALL media
     Level   # Media        Megabytes
       0         2           298523.2
       1         1           289250.1

---

Sample of the output of the available_media command:
----------------------------------------------------

C:\VERITAS\NetBackup\bin\goodies>available_media
media   media   robot   robot   robot   side/   ret    size     status
 ID     type    type      #     slot    face    level  KBytes
----------------------------------------------------------------------------
DataStore pool

MONDAY pool

MON-02  HCART    NONE     -       -      -       -     -        AVAILABLE
301009  HCART    NONE     -       -      -       0     152448575        MPX
301107  HCART    NONE     -       -      -       1     296192159        MPX
A00002  HCART    NONE     -       -      -       0     153239237        MPX

WEDNESDAY pool

081010  HCART    NONE     -       -      -       -     -        AVAILABLE
WED-01  HCART    NONE     -       -      -       -     -        AVAILABLE
WEDTST  HCART    NONE     -       -      -       -     -        AVAILABLE
WXTRA1  HCART    NONE     -       -      -       -     -        AVAILABLE
WXTRA2  HCART    NONE     -       -      -       -     -        AVAILABLE
WED-02  HCART    NONE     -       -      -       0     152687731        MPX
 

There are 4 images on the tape and the size used is 152.687.731 KBytes

The 4 images are the backup of drive F:, E:, Z: and System_State

Hope this info is usefull.

[Tom_VK]

Anybody some more ideas how to solve this matter?

[J_H_Is_gone]

read the blog in Marianne's post -

it shows that if a tape is FULL, it will not be avail to write to again until ALL images on the tape have expired.

How to fix the problem,  buy more tape.

[Marianne]

You can also check output of 'bpmedialist -m A00002 '.

You should see two 'image fields' - one for number of images (valid and expired) and valid images.

There is nothing that can be done to reclaim expired images space. I was hoping that by now you would have read the Blog that I've mentioned on 4 July.

Also - none of the tapes listed in available_media output shows 'FULL'. So, I don't understand where your "Full tape" problem is seen.

I humbly 'bow out' ... nothing more to say.....

[mph999]

Hardware issues can cause this problem.

As my previous post explained, NBU has 0 control over when to mark a tape full.  We do as we are told by the tapedrive/ firmware / tapedriver / operating system.

Martin (UK Senior Symantec TSE)

[Tom_VK]

What command should I use to verify if all images are removed from a tape. If I'm not wrong an expired tape still contains backup images.

The blog ( https://www-secure.symantec.com/connect/blogs/unde...) was very interesting to get an overview how Netbackup handles backups, but I'm missing the commands to use this knowledge in real life.

[Tom_VK]

One thing that will solve the problem: What is the command to force to expire ALL images on the tape?

[Andy_Welburn]

ABSOLUTELY 100% SURE that you do not require any images on the media (& possible images that span to other media) then the command would be bpexpdate -d 0 -m media_id (a -f option will supress the "Are you sure?" prompt).

You may also need bpexpdate -deassignempty to unassign said media.