cancel
Showing results for 
Search instead for 
Did you mean: 

Re-use tape space issue_2

A_Dobe
Level 3
Partner

Hi all,

I work with VTL Solution. And all the time, when backup fails with error code "96", I can no longer re-use the space for other backup. The failed backup still using space on the virtual tape though it's not in the catalog; the space is lost !

This failed backup appears in my output in "NetBackup Management > Tape reports > Tape contents" but not in the catalog so I can not expire this fake image.

Is there any way (CLI or GUI) to fix that and re-use the space on the tape ?

 

bak-master:/ # cat /usr/openv/netbackup/version
HARDWARE LINUX_SUSE_X86
VERSION NetBackup 7.1
RELEASEDATE Thu Feb 03 17:33:27 CST 2011
BUILDNUMBER 20110203
bak-master:/ #

Below is the example of the tape with ID : 8A0084

Catalog output

Capture704.PNG

 

Tape content output

Capture705.PNG

 

FOR ANY MISSING INFORMATION, PLEASE TELL ME !!!

1 ACCEPTED SOLUTION

Accepted Solutions

RLeon
Level 6
   VIP   

This is how it's supposed to work, regardless of whether it is a VTL or a physical tape library.
Please refer to the following excellent blog post:
http://www.symantec.com/connect/blogs/understanding-how-netbackup-writes-tape

 

Basically, for as long as there are valid images on the same tape, the 'gaps' left behind by the invalid ones (i.e., failed or expired images) will not be used by NetBackup for writing data.
Only when the last valid image on the tape has expired (i.e., there are no images left on the tape according to the catalog) would NetBackup go back and write the tape from the very beginning.

 

In your case:
On 25/Apr/2014, you have 1 image fragment on tape 8A0084:
[ Image1 Fragment4 | Empty space ]

Because the backup job succeeded, Image1 and all of its fragments on the tapes 8A007Y, 8A0082, 8A0083 and 8A0084 are considered valid, and are therefore stored in the catalog.

 

Then, on 28/Apr/2014, NetBackup attempted to use the remaining space on tape 8A0084 to store the first fragment of another image:
[ Image1 Fragment4 | Image2 Fragment1 ]

Even though Image2 Fragment1 completed writing on tape 8A0084 using its remaining space, NetBackup couldn't find enough eligible tapes to write Image2's remaining fragment(s).
In short, because of status 96, the job did not complete successfully, and therefore Image2 is not stored in the catalog:
[ Image1 Fragment4 | Used Space ]

The 'Used Space' is just that. Its occupied, but contains no valid image, and NetBackup would not use this space to append further backups. This tape would be considered 'Full' at this point.
As explained previously, only when Image1 Fragment4 has expired on this tape, could the whole tape - including the Used Space - be reused.

 

The Catalog query only shows images that are 'valid' according to the catalog, whereas the the Tape Content report mounts and reads the tape to see what's actually in there.
An image fragment can be 'found' this way, but it does not necessary mean the image has to be recorded in the catalog as a valid image, or to be part of one.

 

I would recommend focusing on trying to prevent status 96 from happening in the first place. With a VTL, you may want to consider presenting more virtual tapes to NetBackup. But if disk space is an issue, then... well that's another problem.

View solution in original post

11 REPLIES 11

revarooo
Level 6
Employee

You do have a backupid, bak-svr-backup01_1398421114 and bak-svr-backup01_13986783133

You can expire them if YOU  wish....

bpexpdate -backupid bak-svr-backup01_1398421114 -d 0

bpexpdate -backupid bak-svr-backup01_1398678133 -d 0

 

These images are successful. Run a verification on them. If you don't want them expire them

A_Dobe
Level 3
Partner

Hi Revaroo,

The fake image does not work, see below ...

Capture706.PNG

A_Dobe
Level 3
Partner

The fake image does not appear in NBU images, see below ...

Capture707.PNG

Marianne
Level 6
Partner    VIP    Accredited Certified

Please show us output of:

bpimmedia -mediaid 8A0084

nbemmcmd -listmedia -mediaid 8A0084

RLeon
Level 6
   VIP   

This is how it's supposed to work, regardless of whether it is a VTL or a physical tape library.
Please refer to the following excellent blog post:
http://www.symantec.com/connect/blogs/understanding-how-netbackup-writes-tape

 

Basically, for as long as there are valid images on the same tape, the 'gaps' left behind by the invalid ones (i.e., failed or expired images) will not be used by NetBackup for writing data.
Only when the last valid image on the tape has expired (i.e., there are no images left on the tape according to the catalog) would NetBackup go back and write the tape from the very beginning.

 

In your case:
On 25/Apr/2014, you have 1 image fragment on tape 8A0084:
[ Image1 Fragment4 | Empty space ]

Because the backup job succeeded, Image1 and all of its fragments on the tapes 8A007Y, 8A0082, 8A0083 and 8A0084 are considered valid, and are therefore stored in the catalog.

 

Then, on 28/Apr/2014, NetBackup attempted to use the remaining space on tape 8A0084 to store the first fragment of another image:
[ Image1 Fragment4 | Image2 Fragment1 ]

Even though Image2 Fragment1 completed writing on tape 8A0084 using its remaining space, NetBackup couldn't find enough eligible tapes to write Image2's remaining fragment(s).
In short, because of status 96, the job did not complete successfully, and therefore Image2 is not stored in the catalog:
[ Image1 Fragment4 | Used Space ]

The 'Used Space' is just that. Its occupied, but contains no valid image, and NetBackup would not use this space to append further backups. This tape would be considered 'Full' at this point.
As explained previously, only when Image1 Fragment4 has expired on this tape, could the whole tape - including the Used Space - be reused.

 

The Catalog query only shows images that are 'valid' according to the catalog, whereas the the Tape Content report mounts and reads the tape to see what's actually in there.
An image fragment can be 'found' this way, but it does not necessary mean the image has to be recorded in the catalog as a valid image, or to be part of one.

 

I would recommend focusing on trying to prevent status 96 from happening in the first place. With a VTL, you may want to consider presenting more virtual tapes to NetBackup. But if disk space is an issue, then... well that's another problem.

A_Dobe
Level 3
Partner

Hi RLeon,

In conclusion the tape space used by the fake image can not be re-used until all images expire.

So, best solution is to prevent status 96.

Thanks for your answer, great !!!

Mark_Solutions
Level 6
Partner Accredited Certified

There is also the possibility that the failed job is marked as incomplete so holds a partial image

This will not clear down until it changes to failed

Obviously always make sure you have plenty of tapes to use for all jobs but if in doubt make sure there are no incomplete backups and to do a nice cleanup run the following to clear up the catalog:

bpimage -cleanup -allclients

Once the image cleanup job has finished run:

bpexpdate -deassignempty -force

This does a catalog cleanup of all expired fragements and then makes all empty tape available again without having to wait until midnight for the regular daily database cleanup.

I have also seen some VTLs that dont clean tapes up even then and need each empty tape to be labelled - but if you have never had that issue dont worry about that sort of thing now!

Marianne
Level 6
Partner    VIP    Accredited Certified

Always good to use disk as DISK and not as tape.

VTL has all the limitations associated with tape.
VTL was good in pre-NBU 6.5 days when NBU did not have the means of addressing disk properly and best worked with tape.

As from NBU 6.5, Advanced Disk and OST with SLPs have made disk backups a breeze.
Please look into possibility to rather use this hardware as disk instead of VTL.

A_Dobe
Level 3
Partner

Hi Marianne, below the output ...

 

bak-master:~ # bpimmedia -mediaid 8A0084
IMAGE bak-svr-backup-01 8 bak-svr-backup-01_1398421114 Sauvegarde 13 Full 0 9 24280 2147483647 1 0
FRAG 1 1 205004224 0 2 6 7 8A007Y bak-media-afnet 65536 7935122 1397577764 14 0 *NULL* 2147483647 0 9 1 *NULL*
FRAG 1 2 712851392 0 2 6 1 8A0082 bak-media-afnet 65536 2 1398421114 18 0 *NULL* 0 0 9 1 *NULL*
FRAG 1 3 712851392 0 2 6 1 8A0083 bak-media-afnet 65536 2 1398421114 17 0 *NULL* 0 0 9 1 *NULL*
FRAG 1 4 441778108 0 2 6 1 8A0084 bak-media-afnet 65536 2 1398421114 13 0 *NULL* 0 0 9 1 *NULL*
bak-master:~ #
bak-master:~ # nbemmcmd -listmedia -mediaid 8A0084
NBEMMCMD, Version:7.1
====================================================================
Media GUID:                     21fe0c80-cc65-11e3-8000-8d8691cd49f0
Media ID:                       8A0084                              
Partner:                        -                                   
Media Type:                     HCART                               
Volume Group:                   000_00013_TLD                       
Application:                    Netbackup                           
Media Flags:                    1                                   
Description:                    Added by Media Manager              
Barcode:                        008A0084                            
Partner Barcode:                --------                            
Last Write Host:                bak-media-afnet                     
Created:                        04/25/2014 10:34                    
Time Assigned:                  04/25/2014 10:18                    
First Mount:                    04/26/2014 20:09                    
Last Mount:                     04/28/2014 19:51                    
Volume Expiration:              -                                   
Data Expiration:                INFINITY                            
Last Written:                   04/28/2014 09:42                    
Last Read:                      04/28/2014 19:51                    
Robot Type:                     TLD                                 
Robot Control Host:             bak-media-afnet                     
Robot Number:                   13                                  
Slot:                           19                                  
Side/Face:                      -                                   
Cleanings Remaining:            -                                   
Number of Mounts:               4                                   
Maximum Mounts Allowed:         0                                   
Media Status:                   FULL                                
Kilobytes:                      712851388                           
Images:                         2                                   
Valid Images:                   1                                   
Retention Period:               9                                   
Number of Restores:             0                                   
Optical Header Size Bytes:      1024                                
Optical Sector Size Bytes:      0                                   
Optical Partition Size Bytes:   0                                   
Last Header Offset:             6902787                             
Adamm Guid:                     00000000-0000-0000-0000-000000000000
Rsm Guid:                       00000000-0000-0000-0000-000000000000
Origin Host:                    NONE                                
Master Host:                    bak-master                          
Server Group:                   NO_SHARING_GROUP                    
Upgrade Conflicts Flag:                                             
Pool Number:                    51                                  
Volume Pool:                    SAUVEGARDE                          
Previous Pool Name:             -                                   
Vault Flags:                    -                                   
Vault Container:                -                                   
Vault Name:                     -                                   
Vault Slot:                     -                                   
Session ID:                     -                                   
Date Vaulted:                   -                                   
Return Date:                    -                                   
====================================================================
Command completed successfully.
bak-master:~ #

Mark_Solutions
Level 6
Partner Accredited Certified

That tape is marked as full and has a retention period of infinity - so you will never be able to reuse that one unless you manually expire the images on it

Marianne
Level 6
Partner    VIP    Accredited Certified
Only 1 image left on tape 8A0084 that is fragment 4 of a big image that spanned 4 tapes. So, if you expire this tape, the entire image with all of its fragments will be lost. Infinity retention on VTL seems odd. This means you can never reuse/overwrite this media. As per RLeon's excellent post, expired fragment space on this tape cannot be overwritten because VTL has the same limitations as physical tapes. Please mark RLeon's post as Solution and let this go....