Empty media despite infinite retention time

[Robert13]

Good day,

We use Netbackup 6.5.4 on a windows 2003 machine wich connects to a Fujitsu Centricstor (Backup-to-disk solution)
The monthly backup has an infinite retention time configured.

I experience the following problem.

The past two monthbackups fail because all drives are offline.
In the logs i can see that Netbackup put these offline.

Netbackup asks the Centricstor to mount a (virtual) tape which according to Netbackup does not contain any data.
The tape DOES actually contain data and the physical tape which contains the virtual tape is stored in a third party location in a tapevault.

The Centricstore knows the tape is unavailable and Netbackup responds with a DOWN Drive command.
This goes on till all virtual drives are down.

bpmedialist -mlist -m M02930 returns:
requested media id is not assigned to this host in the EMM database 

nbemmcmd -listmedia -mediaid M02930 comes up with:
NBEMMCMD, Version:6.5.4
====================================================================
Media GUID:                     7ca014de-77d8-48cf-bbd2-671e592b7394
Media ID:                       M02930
Partner:                        -
Media Type:                     HCART
Volume Group:                   000_00002_ACS
Application:                    Netbackup
Media Flags:                    1
Description:                    CS monthly offsite
Barcode:                        M02930
Partner Barcode:                --------
Last Write Host:                NONE
Created:                        09/07/2012 16:22
Time Assigned:                  -
First Mount:                    10/06/2012 14:41
Last Mount:                     10/06/2012 14:41
Volume Expiration:              -
Data Expiration:                -
Last Written:                   -
Last Read:                      -
Robot Type:                     ACS
Robot Control Host:             NONE
Robot Number:                   2
AcsAcs:                         0
AcsLsm:                         0
Cleanings Remaining:            -
Number of Mounts:               1
Maximum Mounts Allowed:         0
Media Status:                   ACTIVE
Kilobytes:                      0
Images:                         0
Valid Images:                   0
Retention Period:               -
Number of Restores:             0
Optical Header Size Bytes:      0
Optical Sector Size Bytes:      0
Optical Partition Size Bytes:   0
Last Header Offset:             0
Adamm Guid:                     00000000-0000-0000-0000-000000000000
Rsm Guid:                       00000000-0000-0000-0000-000000000000
Origin Host:                    NONE
Master Host:                    hpbck01
Server Group:
Upgrade Conflicts Flag:
Pool Number:                    14
Volume Pool:                    CS_MNTHOffsite
Previous Pool Name:             -
Vault Flags:                    -
Vault Container:                -
Vault Name:                     -
Vault Slot:                     -
Session ID:                     -
Date Vaulted:                   -
Return Date:                    -
====================================================================
Command completed successfully.

When i look in the CentricStor for the physical tape which contains de virtual tape (M02930):

(CSTORE:A)IUP0:~ # plmcmd query -V 000754
    pos PV     TL     PVG    state next-bl    LVs -  val cap/GiB valid/GiB val/%
      1 000754 I500   P_MNTH _v__  2659026      7      7  781.47    649.17  100
      pos   LV     file-Id    LVG         bl_nr   size/MiB   save request at
        1   M02927 0x00000006 L_MNTH          2  95340.132  12-10-06 15:29:48
        2   M02930 0x00000002 L_MNTH     762729  95340.051  12-10-06 15:50:06
        3   M02933 0x00000002 L_MNTH     381365  95340.279  12-10-06 15:46:12
        4   M02938 0x00000002 L_MNTH    1144095  95340.250  12-10-06 16:29:17
        5   M02939 0x00000002 L_MNTH    1525461  92706.754  12-10-06 16:44:31
        6   M02943 0x00000002 L_MNTH    1896294  95340.043  12-10-06 20:31:51
        7   M02945 0x00000002 L_MNTH    2277660  95340.187  12-10-06 21:54:12
 

Here you can see that de physical tape contains 7 virtual tapes which are all written the same day.
Two of them contains, according to netbackup, no valid data. LV M02930 and M02945.
According to the CentricStor all LV's contain valid data.
They are all written at the same day, within the same backup policy and all with Infinite retention time.

Is there a valid reason why Netbackup believes there is no valid data on these two virtual tapes (LV's)?

Thanks in advance,

Robert.

[mph999]

If the backup runs and writes data, but the fails, NBU will 'throw away' that backup.

The data however will not be deleted from the tape, and so data will reside on the tape, but if you like, it is not complete.

Could this explain the issue ?

Martin

[Mark_Solutions]

The key line in your output is:

Data Expiration: -

That means that as far a NetBackup is concerned that tape is over-writable

As Martin says there is physically data on the tape but when NBU uses it next it will overwrite it from the beginning.

If your system lets NetBackup think the tape is available but your 3rd party says "you can't have it" then you can expect down drives / frozen media

You need to find a way of virtually ejecting or freezing anything your system is not prepared to let NetBackup actually have

Hope this helps

[Nicolai]

I am slightly confused. You say Netbackup write to CentricStor, buy you list a tape with the same volume label from a ACSLS enabled robot. How does data from CentricStor go to physical tape ?

How do you avoid Netbackup complaining about duplicate ID. Is there some sort of OST involved ?

[Marianne]

How are you duplicating images from virtual to physical?

I cannot think that any other method except for NBU duplication will be supported.

Please read through Virtual Tape Libraries/Drives section in NBU HCL.

[Robert13]

Hello Martin,

the backup from 12-10-06 (october 6, 2012) was completed successfully.

No failures what so ever in that backup

[Robert13]

How can a tape being concerned overwriteble when the rentention time is infinite?

[Robert13]

CentricStore links Netbackup volume pools to its own volume pools.

When for instance netbackup uses tapes from CS_MNTHOffisite, Centricstore uses it's own linked pool which is, in this case P_MNTH.

Netbackup only talks to the Centricstor. It has now idea there is a tape library behind the Centricstor.
The Centricsor talks to Netbackup as well as the Library, for the CentricStor is the only one who knows which virtual media is written on which physical media.

[Robert13]

The Centricstor writes from virtusl to physical.
As soon as the first virtual media is written and the centricstore mounts a new virtusl tape it starts writing from cache to phtysical.
 

It works like a train. Never problems. Daily incrementals works just fine, as well as weekly full backups.

It's just that Netbackup think that tapes wrtitten whith infinite retention time are over-writable again.

[mph999]

The tape you mention:  M02930

Shows no Assign time / Data expiration time - this is the reason NBU wants to write to it again.

A tape that we thiing should contain data, but shows as 'not assigned' is usually the cause of one of two things :

1.  This was the first backup to the tape and it did not suceed (you confirmed this is not the case)

2.  The backup was sucessful, but the tape was expired.

This is usually a case for NBCC, and so you should really log a call and get NBCC run.

It would be interesting to see the output of this command :

bpimmedia -mediaid M02930

Martin

[Robert13]

bpimmedia -mediaid M02930 says:

ne ontity was found

[mph999]

OK, so that would mean there is no trace of the images at all in NBU DB.

So, either ...

1.  The backup never suceeded (you have confirmed this is not the case)

2.  The image(s) were expired using bpexpdate command

3.  The files were accidently deleted from the images dir at an OS level (if all images were removed that reside on any given tape, NBU would expire the tape during the cleanup process that runs every 12 hrs)

How many tapes are affected, is it just this one you have mentioned.

Martin

[Robert13]

I have 18 tapes at the moment with this problem.

But i'm not going to wait for the next backup to fail.
I will check on which Physical Volumes the Virtual Volumes reside.
Put the Physical Volumes in the library.
Tell the CentricStor to Reorganize the Virtual Volumes.(They will be written to other Physicsal volumes)
Delete the Virtual Volumes from NBU.

Then i'll have to keep an eye on it.
If the problem re-occurs i'll log a call at Symantec.

Thanks for the help everyone.

Regards,  Robert.

[Mark_Solutions]

There is another possibility here - and all down to it now being after April 2013!!

You are using 6.5.4 of NetBackup and you say the retention is infinite - is it actually infinite or a fixed long retention period? - this really makes a difference here.

My though it that unix time ends in 2038 - so if you use a 25 year retention period then that sets the expiration date after the end of time!! (In Unix terms anyway - epoch time) which can make the backup expire straight away!

See this tech note : http://www.symantec.com/docs/TECH200501

If you do use Infinity then you are OK - but wanted to check this with you.

As well as being unsupported there are a heap of bugs in 6.5.4 - some of which include the possibility of data loss - such as running bpexpdate with the -stype switch expiring other data than that which you intended to expire.

It may be that you issue is related to just bugs in the version of NetBackup you are using but if it has only just started to happen i am wondering if you have been affected by the epoch time issue - which is not actually resolved until 7.5.0.5!

Hope this helps

[Marianne]

Please tell us more about this process?

Put the Physical Volumes in the library.
Tell the CentricStor to Reorganize the Virtual Volumes.(They will be written to other Physicsal volumes)
Delete the Virtual Volumes from NBU.

You cannot delete anything from NBU as long as there are images associated with the volumes.

Any method of copying images from virtual to physical other than NBU duplication will not be supported by Symantec.

[Robert13]

OK, here goes.....

NBU thinks M02930 does not contain any valid data.
The CentricStore thinks M02930 does have valid data. (see first post).

When i insert physical tape 000754 (which contains virtual tape M02930) in the library i can tell the CentricStor to reorganise M02930. For Tape 000754 was written on 12-10-06, removed from the library and put away at a third party vault storage, the data should physically be present on the tape. Although NBU thinks not. When CentricStor start reorganizing, it asks the library for the tape 000754 and to mount another physical tape from scratch pool. CentricStor reads M02930 from 000754 and writes it on the other tape.
In the centricstor database M02930 is now present on another tape.

Great process but NBU still thinks M02930 does not contain valid data.......hahahahaha
My brains are melting!!!!
I just want to delete de Logical Volumes from NBU.
But if i do that, the database from the CentricStor will be contaminated with Logical Volumes which will never be used anymore.

Maybe i can put the tape in the library and asks NBU to read the tape.
 

[Robert13]

Anyway, just had a good conversation with someone from Fujitsu.
There is a script on the NBU server which tells the Centricstor which tapes have suspended retention time. The centricstor deletes them from it's own database. Centricstor needs the physical tape to do that so i am going to put the physical tapes in the library. Centricstor should read the tapes and delete de logical volumes from database.

Regards, Robert.

[Marianne]

Again, any duplication and/or manipulation outside of NetBackup will not be supported by Symantec.

Extract from HCL (link above):

Copying virtual media to physical media or a 2nd virtual media is not supported by Symantec; any support for this must come from the VTL vendor providing the capability.
If the VTL copies virtual media to physical media (or a 2nd VTL), NetBackup is not aware of the physical media or 2nd virtual media and may not be able to recover that data directly from it.

[mph999]

... agree with Mariaane, but, does this detailed explanation given by Robert explain why the images details are completely missing from the NBU catalog ???

I've seen many issues with 'VTL' cleanup scripts causing lost data, but this has been on the VTL, when the scripts 'delete' the wrong tape (or the right tape but too early) - this leaves the catalog information in the NBU catalog, but an 'empty' tape, we seem to have the opposite of that in this case, or, am I missing something ?

M

[Marianne]

We have no isight into how exactly the following is done:

.......
Tell the CentricStor to Reorganize the Virtual Volumes.(They will be written to other Physicsal volumes)
Delete the Virtual Volumes from NBU.

As we know, it is impossible to delete NBU volumes without expiring it first.