- LTO4 tapes don`t get expired, they are not moved into Scratch Volume Pool -

[Tom_Dooley]

In our Company we are using LTO4 tapes for backup from one of our NDMP

hosts.These tapes are inserted in 2 volume pools: one for incremental

backups (Daily_NDMP_LTO4) and one for full backups (Weekly_NDMP_LTO4).We

had some problems at network side and we had to change the NDMP host FQDN.

So the old NDMP host name was deleted from Netbackup, and the new NDMP

host name was inserted.Everything is working well.

The problem is with the tapes that were used for backup from the old NDMP

host, they are not expired and put in Scratch Volume Pool.All the images

on the tapes have the retention period set to 1 month, and this retention

period expired but the tapes are still in the volume pools mentioned in

the upper rows. Already tried to expire the media :

1.bpexpdate -m 400056 justmedia -d 0

Are you SURE you want to delete 400056 y/n (n)? y

database system error

2.bpimmedia -mediaid 400056

no entity was found

3.Followed the steps from:http://support.veritas.com/docs/300927

but no success.

nbemmcmd -listmedia -mediaid 400056

NBEMMCMD, Version:6.5

Media ID: 400056

Partner: -

Media Type: HCART

Volume Group: 002_00000_TLD

Application: Netbackup

Media Flags: 1

Description: Added by Media Manager

Barcode: 400056

Partner Barcode: --------

Last Write Host:

Created: 03/10/2008 09:12

Time Assigned: 04/12/2008 22:14

First Mount: 04/13/2008 00:22

Last Mount: 05/19/2008 10:27

Volume Expiration: -

Data Expiration: 05/20/2008 22:34

Last Written: 04/19/2008 22:34

Last Read: 05/19/2008 10:27

Robot Type: TLD

Robot Control Host: tdn1m02bks0w

Robot Number: 0

Slot: 269

Side/Face: -

Cleanings Remaining: -

Number of Mounts: 3

Maximum Mounts Allowed: 0

Media Status: FULL

Kilobytes: 1014913736

Images: 52

Valid Images: 52

Retention Period: 3

Number of Restores: 1

Optical Header Size Bytes: 1024

Optical Sector Size Bytes: 0

Optical Partition Size Bytes: 0

Last Header Offset: 14104927

Adamm Guid: 00000000-0000-0000-0000-000000000000

Rsm Guid: 00000000-0000-0000-0000-000000000000

Origin Host: NONE

Master Host: tdn1m02bks0w

Server Group: NO_SHARING_GROUP

Upgrade Conflicts Flag:

Pool Number: 12

Volume Pool: Weekly_NDMP_LTO4

Previous Pool Name: Scratch

Vault Flags: -

Vault Container: -

Vault Name: -

Vault Slot: -

Session ID: -

Date Vaulted: -

Return Date:

-

4.bptm -makedbentry -m 400056 -den hcart -poolnum 12

Error - database system error

What to do next?

Thank You in advance.

[bumj1]

This error message occurs if all of the images are expired on the tape and you try to run bpexpdate against it.  The tapes are probably not moving back to scratch because when you first added them to the library you may have checked them into the daily or weekly volume pool.  Therefore since the tapes started in those pools, they think they belong in that pool so they are not going to move back to scratch. 

Manually move the tapes to the scratch pool.  When NBU sees no tapes are in the daily or weekly pool it will grab the tapes from scratch and then return them as they expire.  Also, if the tapes are truely not expired you will not be able to manually move them back to scratch.

HTH

[Dion]

Sounds silly and might not work in your situation, but have you tried, simply
bpexpdate -m <mediaID> -h <oldhostname> -d 0

Failing that (and as a last resort) you can physically delete the media from the NB database (if you are certain that that there is no data you need) and reiventory the library. Just take note that you will loose certain information about each of these tapes e.g. Number of Mounts.

If you don't have any import rules set up to automatically move new media to the SCRATCH, you will be able to move them into the SCRATCH pool manually.  The media should act normally from here on.

[Tom_Dooley]

Hello guys,

Thank you for your quick reply, in addition of what I`ve posted yesterday; please look also on the tape summary report in the volume pool Weekly_NDMP_LTO4:

1. the report for the new NDMP host:
*******************************************************************************
MEDIA SUMMARY FOR SERVER tdn1n03isv0n.mgmt01.local, POOL Weekly_NDMP_LTO4 ON 28.05.2008 09:07:44
*******************************************************************************

ACTIVE   FULL  SUSPENDED  FROZEN  IMPORTED
    2     17        0        0        0

Number of NON-ACTIVE media that:
       7 - will expire between 2 and 3 weeks
            400029 expires 06/13/2008 17:51
            400026 expires 06/13/2008 20:35
            400024 expires 06/17/2008 20:00
            400037 expires 06/13/2008 20:35
            400028 expires 06/13/2008 20:35
            400027 expires 06/13/2008 20:35
            400032 expires 06/17/2008 23:42
      10 - will expire between 3 weeks and 1 month
            400031 expires 06/18/2008 16:00
            400033 expires 06/18/2008 16:00
            400035 expires 06/24/2008 20:00
            400036 expires 06/18/2008 16:00
            400038 expires 06/24/2008 22:30
            400041 expires 06/24/2008 20:09
            400040 expires 06/24/2008 22:15
            400039 expires 06/25/2008 16:00
            400042 expires 06/25/2008 16:00
            400045 expires 06/25/2008 16:00

Number of ACTIVE media that, as of now:
       2 - will expire between 3 weeks and 1 month
            400043 expires 06/24/2008 22:30
            400044 expires 06/25/2008 16:00

Summary by retention level of ALL media
     Level   # Media        Megabytes
       3        19         16958538.0

2.the report for the old NDMP host:

*******************************************************************************
MEDIA SUMMARY FOR SERVER tdn1n03ism0n.mgmt01.local, POOL Weekly_NDMP_LTO4 ON 28.
05.2008 09:12:04
*******************************************************************************

ACTIVE   FULL  SUSPENDED  FROZEN  IMPORTED
    0      0        0        0        0

3.list all the tapes from Weekly_NDMP_LTO4 volume pool

vmquery -pn Weekly_NDMP_LTO4 | qgrep media :

media ID:              400000
media ID:              400003
media ID:              400004

media ID:              400007
media ID:              400009

media ID:              400011
media ID:              400012
media ID:              400013
media ID:              400014
media ID:              400016
media ID:              400017
media ID:              400018
media ID:              400019
media ID:              400024

media ID:              400026
media ID:              400027
media ID:              400028
media ID:              400029
media ID:              400031
media ID:              400032
media ID:              400033
media ID:              400035
media ID:              400036

media ID:              400037
media ID:              400038
media ID:              400039
media ID:              400040
media ID:              400041
media ID:              400042
media ID:              400043
media ID:              400044
media ID:              400045
media ID:              400052
media ID:              400053
media ID:              400054
media ID:              400055
media ID:              400056
media ID:              400057
media ID:              400058
media ID:              400059
media ID:              400060
media ID:              400061
media ID:              400062
media ID:              400063

media ID:              400065
media ID:              400066
media ID:              400067
media ID:              400068
media ID:              400069

When I try to see the tapes summary the tape 400056 is not in the summary but it appears when I`m listing the contents of the Weekly_NDMP_LTO4 volume pool.

@Dion "you can physically delete the media from the NB database" - How can I do this? I tried to delete the media from the GUI but it is telling me that "cannot delete assigned volume (92)".

Thank You.

[Dion]

Seems as if your database has become a bit confured there.  If you images for these media have definitely expired, it seems that NetBackup has not flagged the tapes as "empty".  When I mentioned about deleting the media, yes, it was from the GUI but it seems that because of your problems, NetBackup will not allow you to delete the media as it thinks that the media is still linked to unexpired data.

Two quick things...

1. Did you try to bpexpdate with the old owner of these tapes?  If not try that first.

2. I haven't had to tried this yet but hopefully it will work.  Try the vmdelete -m <mediaID>.

Another Q.  Did you change EMM servers at some stage or did you just remove the old media server?

If this doesn't work you might need to log a fault

Good luck

[Tom_Dooley]

Hello,

No changes were done to the EMM server, only the NDMP host name was changed from tdn1n03ism0n to tdn1n03isv0n.

1. I cannot use bpexpdate -m <mediaid> -h <oldhostname> -d 0 the sintax doesn`t allow do do this:

bpexpdate -m 400056 -h tdn1n03ism0n.mgmt01.local -d 0

bpexpdate: unrecognized option -h

bpexpdate: -m <media id> -d <mm/dd/yyyy HH:MM:SS | 0 | infinity>
             [-host <name>] [-force] [-nodelete] [-notimmediate]
             [-M <master_server,...,master_server>]
bpexpdate: -deassignempty [-m <media id>] [-force]
             [-M <master_server,...,master_server>]
bpexpdate: -backupid <backup id> -d <mm/dd/yyyy HH:MM:SS | 0 | infinity>
             [-client <name>] [-copy <number>] [-force] [-nodelete] [-notimmedia
te]
             [-M <master_server,...,master_server>]
bpexpdate: -recalculate [-backupid <backup id>] [-copy <number>]
             [-d <mm/dd/yyyy HH:MM:SS | 0 | infinity>] [-client <name>]
             [-policy <name>] [-ret <retention level>] [-sched <type>]
             [-M <master_server,...,master_server>]
bpexpdate: -stype <server type>
             [-dp <disk pool> [-dv <disk volume>]]
             [-nodelete] [-notimmediate]
             [-M <master_server,...,master_server>]
        legal values for sched: 0=full,1=differential incr,2=user,3=arch
                                4=cumulative incr
2. vmdelete -m 400056
cannot delete assigned volume (92)

So the same error as trying from GUI :(.

Do you think that I should use NBCC (Netbackup Catalog Consistency Utility)? 

Any other ideas?

Thank You.

[Dion]

Sorry, I don't alway remember the switches off-hand but change the -h to -host <oldservername>

Last chance!

[Tom_Dooley]

Hello here is the output:

bpexpdate -m 400056 -host tdn1n03ism0n.mgmt01.local -d 0
Are you SURE you want to delete 400056 y/n (n)? y
requested media id is not assigned to this host in the EMM database

So bad luck I think :(.

Thank You for your help.

[Dion]

Yip, looks like you may have to call in the professionals unless someone else has any good ideas?  Cheers

[Tom_Dooley]

I have opened a case to Symantec mysupport, I`m waiting for their response.

If we find a solution I will post it right here in this thread. Hope we will find it soon :).

[Tom_Dooley]

Hello all,

Excuse my late reply (lot of work to do).

Opened a CASE to Symantec related to this here are the major steps that solved the issue:

ISSUE:

Cannot expire tapes

ERROR CODE/MESSAGE:

Error - database system error

ENVIRONMENT/CONDITIONS:

An NDMP host that had the media assigned to it was deleted and re-added with a new name

SOLUTION/WORKAROUND:

This condition is a result of a known issue with NetBackup 6.5 and 6.5.1. The EMM database referential integrity is compromised if a host that has media assigned to it is deleted.

The root cause of this issue is being investigated by Symantec with internal tracking number ET1185547 and the fix is currently targeted for 6.5.4 release. Meanwhile, if you need to move media from one host to another, please expire this media or move it to a different media server with bpmedia -movedb command.

Scripts were prepared and delivered to fix the database referential integrity.

Also here is a great link on how to correctly decommission a media server or NDMP host:

http://seer.entsupport.symantec.com/docs/230047.htm

Regards,

Tom.