Cannot Expire Media (invalid command parameter)

[erosenburg]

Hello,

I've got 3 pieces of media that I can't expire, no matter what I do.  This media has a status of "active".  It's not frozen, and it exists in the database to the best of my knowledge.  Adding it to the database has no effect on expiring it.  Here is the message I get when I attempt to expire it.

T5120-04# bpexpdate -m STK470 -d 0
Media STK470 is due to expire at INFINITY
Are you sure that the data on this media is not critical to
your business, and you are sure you want to delete STK470 y/n (n)? y
invalid command parameter
T5120-04#

I have logging turned on, so if the experts need something from logs, just let me know.  I'd hate to have to blow away netbackup, but it is a test environment, so if I absolutely have to, i will.

Thanks!

[erosenburg]

I forgot to mention the details.

Server: Solaris 10 SPARC
Netbackup Version: 6.5.5

[CRZ · Accepted solution]

1. Stop me if you've heard this one: What version of NetBackup are you using?
2. If you're using 6.5.x, are Storage Lifecycle Policies in use?

I have a hunch you may need to open a support case and be prepared to send us some NBCC output but your answers may change that hunch.  :)

DOCUMENTATION and DOWNLOAD: How to install and use Symantec's Veritas NetBackup Catalog Consistency Utility (NBCC) 7.0
 http://support.veritas.com/docs/340564

[erosenburg]

Sorry about that, CRZ.

Yes, ti's NBU 6.5.5 on Solaris 10 Sparc.  Yes, I am using Storage Lifecycle Policies in the particular environment.

I've downloaded NBCC and I'm in the process of installing it.

[erosenburg]

This does appear to be SLP related.

First off, I have NBCC installed.

I have deleted all the drive and stoage units, prior to this, however when I just tried to delete the SLP, I got the following:

T5120-04# nbstl T9840B -delete
Exit error: images are in process
EXIT status = 1519

That lead me to run this command and it looks like there are jobs "IN PROCESS".  Could this be the cause?  I can't delete the SLP either.

T5120-04# nbstlutil list -U -lifecycle T9840B
Image:
 Master Server            : T5120-04
 Backup ID                : T5120-04_1267122167
 Client                   : T5120-04
 Backup Time              : 1267122167 (Thu Feb 25 11:22:47 2010)
 Policy                   : T5120-04_T9840B_data2
 Client Type              : 0
 Schedule Type            : 0
 Storage Lifecycle Policy : T9840B
 Storage Lifecycle State  : 2 (IN_PROCESS)
 Time In Process          : 1267124146 (Thu Feb 25 11:55:46 2010)
 Data Classification ID   : (none specified)
 Version Number           : 0

 Copy:
   Master Server       : T5120-04
   Backup ID           : T5120-04_1267122167
   Copy Number         : 1
   Expire Time         : 2147483647 (Mon Jan 18 20:14:07 2038)
   Expire LC Time      : 2147483647 (Mon Jan 18 20:14:07 2038)
   Try To Keep Time    : 1268331767 (Thu Mar 11 11:22:47 2010)
   Residence           : T5120-04-hcart-robot-acs-0
   Duplication Status  : 3 (COMPLETE)
   Job ID              : 0
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   Source              : 0
   Destination ID      : *NONE*

   Fragment:
     Master Server     : T5120-04
     Backup ID         : T5120-04_1267122167
     Copy Number       : 1
     Fragment Number   : 1
     Resume Count      : 0
     Media ID          : ST0000
     Media Server      : T5120-04
     Storage Server    : T5120-04
     Media Type        : 2 (RMEDIA)
     Media Sub-Type    : 1 (BASIC)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 21186510848
     Delete Header     : 1
     Fragment ID       : ST0000

 Copy:
   Master Server       : T5120-04
   Backup ID           : T5120-04_1267122167
   Copy Number         : 2
   Expire Time         : 2147483647 (Mon Jan 18 20:14:07 2038)
   Expire LC Time      : 2147483647 (Mon Jan 18 20:14:07 2038)
   Try To Keep Time    : 1268331767 (Thu Mar 11 11:22:47 2010)
   Residence           : T5120-04-hcart3-robot-acs-0
   Duplication Status  : 1 (NOT_STARTED)
   Job ID              : 0
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   Source              : 1
   Destination ID      :

 Copy:
   Master Server       : T5120-04
   Backup ID           : T5120-04_1267122167
   Copy Number         : 3
   Expire Time         : 1268331767 (Thu Mar 11 11:22:47 2010)
   Expire LC Time      : 1268331767 (Thu Mar 11 11:22:47 2010)
   Try To Keep Time    : 1268331767 (Thu Mar 11 11:22:47 2010)
   Residence           : T5120-04-hcart2-robot-acs-2-x4200jj
   Duplication Status  : 3 (COMPLETE)
   Job ID              : 6102
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   Source              : 1
   Destination ID      :

   Fragment:
     Master Server     : T5120-04
     Backup ID         : T5120-04_1267122167
     Copy Number       : 3
     Fragment Number   : 1
     Resume Count      : 0
     Media ID          : STK470
     Media Server      : T5120-04
     Storage Server    : T5120-04
     Media Type        : 3 (NDMP)
     Media Sub-Type    : 1 (BASIC)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 21186510848
     Delete Header     : 1
     Fragment ID       : STK470

Image:
 Master Server            : T5120-04
 Backup ID                : T5120-04_1267122168
 Client                   : T5120-04
 Backup Time              : 1267122168 (Thu Feb 25 11:22:48 2010)
 Policy                   : T5120-04_T9840B_data1
 Client Type              : 0
 Schedule Type            : 0
 Storage Lifecycle Policy : T9840B
 Storage Lifecycle State  : 2 (IN_PROCESS)
 Time In Process          : 1267124147 (Thu Feb 25 11:55:47 2010)
 Data Classification ID   : (none specified)
 Version Number           : 0

 Copy:
   Master Server       : T5120-04
   Backup ID           : T5120-04_1267122168
   Copy Number         : 1
   Expire Time         : 2147483647 (Mon Jan 18 20:14:07 2038)
   Expire LC Time      : 2147483647 (Mon Jan 18 20:14:07 2038)
   Try To Keep Time    : 1268331768 (Thu Mar 11 11:22:48 2010)
   Residence           : T5120-04-hcart-robot-acs-0
   Duplication Status  : 3 (COMPLETE)
   Job ID              : 0
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   Source              : 0
   Destination ID      : *NONE*

   Fragment:
     Master Server     : T5120-04
     Backup ID         : T5120-04_1267122168
     Copy Number       : 1
     Fragment Number   : 1
     Resume Count      : 0
     Media ID          : ST0001
     Media Server      : T5120-04
     Storage Server    : T5120-04
     Media Type        : 2 (RMEDIA)
     Media Sub-Type    : 1 (BASIC)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 21186510848
     Delete Header     : 1
     Fragment ID       : ST0001

 Copy:
   Master Server       : T5120-04
   Backup ID           : T5120-04_1267122168
   Copy Number         : 2
   Expire Time         : 2147483647 (Mon Jan 18 20:14:07 2038)
   Expire LC Time      : 2147483647 (Mon Jan 18 20:14:07 2038)
   Try To Keep Time    : 1268331768 (Thu Mar 11 11:22:48 2010)
   Residence           : T5120-04-hcart3-robot-acs-0
   Duplication Status  : 1 (NOT_STARTED)
   Job ID              : 0
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   Source              : 1
   Destination ID      :

 Copy:
   Master Server       : T5120-04
   Backup ID           : T5120-04_1267122168
   Copy Number         : 3
   Expire Time         : 1268331768 (Thu Mar 11 11:22:48 2010)
   Expire LC Time      : 1268331768 (Thu Mar 11 11:22:48 2010)
   Try To Keep Time    : 1268331768 (Thu Mar 11 11:22:48 2010)
   Residence           : T5120-04-hcart2-robot-acs-2-x4200jj
   Duplication Status  : 3 (COMPLETE)
   Job ID              : 6116
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   Source              : 1
   Destination ID      :

   Fragment:
     Master Server     : T5120-04
     Backup ID         : T5120-04_1267122168
     Copy Number       : 3
     Fragment Number   : 1
     Resume Count      : 0
     Media ID          : STK470
     Media Server      : T5120-04
     Storage Server    : T5120-04
     Media Type        : 3 (NDMP)
     Media Sub-Type    : 1 (BASIC)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 21186510848
     Delete Header     : 1
     Fragment ID       : STK470

[rjrumfelt]

to bounce NetBackup to kill the SLP jobs in process and then try bpexpdate again?

[erosenburg]

I've done that.  In fact, I've power cycled the master server, and the 4 other media servers, plus power cycled the ACSLS server, along with the SL8500 library.  It still fails.

[rjrumfelt]

when you bounce NBU, SLP starts back up before you can expire the tape - what about executing nbstlutil -cancel -force T5120-04_1267122168

and do this for all backup id's that are being written to your tape

I have a feeling crz is right though, this may require a technician to clean up your catalog in some shape or fashion.

[erosenburg]

Thanks for the help.  I ran that command, then tried to expire.  I got the same error.  I then bounced NBU, then ran the comman again, and attempted to expire, and it still failed.  Since this isn't a prodcution environment, reloading NBu isn't such a big deal.  I was hoping there may be a quick fix, but that's looking less likely...LOL.

[Marianne]

I'd like to check NBU logs before going the drastic route of reloading.
The 1st one would be admin log. Hopefully it will contain exact parameters passed to bpexpdate command.
Have you tried to add the media server name that owns the tape: '-host <media-server>' and/or -force?
Also check command output of:
bpimmedia -mediaid STK...
nbemmcmd -listmedia -mediaid STK...

[Nicolai]

But got around the issue by specifying the -h {host_option}.

Try bpexpdate -m STK470 -d 0 -h {media_server_that wrote_the_backup}

[CRZ]

Sorry e (may I call you e?) but from what you've shared here, I am almost certain you will have to open a case with us, send NBCC output and (likely) get it escalated.  We'll probably have to send you some SQL commands to run against your database to clear things up by the time it's all said and done.