How do I delete backups on an old DiskPool

[khemmerl]

Hello everyone,

I hope you can help me out.  Several months ago, my company added a new MSA disk pool to replace our aging DataDomain disk pool.  All backup policies were reconfigured to stop using DataDomain and, originally, the usage on DataDomain dropped as backups aged and expired.  The problem is that the Data Domain usage only dropped to 14% and has been stuck there for over a month.  When I go into the NetBackup GUI (version 7.1) and navigate to "Media and Device Management" > Devices > Disk Pools, it displays 2.5677 TB used oiut of 18.1005 TB total (about 14%). 

I'm not sure why these backups are not expiring.  After doing some reading, I ran the following from D:\Program Files\Veritas\NetBackup\bin\admincmd on my NetBackup admin host:

bpimmedia -L -stype DiskPool -dp dd670
Backup-ID             Policy     ScheduleType
Copy  Frag  Expires     DiskType  DiskPoolName  DiskVolume  StorageServerName
-----------------------------------------------------------------------------
APP-SRV29_1357349450  APP_SRV29  FULL
1     1     1360027850  DiskPool  dd670         DDSTU1      inf-srv17
APP-SRV29_1357349451  APP_SRV29  FULL
1     1     1360027851  DiskPool  dd670         DDSTU1      inf-srv17
... (and on for hundreds of entries)

The server APP_SRV29 no longer exists so we certainly don't need to keep these backups (because all backups are for disaster recovery, not historical snapshots).  After some more reading, I found documents talking about bpexpdate.  Unfortunatetly, using this command gives me "no entry was found":

bpexpdate  -backupid APP-SRV29_1357349450 -d 0
Are you SURE you want to delete APP-SRV29_1357349450 y/n (n)? y
no entity was found

Oddly, even though the bpimmedia command shows the Policy of APP_SRV29, the bpimagelist shows nothing:

bpimagelist -policy APP_SRV29
no entity was found

I am very frustrated.  I have 2.6 TB of backups sitting in my disk pool that I can see but not clean up.  Can anyone offer me help on how to delete these backup IDs?  Any help would be greatly appreciated.

Ken

[RiaanBadenhorst]

Hi,

The following will wipe the entire DD.

bpexpdate -stype DataDomain [-dp disk_pool_name] [-nodelete]

nbdelete -allvolumes

[khemmerl]

Thanks for the reply Riaan.Badenhorst.  I think I'll use that as a last resort.  I was hoping to be able to go through and clean up backups that are not needed anymore and then review the remaining to ensure there are backups to the new disk THEN use the command you've provided. 

Does Netbackup somehow treat diskpools so differently that there's no way to delete individual backups?

Ken

[Marianne]

I find it strange that bpimmedia is still listing this image: APP-SRV29_1357349450 (and the other one too), as the expiration date (1360027850) was more than a year ago:

UNIX time 1360027850 is 02/05/2013 1:30am GMT

Try this:
nbdelete -allvolumes -force

[khemmerl]

That command doesn't appear to do anything.

D:\Program Files\Veritas\NetBackup\bin\admincmd>nbdelete -allvolumes -force

D:\Program Files\Veritas\NetBackup\bin\admincmd>

The output of bpimmedia does not appear to be any different after running the nbdelete command

[Mark_Solutions]

Did you try bpexpdate -backupid <Backup ID> -d 0 -nodelete?

Was it part of a SLP whose Lifecycle has not completed? If you you may need to cancel its lifecycle first.

There is also this one if you are really struggling:

nbstlutil remove_exp -force

[Marianne]

Also see what this command tells us:

bpimagelist - client APP_SRV29 -d 01/01/2013 -U

[mph999]

nbdelete -allvolumes -force will only 'work' if there is reference to the images in NBDB (in the correct tables) 

Else, the files on the STU are just that, files that NBU will ignore.

So. :

1.  Output of command from marianne above.

2.  Answer to Marks question as to if these are SLP images or not.

(I suspect not else when you try to bpexpdate them, it will actually tell you they are under SLP control, however, it does no harm to x2 check to avaoid any misunderstanding).

If 1/ and 2/ are 'clear' (ie. 1 shows nothing and answer to 2 is no) I suspect that in the past a bpexpdat was run with the nodelete option.  This clears the entries from the DB, but leaves the actual images on disk.  They then can't be deleted as NBU knows nothing about them - however, i could be wrong ...

No matter the way to investigate is the same:

Create an empty folder eg /data_unload

Run 

/usr/openv/db/bin/nbdb_unload /data_unload

This will create a  copy of NBDB, search every file for :

APP-SRV29_1357349450

grep 1357349450 *.dat

NOTE: Important to only search for the ctime from the backupid

May need to make other serchs using other backupids, but this is one you report the error on, so we'll try this first.

If entries apprear in some of the dat files, post up which .dat files they are and also a complete copy or the reload sql file which will also have been created.

Then, we identify which tables the .dat files reference, and knowing this can decide on the next step.

Hopefully, the backupids won't appear in any .dat files, in which case the files on the storage can simply be deleted via 'os' commands, however, we would need to double check the understanding of the issue before doing this, as it's fairly terminal ...

If they do appear in some .dat files, then depending on which ones, depend on what we do.  It could also mean other 'seraches' being required, not sure yet.

[khemmerl]

Your first command give "no entity was found":

bpexpdate -backupid APP-SRV29_1357349450 -d 0 -nodelete

	Are you SURE you want to delete APP-SRV29_1357349450 y/n (n)? y

	no entity was found

The second command gives "Not implemented yet"

nbstlutil remove_exp -force

	This operation will remove all expired image fragment entries in the database.  Any files containing backup images corresponding to these en

	.  Do you wish to proceed? (y/n) >y

	Not implemented yet.

nbstlutil does manage to see the backups though...

nbstlutil list

	V7.0.1 I inf-srv17 APP-SRV29_1357349450 APP-SRV29 1357349450 APP_SRV29 13 0 Disk-Disk-Tape-Monthly 3 1357349941 *NULL* 5  00000000-0000-0000884

	V7.0.1 C inf-srv17 APP-SRV29_1357349450 1 1360027850 1360027850 dd670-1dsu 3 0 0 0 0 *NONE* 1360027850 0

	V7.0.1 F inf-srv17 APP-SRV29_1357349450 1 1 0 @aaaac inf-srv17 *NULL* 0 6 1 5560740864 1 @aaaac

	V7.0.1 C inf-srv17 APP-SRV29_1357349450 2 1360027850 1360027850 dd670-2dsu 3 632768 0 0 1 *NULL* 1360027850 1357350012

	V7.0.1 F inf-srv17 APP-SRV29_1357349450 2 1 2 @aaaae inf-srv17 *NULL* 0 6 1 5560740864 1 @aaaae

	V7.0.1 I inf-srv17 APP-SRV29_1357349451 APP-SRV29 1357349451 APP_SRV29 13 0 Disk-Disk-Tape-Monthly 3 1357350864 *NULL* 5  00000000-0000-0000683

	V7.0.1 C inf-srv17 APP-SRV29_1357349451 1 1360027851 1360027851 dd670-1dsu 3 0 0 0 0 *NONE* 1360027851 0

	V7.0.1 F inf-srv17 APP-SRV29_1357349451 1 1 0 @aaaac inf-srv17 *NULL* 0 6 1 6437824512 1 @aaaac

	V7.0.1 C inf-srv17 APP-SRV29_1357349451 2 1360027851 1360027851 dd670-2dsu 3 632793 0 0 1 *NULL* 1360027851 1357350962

	V7.0.1 F inf-srv17 APP-SRV29_1357349451 2 1 1 @aaaae inf-srv17 *NULL* 0 6 1 6437824512 1 @aaaae

	317

[khemmerl]

In reply to the suggestion from Marianne, I tried the bpimagelist specifying both APP_SRV29 and APP-SRV29 (dash instead of underscore).  Neither command returns anything:

bpimagelist -client APP_SRV29 -d 01/01/2013 -U
no entity was found

bpimagelist -client APP-SRV29 -d 01/01/2013 -U
no entity was found

[khemmerl]

Regarding the question about whether these files are SLP images or not:  I'm sorry but can you explain how to answer that question?  

[khemmerl]

To answer the SLP question:  I have a Storage Lifecycle Policy named "Disk-Disk-Tape-Monthly" which is supposed to

• send a backup to the local DataDomain for a one month retention
• replicate to the remote DataDomain for a one month retention
• replicate to tape for offsite storage for one year

When I run "nbstlutil list", I see:

V7.0.1 I inf-srv17 APP-SRV29_1357349450 APP-SRV29 1357349450 APP_SRV29 13 0 Disk-Disk-Tape-Monthly 3 1357349941 *NULL* 5  00000000-0000-0000884

so it appears that these are SLP backups.

[khemmerl]

> If entries apprear in some of the dat files,
> post up which .dat files they are and also a
> complete copy or the reload sql file which
> will also have been created.

"grep 1357349450 *.dat" returned one entry in 808.dat and two entries in each of 810.dat and 811.dat.

grep 1357349450 *.dat

808.dat:'1000003','APP-SRV29_1357349450','APP-SRV29','1357349450','A
PP_SRV29','13','0','Disk-Disk-Tape-Monthly','3','1357349941','','5',
'',0x00000000,'0','0','2013-01-05 01:38:04.949000','2013-01-05 01:49
:13.507000'

810.dat:'1000003','APP-SRV29_1357349450','1','0','1360027850','13600
27850','dd670-1dsu','3','0','0','0','0','0','*NONE*','0','0','136002
7850','0','2013-01-05 01:38:04.949000','2013-01-05 01:49:13.351000'

810.dat:'1000003','APP-SRV29_1357349450','2','1','1360027850','13600
27850','dd670-2dsu','3','632768','0','0','1','1357350012','','1','0'
,'1360027850','0','2013-01-05 01:39:01.842000','2013-01-05 01:49:13.
366000'

811.dat:'1000003','APP-SRV29_1357349450','1','1','0','@aaaac','10000
03','0','0','6','1','5560740864','1','@aaaac','','2013-01-05 01:38:0
4.964000','2013-01-05 01:38:04.964000'

811.dat:'1000003','APP-SRV29_1357349450','2','1','2','@aaaae','10000
03','0','0','6','1','5560740864','1','@aaaae','','2013-01-05 01:40:1
2.651000','2013-01-05 01:40:12.651000'

"reload.sql" attached as requested.

[Marianne]

I suggest you log a Support call right away.

You will need assistance from NBU support engineer to fix inconcistencies.

[mph999]

OK, so if you run :

nbstlutil stlilist

This will give a slightly different output :

Example

V7.6.0 I womble_1404298590 slp1 3

If there is a 3 at the end, the image is complete and is not under SLP control

If it is a 1, or a 2 the image is either not started (1) or in progress (2) which will be stopping them being deleted.

Could you also try :

bpimagelist -backupid APP-SRV29_1357349450 -l

This should in theory return no entitiy found (based on your other bpimagelist command) but will be interesting to see for sure.  However, on the flip side, seeing nbstlutil list sees the image, so should bpimagelist ...

So in summary.

nbstlutil list shows the imeage

bpimage list doesn't (but please check using my command above backupid)

Even if under SLP (ie. 1 or 2 in nbstlutil stlilist output) this should't cause the no entity found, it should still be found, it would just fail to delete.

Other thing to check would be bpimagelist -l -d 01/04/13 -e 01/06/13 output, do you see the backupid in there (1357349450 converts to Jan 05 13

... in other words, is it list when not specifiacally searched for by backupid.

Got to pop out, will check back later

[mph999]

OK, scrap my above post - sorry didn;t see your post about the .dat file.

Let me take a look at that before we do anything else ...

[mph999]

Need to pop out so will do when I get back, in about an hour ...

[mph999]

Let me look before you call support (Mariaane is probablt right in that a support call will be needed) but let me chack as even with the delay of me going out, I will be quicker than support (they'll probably want to run some commands etc ... which prob aren't needed).

I'll forward the details to my ipad and look from there, try and shorten the time before I reply, otherwisew it'll be about an hour ...

[mph999]

OK.

808/ 810 /811 are Image / Copy / Frag tables

From reload file

INPUT INTO "EMM_MAIN"."EMM_Image" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/808.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MasterServerKey","BackupID","ClientName","BackupTime","PolicyName","ClientType","ScheduleType","StorageServiceName","StorageServiceState","TimeInProcess","ClassificationID","StorageServiceVersionNumber","OriginMasterServer","OriginMasterServerID","RequiredExpirationDate","ImportFromReplicaTime","CreatedDateTime","LastModifiedDateTime")
    ENCODING 'UTF-8'
go

call sa_unload_display_table_status( 17737, 80, 91, 'EMM_MAIN', 'EMM_MS_StorageServer_Connection' )
go

INPUT INTO "EMM_MAIN"."EMM_MS_StorageServer_Connection" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/809.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MediaServerKey","StorageServerKey","StateFlags","MaintOpPriority")
    ENCODING 'UTF-8'
go

call sa_unload_display_table_status( 17737, 81, 91, 'EMM_MAIN', 'EMM_ImageCopy' )
go

INPUT INTO "EMM_MAIN"."EMM_ImageCopy" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/810.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MasterServerKey","BackupID","CopyNumber","CopyType","ExpireTime","TryToKeepTime","Residence","CopyState","JobID","ExpirationType","MpxState","RetryCount","LastRetryTime","LifecycleDestinationTag","LifecycleSourceTag","DeferredDuplicationTime","ExpireLifecycleTime","IsReplica","CreatedDateTime","LastModifiedDateTime")
    ENCODING 'UTF-8'
go

call sa_unload_display_table_status( 17737, 82, 91, 'EMM_MAIN', 'EMM_ImageFragment' )
go

INPUT INTO "EMM_MAIN"."EMM_ImageFragment" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/811.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MasterServerKey","BackupID","CopyNumber","FragmentNumber","ResumeCount","MediaID","MediaServerKey","StorageServerKey","StorageUnitType","StuSubType","FragmentState","FragmentSize","DeleteHeader","FragmentID","MediaExtents","CreatedDateTime","LastModifiedDateTime")
    ENCODING 'UTF-8'

So if bpimagelist is not showing them, clearly something is wrong - so I think a support call is a good idea as I suspect this will need investigation via webex and form what I see at the moment is going to need engineering.  I'll look later to see if I can see what is wrong with the entries, but that's not necessarily obvious unfortunately.

Log call, pls. post up case number and show them this post:
Also run
bpimagelist -backupid <backupid> -l 

If this command returns 'noentity found' then mention this also

bpimagelist -backupid <backupid> shows no entity found, but image is referenced in Image / Copy/ Frag tables of NBDB (808 /810/ 811.dat) (mention if it shows the output or not, presume not for the min))

And

bpimagelist -client APP_SRV29 -d 01/01/2013 -U
no entity was found

Run 

bpimagelist -l -d 01/04/13 -e 01/06/13 

Check if this shows the backupid or not

Include nbdb_unload output in the case, and nbsu -c -t

I'll check back later

[khemmerl]

> nbstlutil stlilist
>
> If there is a 3 at the end, the image is complete and is not under SLP control

Results of "nbstlutil stlilist":

V7.0.1 I APP-SRV29_1357349450 Disk-Disk-Tape-Monthly 3
V7.0.1 I APP-SRV29_1357349451 Disk-Disk-Tape-Monthly 3

After just a quick check it appears that all rows end with a '3'.

> bpimagelist -backupid APP-SRV29_1357349450 -l
> should in theory return no entitiy found...

Correct:

bpimagelist -backupid APP-SRV29_1357349450 -l
no entity was found

> Other thing to check would be bpimagelist -l -d 01/04/13 -e 01/06/13

This returns some rows but not any for APP-SRV29.

bpimagelist -l -d 01/04/13 -e 01/06/13

IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434038 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434038 6640 1420506038 0 0 30119133 539 1 1 0 INF-SRV
42_1 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633683 2 0 6
5251 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NULL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 30119133 0 2 6 2 APA290 inf-srv17 65536 4713499 1357413731
11 0 *NULL* 1420506038 0 65546 0 0 0 1 1420506038 1357441439 0 *NULL
* *N
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434037 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434037 4639 1420506037 0 0 13360959 55285 1 1 0 INF-S
RV42L* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633682 2 0
 7246877 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *N
ULL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 13360959 0 2 6 5 APA071 inf-srv17 65536 430877 1357350254 9
 0 *NULL* 1420506037 0 65546 0 0 0 1 1420506037 1357439355 0 *NULL*
*NUL
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434036 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434036 2996 1420506036 0 0 6757899 35919 1 1 0 INF-SR
V42_* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633681 2 0
4714550 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NU
LL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 6757899 0 2 6 4 APA071 inf-srv17 65536 325282 1357350254 10
 0 *NULL* 1420506036 0 65546 0 0 0 1 1420506036 1357437398 0 *NULL*
*NUL
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434035 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434035 6301 1420506035 0 0 14874879 7923 1 1 0 INF-SR
V42_* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633680 2 0
992471 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NUL
L*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 14874879 0 2 6 6 APA071 inf-srv17 65536 639644 1357350254 0
 0 *NULL* 1420506035 0 65546 0 0 0 1 1420506035 1357440916 0 *NULL*
*NUL

I've checked the output of the "bpimmedia -L -stype DiskPool -dp dd670" command and there is no reference to INF-SRV42 in that output so these entries appear to be something different.