cancel
Showing results for 
Search instead for 
Did you mean: 

List Images On DiskPool Before Phase I Import

ianhoskins
Level 4

Is there a way to query a diskpool to get a list of images on it before I initiate a Phase I import? 
I am hesitant to run the Phase I import until I have time to finish the Phase II import as I have been burned in the past where NBU will expire the images off disk because the Phase I sets a 1 week retention on the images in the catalog.
The diskpool I want to query if from a legacy NBU environment that has been decommisioned.
 

I was hoping I could use bpstsinfo to grab the info, but it requires a mediaid which isn't created until Phase I

NBU 7.6.0.2 on RHL

1 ACCEPTED SOLUTION

Accepted Solutions

Amol_Nair
Level 6
Employee
As far as I know the disk media id is generated as soon as you add disk volumes in netbackup.. It wouldn't really wait for you to run phase 1 import.. run the "nbdevquery -listdv" command and it should list the media id that you are looking for.

View solution in original post

24 REPLIES 24

GulzarShaikhAUS
Level 6
Partner Accredited Certified

What is the type of diskpool? Basic/advanced/PureDisk?

mph999
Level 6
Employee Accredited

No, NBU knows nothing about the images, so the only way is to run phase 1 import.

If the storage is basic or advanced disk, you could just run ls -al (Unix) or dir (win) directly on the 'disk' and look at the filenames.

ianhoskins
Level 4

Its IBM-ProtecTier (IBM's version of DataDomain).

An OST device.

ianhoskins
Level 4

Thanks Martin, so after the Phase I import can I extend the retention using bpexpdate on the images so I can have more time to get the Phase II done?

sdo
Moderator
Moderator
Partner    VIP    Certified

I've just done a test:

- backup to basic disk STU on master "A" Windows 2008 R2 SP1 NetBackup Server v7.6.1.2 - backup of some files with 1 day retention

- CIFS/SMB network copy of backup image files to basic disk STU on master "B" Windows 2012 R2 NetBackup Server v7.7

- phase 1 import on master "B"

- on master "B" - bpimagelist -d 01/01/1970 00:00:00 -option ONLY_PRE_IMPORT, to get the image ID, then a bpimagelist -backupid blah -L -option ONLY_PRE_IMPRT, which shows expiring tomorrow.

- on master "B" - bpexpdate -backupid blah -d 31/12/2015 23:59:59

- on master "B" - bpimagelist -backupid blah -L -option ONLY_PRE_IMPORT

...which shows new/correct future date.

.

I would think that catalog behaviour/functionality is irrespective of storage unit type and irrespective of OS upon which NetBackup Server is running, so I think you'll be ok be doing a phase 1 import and then setting a future date (...unless anyone on here knows different?).

ianhoskins
Level 4

Interesting, because I thought Phase I import didn't read in retention, so where did your retention of 1 day come from?  I though after a Phase I there was a generic 1 week retention on the image until a Phase II was done and then the "real" retention was applied.

sdo
Moderator
Moderator
Partner    VIP    Certified

Maybe I missed that - let me test again...

Amol_Nair
Level 6
Employee
If the storage server is added in netbackup then you could try running the bpstsinfo command that could help you list the images i.e. backupid of images present on the ost.. This command is usually used to compare and get a list of images present in Netbackup catalog vs images only on the storage and helps in identifying orphan images. But this could help you get the list of images that you are looking for "bpstsinfo -comparedbandstu -storage_server < storage_server > -stype < server_type > -lsuname < lsu_name >" To get the server_type and the lsu_name you can use the nbdevquery commads

sdo
Moderator
Moderator
Partner    VIP    Certified

Testing with two backup images copies, after phase 1 import, I see:

D:\>bpimagelist -idonly -d 01/01/1970 00:00:00 -option ONLY_PRE_IMPORT -L | findstr /i "^$ ^Backup Expir Copy"

Backup ID:         hserver_1438887221
Backup Time:            06/08/2015 19:53:41 (1438887221)
Expiration Time:        13/08/2015 20:13:17 (1439493197)
Maximum Expiration Time:        13/08/2015 20:13:17 (1439493197)
TIR Expiration:         01/01/1970 00:00:00 (0)
Backup Status:     0
Backup Copy:       Standard (0)
Primary Copy:      1
Resume Expiration:      01/01/1970 00:00:00 (0)
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Copy number:       1
 Expiration Time:       13/08/2015 20:13:17 (1439493197)
 Copy Creation Time:       01/01/1970 00:00:00 (0)
 Copy on hold:     0

Backup ID:         hserver_1438881613
Backup Time:            06/08/2015 18:20:13 (1438881613)
Expiration Time:        13/08/2015 20:13:16 (1439493196)
Maximum Expiration Time:        13/08/2015 20:13:16 (1439493196)
TIR Expiration:         01/01/1970 00:00:00 (0)
Backup Status:     0
Backup Copy:       Standard (0)
Primary Copy:      1
Resume Expiration:      01/01/1970 00:00:00 (0)
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Copy number:       1
 Expiration Time:       13/08/2015 20:13:16 (1439493196)
 Copy Creation Time:       01/01/1970 00:00:00 (0)
 Copy on hold:     0

....so, yes, expiration is one week in the future for the image header and the copy details...

...now setting one image to expire at end of year...

D:\>bpexpdate -backupid hserver_1438881613 -d 31/12/2015 23:59:59
Are you SURE you want to change hserver_1438881613
to expire on      31/12/2015 23:59:59 y/n (n)? y​

...and now the imagelist command shows...

D:\>bpimagelist -idonly -d 01/01/1970 00:00:00 -option ONLY_PRE_IMPORT -L | findstr /i "^$ ^Backup Expir Copy"

Backup ID:         hserver_1438887221
Backup Time:            06/08/2015 19:53:41 (1438887221)
Expiration Time:        13/08/2015 20:13:17 (1439493197)
Maximum Expiration Time:        13/08/2015 20:13:17 (1439493197)
TIR Expiration:         01/01/1970 00:00:00 (0)
Backup Status:     0
Backup Copy:       Standard (0)
Primary Copy:      1
Resume Expiration:      01/01/1970 00:00:00 (0)
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Copy number:       1
 Expiration Time:       13/08/2015 20:13:17 (1439493197)
 Copy Creation Time:       01/01/1970 00:00:00 (0)
 Copy on hold:     0

Backup ID:         hserver_1438881613
Backup Time:            06/08/2015 18:20:13 (1438881613)
Expiration Time:        31/12/2015 23:59:59 (1451606399)
Maximum Expiration Time:        31/12/2015 23:59:59 (1451606399)
TIR Expiration:         01/01/1970 00:00:00 (0)
Backup Status:     0
Backup Copy:       Standard (0)
Primary Copy:      1
Resume Expiration:      01/01/1970 00:00:00 (0)
Remote Expiration Time:      01/01/1970 00:00:00 (0)
Copy number:       1
 Expiration Time:       31/12/2015 23:59:59 (1451606399)
 Copy Creation Time:       01/01/1970 00:00:00 (0)
 Copy on hold:     0

....so it does appear to have worked - but probably best to wait one day and re-check (to see if some background processes 'change' things for us)... and/or wait a week - and see if the unchanged phase 1 image expires, and hopefully the updated phase 1 image will hang around.  Hmm... I have a VM - I'll shutdown, snapshot, reboot, change date to two weeks in the future, stop/restart NetBackup, and see if image still exists...

ianhoskins
Level 4

Yea I looked at that option earlier but unfortunatly -media_id is required.  Since it hasn't done a Phase I yet there is no Media ID defined for it

mph999
Level 6
Employee Accredited

Sorry, not had time to look at this today.

What sdo has shown is my understanding. 1 week is standard retention after phase one import, and it can be extended. 

Amol_Nair
Level 6
Employee
As far as I know the disk media id is generated as soon as you add disk volumes in netbackup.. It wouldn't really wait for you to run phase 1 import.. run the "nbdevquery -listdv" command and it should list the media id that you are looking for.

sdo
Moderator
Moderator
Partner    VIP    Certified

...and with a date change in to the future, and bpimage cleanup:

C:\Users\Administrator>date
The current date is: 06/08/2015
Enter the new date: (dd-mm-yy) 30/08/2015

C:\Users\Administrator>date /t
30/08/2015

C:\Users\Administrator>bpup -f -v
NetBackup 7.7 -- Startup Utility
Starting services
> NetBackup Legacy Network Service
> NetBackup Legacy Network Service -- STARTED
...
> Netbackup Bare Metal Restore Boot Server
> Netbackup Bare Metal Restore Boot Server -- STARTED
Start up completed successfully.

C:\Users\Administrator>bpimagelist -d 01/01/1970 00:00:00 -option ONLY_PRE_IMPORT -idonly
Time:      06/08/2015 19:53:41   ID: hserver_1438887221   FULL (0)
Time:      06/08/2015 18:20:13   ID: hserver_1438881613   FULL (0)

C:\Users\Administrator>bpimage -cleanup -allclients

C:\Users\Administrator>bpimagelist -d 01/01/1970 00:00:00 -option ONLY_PRE_IMPORT -idonly
Time:      06/08/2015 18:20:13   ID: hserver_1438881613   FULL (0)

...which shows that the phase 1 image which had its date set in to the future has survived beyond one week, whereas the other (which did not have a future expiration set) did not survive and was auto-purged from the catalog as an incomplete (un-phased-2) phase 1 image..

sdo
Moderator
Moderator
Partner    VIP    Certified

apologies - the original retention of 1 day was from the original backup - I thought it best to test with a backup with a retention shorter than the auto 1 week retention that phase 1 applies - just to make sure.

sdo
Moderator
Moderator
Partner    VIP    Certified

...

...

...and as a slightly different side topic...

...after reverting the VM to original state/date... how about this... expiring the phase 1 (not yet phase 2) images from the catalog did not remove them from the basic disk STU:

C:\Users\Administrator>bpimagelist -d 01/01/1970 00:00:00 -idonly -option ONLY_PRE_IMPORT
Time:      06/08/2015 19:53:41   ID: hserver_1438887221   FULL (0)
Time:      06/08/2015 18:20:13   ID: hserver_1438881613   FULL (0)

C:\Users\Administrator>bpexpdate -backupid hserver_1438887221 -d 0
Are you SURE you want to delete hserver_1438887221 y/n (n)? y

C:\Users\Administrator>bpexpdate -backupid hserver_1438881613 -d 0
Are you SURE you want to delete hserver_1438881613 y/n (n)? y

C:\Users\Administrator>bpimage -cleanup -allclients

C:\Users\Administrator>bpdbjobs -summary
MASTER SERVER QUEUED REQUEUED ACTIVE SUCCESS PARTSUCC FAILED INCOMP SUSP WAITING_RETRY  TOTAL
master01           0   0      0     77       0        5      0      0    0     82

C:\Users\Administrator>bpimagelist -d 01/01/1970 00:00:00 -idonly -option ONLY_PRE_IMPORT
no entity was found

C:\Users\Administrator>dir F:\NBU_DSU2
Volume in drive F is NBU_Logs
Volume Serial Number is 5223-FA98
Directory of F:\NBU_DSU2
06/08/2015  20:09    <DIR>          .
06/08/2015  20:09    <DIR>          ..
06/08/2015  18:20       118,167,552 hserver_1438881613_C1_F1.1438881613.img
06/08/2015  18:20             1,024 hserver_1438881613_C1_F1.1438881613.info
06/08/2015  18:20               512 hserver_1438881613_C1_HDR.1438881613.img
06/08/2015  18:20             1,024 hserver_1438881613_C1_HDR.1438881613.info
06/08/2015  19:54       118,167,552 hserver_1438887221_C1_F1.1438887221.img
06/08/2015  19:54             1,024 hserver_1438887221_C1_F1.1438887221.info
06/08/2015  19:53               512 hserver_1438887221_C1_HDR.1438887221.img
06/08/2015  19:53             1,024 hserver_1438887221_C1_HDR.1438887221.info
               8 File(s)    236,340,224 bytes
               2 Dir(s)  19,932,418,048 bytes free

...I wonder if OST and/or MSDP STUs would behave in a similar fashion.

ianhoskins
Level 4

I am not sure what the differences would be... all I know is that last time I did this with a different DiskPool and didn't get to the Phase II whithin the week, the images were wiped off disk.  I had a Sev 1 case open with IBM and Symantec on this for about a year while they tried to figure out where the images went and the RCA came down to that Symantec removed the images off disk beause the Phase II did not complete on all the images after the Phase I was complete.... Hence the reason I am leary about running a Phase I without knowing what NBU "can see" on the DiskPool before I run the Phase I.

 

mph999
Level 6
Employee Accredited

After phase 1, NBU will know about the fragments, and most of the info about the backup.

What it doesn't know, is info about the individual files in the backup = the contents of the catalog .f file. This is the long part of an import where NBU looks at each file, and rebuilds the .f file (= list of files in the backup).

It's like a chapter in a book.

Phase one reads the chapter titles (the backup headers) - so a quick flick through the images/ fragments of the image

Phase two settles down for a good old read, where each word in the chapter is a file ...
 

sdo
Moderator
Moderator
Partner    VIP    Certified

Martin, was there ever a bug which caused phase 1 images which reach 7 days age being auto-expired AND also deleted/expunged from disk (i.e. any of BasicDisk, AdvancedDisk, PureDisk, OST) ?

I've had a quick search through the v7.1.0.4 release notes, and the EEB listings for v7.5 and v7.6 and v7.6.1 but I couldn't find anything along those lines... but maybe my searching is not too good :p

mnolan
Level 6
Employee Accredited Certified

Regarding phase 1 imports, those images are only ever held for 7 days before being release again.  Disk based images are more finiky about being overwrite due to the very nature of disk images. As long as your OST devices was told to mark them as garbage, its own garbage cleanup would be responsible for deleting these from disk. If they were already expired, they already are at risk of being overwritten at a sector level (basic disk, adv disk) or being marked as garbage and subject to OST cleanup.

In regards to finding orphaned images

Maybe something by the way of bpstsinfo?

https://support.symantec.com/en_US/article.HOWTO103846.html

-comparedbandstu | -cdas

Compares the image information in the catalog to image information on the storage server physical media. -comparedbandstucompares the specified previous OpenStorage server(s) and volume(s) to the current OpenStorage server and volume.

This will read through that storage (same that phase 1 does) and compares to the EMM records and will list if EMM has something that the storage does not or if the storage has something that EMM does not.

So an example on my local msdp pool on an appliance.

bpstsinfo -cdas -stype PureDisk -ln PureDiskVolume -media_id @aaaao -sn ros5330-01

Results in nothing;

Exits with 

"
MAIN - Verify orphan list
MAIN - Compare DB and Stu complete"

Also some examples to pipe it correctly if you need to output it to a file.

https://support.symantec.com/en_US/article.TECH63720.html