01-21-2016 05:41 AM
Hi, We recenty have a space crunch issue on our disk pool. So as a temporary resolution we want to expire the images which are on disk and has already been duplicated to tape by SLP.
We take backups on SLP where in we have set the settings as the backups to stay on disk for a month and then duplicate to tape. Now we have changed the setting in SLP to expire after duplication, But i was wondering that there might be images on disk that have already been duplicated to tape but not yet expired because we have specified it to stay on disk for a month.
Please help with your valuable suggestions.
Solved! Go to Solution.
01-21-2016 05:59 AM
If you change a SLP you will get an additional version or revision of SLP (not seen in GUI). New settings will have affect on new backup images. The images which are finished or in progress or waiting in SLP will not changed. There are using the old revision.
As a (short) temporary solution I suggest:
- pick up the oldest images on disk
- find the 2nd copy on tape and set it as pimary copy
- expire the (now not primary) first copy on disk
This will you give free space. The newest images with higher chance of need for restore are still on disk.
Are the diskpool of type msdp ? If so it will take a while between expiration and freeing space. The effect of freeing space will be (very) slow depending on your dedup-rate.
ciao
Martin
01-21-2016 03:58 PM
Maybe write a script to:
1) Use bpimagelist -idonly to list all images to a.txt
2) Read a.txt to pick out date and imageid, and write b.txt to contain yyyy-mm-dd imageid
3) sort b.txt into ascending order
4) manually review the list, and pick out the lines that you want to expire, and save these to c.txt
5) write another script to read c.txt, and for each backupid do a bpimagelist -backupid to d.txt, and read d.txt and check to see how many copies it has, and where they are and if another copy exists on tape, then make a note of that copy number, make a number of the copy number for the copy on disk, and then promote the copy number which is on tape to primary copy, and expire the copy number which is on disk.
.
I would suggest a dry run on paper first, for one image, to understand the process and the nature of the fields inthe files, before starting to code this in a script.
01-21-2016 05:59 AM
If you change a SLP you will get an additional version or revision of SLP (not seen in GUI). New settings will have affect on new backup images. The images which are finished or in progress or waiting in SLP will not changed. There are using the old revision.
As a (short) temporary solution I suggest:
- pick up the oldest images on disk
- find the 2nd copy on tape and set it as pimary copy
- expire the (now not primary) first copy on disk
This will you give free space. The newest images with higher chance of need for restore are still on disk.
Are the diskpool of type msdp ? If so it will take a while between expiration and freeing space. The effect of freeing space will be (very) slow depending on your dedup-rate.
ciao
Martin
01-21-2016 06:21 AM
Thanks Martin for your response. Could you please elaborate on how to perform the below steps, as in commands or how to get the outputs,
01-21-2016 03:58 PM
Maybe write a script to:
1) Use bpimagelist -idonly to list all images to a.txt
2) Read a.txt to pick out date and imageid, and write b.txt to contain yyyy-mm-dd imageid
3) sort b.txt into ascending order
4) manually review the list, and pick out the lines that you want to expire, and save these to c.txt
5) write another script to read c.txt, and for each backupid do a bpimagelist -backupid to d.txt, and read d.txt and check to see how many copies it has, and where they are and if another copy exists on tape, then make a note of that copy number, make a number of the copy number for the copy on disk, and then promote the copy number which is on tape to primary copy, and expire the copy number which is on disk.
.
I would suggest a dry run on paper first, for one image, to understand the process and the nature of the fields inthe files, before starting to code this in a script.
01-24-2016 01:28 AM
Hi Vamika... your question was an interesting one. Do you still need to expire any copies of images on disk, but only those images which do actually already have a copy on tape?
If so, I might have a script for you... and I say "might have" because the script that I have developed might not work in your environment.
Let me know if you want to try the script.
BTW, the script will be safe to run... as it will NOT actually expire anything anywhere. What the script does is produce a file containing commands that you might want to run.
Here's the header of the script. Have a read through it, and then let me know if you want a copy of the script to test with.
REM ****************************************************************************************************** REM * File: expire-oldest-images.bat REM * Desc: Determine oldest images on disk and/or on tape, and... REM * ...optionally choose to retain any copies on tape... REM * ...and thus potentially select only those copies of images residing upon disk... REM * ...and write a ".zap" file which could be used to expire only the selected copies. REM * REM * Vers Date Who Description REM * ---- ---- --- ----------- REM * v0.01 23-JAN-2016 sdo Initial version. REM * v0.02 24-JAN-2016 sdo Improved copy number loop. Added more comments. REM ****************************************************************************************************** REM * !!! WARNING and DISCLAIMER !!! REM * !!! WARNING and DISCLAIMER !!! REM * !!! WARNING and DISCLAIMER !!! REM * REM * Before using this script, be aware of the following points: REM * - Use of this script is entirely at the end user's own risk. REM * - Carefully read the entire header section of this script. Read all of the notes and make sure REM * you have an understanding of the purpose of this script before using it. REM * - Neither the author of this script, nor Veritas, will accept any responsbility for issues or REM * problems or data loss caused by use, or mis-use, of this script, in either its original form or REM * a modified form. REM * - This script is not endorsed by Veritas. REM * - This script is not supported by Veritas. REM * - This script has not been tested by Veritas. REM * - This script may not be suitable for use in any given NetBackup environment. REM * - This script is furnished on an example basis only. REM * - Whilst every effort has been made (by the author of this script) to produce something that is REM * both useful and viable, be aware that bugs and errors may remain which may cause data loss. REM ****************************************************************************************************** REM * Sharing REM * ------- REM * This script is free to share and modify, as long as this entire header section is kept with REM * the script, and maintained and updated appropriately as any changes are made. REM ****************************************************************************************************** REM * Notes REM * ----- REM * 1) This script was developed and tested using: REM * Windows 2008 R2, with NetBackup Server v7.7.1 REM * Windows 2012 R2, with NetBackup Server v7.7.1 REM * ...but should work with any Windows version since 2003 and any v7.* of NetBackup Server. REM * 2) This script is a form of "What if?" analysis script. This script should be safe to run on any REM * system, for several reasons: REM * i) this script will not actually exire any backup images. What this script does... REM * is to identify specific copies of backup images which could be expired, and write REM * the commands to achieve this to a separate file. REM * ii) The commands will be REM'd out, so even an accidental run of the file created by REM * this script will not expire anything either. REM * iii) The net effect is... the only way that specific copy numbers of backup images REM * could actually be expired is that the administrator has to make a concious effort REM * to edit the the created file and de-REM it, and rename it, so that it can be run. REM * 3) This script tries to detect the date format, and also check that the date enterred by the REM * user is a valid date, and also checks that the entered date is less than today's date. REM * 4) This script does not check for, nor handle, SLP incomplete backup images. REM * 5) The v7.6.0.3 commands manual shows an example date format "mm/dd/yy" but says actual usage REM * is based upon NetBackup's locale setting. Luckily, we can make several different dummy REM * calls to bpexpdate, because bpexpdate will return a status 227 if a backup image does REM * does not exist, but will return status 20 if one or more of the input parameters is REM * incorrect. We know that our parameters are correct, except for the date, so... any REM * status 20 must indicate a bad date format. Thus, multiple dummy calls to bpexpdate can REM * be used to validate its own parameter formats, and depending upon which call generates REM * a status 227, we can thus determine the correct date format for bpexpdate. REM * 6) This script could result in making many calls to bpimagelist and bpverify and so could take REM * some time to run on medium to large NetBackup systems. REM * 7) This script was written because it can be even more time consuming to manually check many REM * images, and check that any given copy number applies to the desired storage unit type. REM * 8) This script will probably fail if you have 'spaces' or any strange characters in any of your REM * client names, or image names, or storage unit details, or disk pool details, or disk volume REM * details, or tape media IDs. Basically, if your NetBackup environment is in a bit of a mess REM * then you probably wouldn't want to bother attempting to run this script. REM ****************************************************************************************************** REM * List Of Files Created By This Script REM * ------------------------------------ REM * <script>.csv filtered images in: YYYY-MM-DD,image-name,copy-number,stu-type,stu-ident,stu-server REM * <script>.det the bpimagelist of the last image inspected REM * <script>.eva the list of images to evaluate for copies to potentially expire REM * <script>.ima the bpimagelist -idonly of all SLP complete images REM * <script>.lis list of filtered images before selected date in: YYYY-MM-DD image-name REM * <script>.log the log of the most recent run of this script REM * <script>.med list of unique tape media id discovered REM * <script>.srt the sorted list of filtered images in: YYYY-MM-DD image-name REM * <script>.stp list of all NetBackup disk pools REM * <script>.stu list of all NetBackup storage units REM * <script>.stv list of last NetBackup disk volume inspected REM * <script>.ver output of bpverify -pb for the last image copy inspected REM * <script>.xxx the accumulated log of all completed runs of this script REM * <script>.zap the purpose of this script, is to write this file, which is a date filtered, REM * and storage unit type filtered, list of commands which could be used to REM * expire the specific copies of images on selected storage unit types. REM * N.B: the commands written to this file are REM'd out. REM ****************************************************************************************************** REM * Advice About Using This Script REM * ------------------------------ REM * If you do want to use this script, then my advice is to run this script several times, and choose REM * different options each time. Whilst it may take some time to run on medium to large environments, REM * it is useful to run it several times in order to gain an understanding of how it selects copies of REM * images to potentially expire. So, check the log output, and check to make sure that it really is REM * only selecting the images that you think that you need/want to expire. REM ****************************************************************************************************** REM * How To Test This Script At Your Site REM * ------------------------------------ REM * 1) Identify a media server with a little bit of space. REM * 2) Create a basic disk storage unit on that media server. REM * 3) Use the management console GUI to identify one of the oldest images that you have. REM * 4) Use the management console GUI to duplicate that image to your new basic disk storage unit. REM * 5) Run this script in an attempt to select only that image for expiry. REM * Remember, this script won't actually expire anything, all it does is write a list of REM'd out REM * commands to the ".zap" file. REM ****************************************************************************************************** REM * Ideas For Future Development REM * ---------------------------- REM * 1) Instead of selecting images only by storage unit type, also select only copies of images REM * residing within specific storage units, or specific volumes pools, or specific volume groups. REM ******************************************************************************************************
01-24-2016 01:47 AM
...and here's an example of running the script...
...and at the end you will see the contents of the ".zap" file, which shows that the script only selected two copies to expire, for images which do also already have a copy on tape...
24/01/2016 9:37:49 24/01/2016 9:37:49 24/01/2016 9:37:49 ******************************************************** 24/01/2016 9:37:49 ******************************************************** 24/01/2016 9:37:49 Script session details... 24/01/2016 9:37:49 ...script started: 24/01/2016 9:37:49 24/01/2016 9:37:49 ...script name: expire-oldest-images 24/01/2016 9:37:49 ...script path: E:\sdo\Scripts\expire-oldest-images\ 24/01/2016 9:37:49 ...script log: E:\sdo\Scripts\expire-oldest-images\expire-oldest-images.log 24/01/2016 9:37:49 ...computer name: HSERVER 24/01/2016 9:37:49 ...username: Administrator 24/01/2016 9:37:49 ...client name: sdo-mbpro.loca 24/01/2016 9:37:49 ...session name: RDP-Tcp#0 24/01/2016 9:37:49 ...done... 24/01/2016 9:37:49 24/01/2016 9:37:49 Checking for presence of NetBackup... 24/01/2016 9:37:49 ...found NetBackup ok... 24/01/2016 9:37:49 ...client name: hserver 24/01/2016 9:37:49 ...type: NetBackup Master Server 24/01/2016 9:37:49 ...path: C:\Program Files\Veritas\ 24/01/2016 9:37:49 ...done... 24/01/2016 9:37:49 24/01/2016 9:37:49 Determining storage units and types... 24/01/2016 9:37:49 ...found: hserver-hcart2 Tape 24/01/2016 9:37:49 ...found: hserver-catalog Disk Basic 24/01/2016 9:37:49 ...found: hserver-dsu Disk Basic 24/01/2016 9:37:49 ...found: hserver-msdp Disk DiskPool 24/01/2016 9:37:49 ...done... 24/01/2016 9:37:49 24/01/2016 9:37:49 Listing the `1` discovered disk pools... 24/01/2016 9:37:49 Storage Unit Disk Pool Name Disk Pool Type Disk Pool Media ID 24/01/2016 9:37:49 ------------ -------------- -------------- ------------------ 24/01/2016 9:37:49 hserver-msdp hserver-msdp PureDisk aaaal 24/01/2016 9:37:49 ...done... 24/01/2016 9:37:49 24/01/2016 9:37:49 Determining date format for bpexpdate... 24/01/2016 9:37:50 ...bpexpdate format is `dd/mm/yyyy`... 24/01/2016 9:37:50 ...bpexpdate date format is `UK`... 24/01/2016 9:37:50 ...year has `4` digits... 24/01/2016 9:37:50 ...done... 24/01/2016 9:37:50 24/01/2016 9:37:50 N.B: You now have a choice to make: 24/01/2016 9:37:50 ...enter YES, to check for duplicates, and only expire images on disk which do have a copy on tape... 24/01/2016 9:37:50 ...enter NO, to NOT check for duplicates, and expire images on disk which do NOT have a copy on tape... 24/01/2016 9:37:50 24/01/2016 9:37:50 _Check for duplicates on tape? (YES,NO,exit) [exit] : 24/01/2016 9:37:57 ...user entered `YES`... 24/01/2016 9:37:57 ...done... 24/01/2016 9:37:57 24/01/2016 9:37:57 _Enter cutoff date in format `YYYY-MM-DD` [exit] : 24/01/2016 9:38:08 ...user entered `2015-09-01`... 24/01/2016 9:38:08 ...checking whether cuttoff date is a valid date... 24/01/2016 9:38:08 ...cutoff date of `2015-09-01` appears to be valid... 24/01/2016 9:38:08 ...comparing to today's date... 24/01/2016 9:38:08 ...cutoff date appears to be a previous date... 24/01/2016 9:38:08 ...done... 24/01/2016 9:38:08 24/01/2016 9:38:08 24/01/2016 9:38:08 Now choose which image copies to `Expire`... 24/01/2016 9:38:08 1. Expire all copies, no matter whether they are on disk or tape 24/01/2016 9:38:08 2. Expire only copies that reside upon any form of disk, i.e. BasicDisk, AdvancedDisk, PureDisk, DataDomain 24/01/2016 9:38:08 3. Expire only copies that reside upon any storage unit of BasicDisk 24/01/2016 9:38:08 4. Expire only copies that reside upon any storage unit of AdvancedDisk 24/01/2016 9:38:08 5. Expire only copies that reside upon any storage unit of PureDisk 24/01/2016 9:38:08 6. Expire only copies that reside upon any storage unit of DataDomain 24/01/2016 9:38:08 7. Expire only copies that reside on any tape 24/01/2016 9:38:08 24/01/2016 9:38:08 _Which option (1-7,EXIT) [exit] : 24/01/2016 9:38:12 ...user entered `3`... 24/01/2016 9:38:12 ...script will select image copies on storage of `BasicDisk`... 24/01/2016 9:38:12 ...done... 24/01/2016 9:38:12 24/01/2016 9:38:12 Listing backup images... 24/01/2016 9:38:12 ...counting images... 24/01/2016 9:38:12 ...found `60` backup images... 24/01/2016 9:38:12 ...done... 24/01/2016 9:38:12 24/01/2016 9:38:12 Filtering image list into YYYY-MM-DD order... 24/01/2016 9:38:12 ...image: 2015-06-13 sdo-mbpro_1434213287 24/01/2016 9:38:12 ...image: 2015-06-12 zzz-mbpro_1434105353 24/01/2016 9:38:12 ...image: 2015-06-09 hserver_1433834317 24/01/2016 9:38:12 ...image: 2015-06-08 hserver_1433767241 24/01/2016 9:38:12 ...image: 2014-10-26 hserver_1414334444 24/01/2016 9:38:12 ...image: 2014-08-06 hserver_1407307479 24/01/2016 9:38:12 ...image: 2014-08-05 hserver_1407235982 24/01/2016 9:38:12 ...image: 2015-07-09 macosx10-10-4-test_1436462856 24/01/2016 9:38:12 ...image: 2015-07-09 macosx10-10-4-test_1436451608 24/01/2016 9:38:12 ...image: 2015-06-14 win7pro_1434316888 24/01/2016 9:38:12 ...image: 2015-06-14 win7pro_1434315549 24/01/2016 9:38:12 ...sorting... 24/01/2016 9:38:12 ...count of images in source list: 60 24/01/2016 9:38:12 ...count of images before cutoff: 11 24/01/2016 9:38:12 ...done... 24/01/2016 9:38:12 24/01/2016 9:38:12 Reporting number of copies of each image... 24/01/2016 9:38:13 ...image: 2014-08-05 hserver_1407235982 copies: 1 24/01/2016 9:38:13 ...image: 2014-08-06 hserver_1407307479 copies: 1 24/01/2016 9:38:14 ...image: 2014-10-26 hserver_1414334444 copies: 1 24/01/2016 9:38:14 ...image: 2015-06-08 hserver_1433767241 copies: 1 24/01/2016 9:38:14 ...image: 2015-06-09 hserver_1433834317 copies: 1 24/01/2016 9:38:15 ...image: 2015-06-12 zzz-mbpro_1434105353 copies: 1 24/01/2016 9:38:15 ...image: 2015-06-13 sdo-mbpro_1434213287 copies: 1 24/01/2016 9:38:16 ...image: 2015-06-14 win7pro_1434315549 copies: 2 24/01/2016 9:38:16 ...image: 2015-06-14 win7pro_1434316888 copies: 2 24/01/2016 9:38:16 ...image: 2015-07-09 macosx10-10-4-test_1436451608 copies: 2 24/01/2016 9:38:17 ...image: 2015-07-09 macosx10-10-4-test_1436462856 copies: 2 24/01/2016 9:38:17 ...images checked: 11 24/01/2016 9:38:17 ...images to evaluate: 4 24/01/2016 9:38:17 ...done... 24/01/2016 9:38:17 24/01/2016 9:38:17 Selecting image copies on storage of `BasicDisk`... 24/01/2016 9:38:17 24/01/2016 9:38:17 ...checking image `1` of `4`, image: win7pro_1434315549 24/01/2016 9:38:17 ......number of copies: 2 24/01/2016 9:38:17 ......copy number list: 1 2 24/01/2016 9:38:18 ......found image copy: 1 on: BasicDisk 24/01/2016 9:38:18 ......found image copy: 2 on: Tape 24/01/2016 9:38:18 24/01/2016 9:38:18 ...checking image `2` of `4`, image: win7pro_1434316888 24/01/2016 9:38:19 ......number of copies: 2 24/01/2016 9:38:19 ......copy number list: 1 2 24/01/2016 9:38:19 ......found image copy: 1 on: BasicDisk 24/01/2016 9:38:20 ......found image copy: 2 on: Tape 24/01/2016 9:38:20 24/01/2016 9:38:20 ...checking image `3` of `4`, image: macosx10-10-4-test_1436451608 24/01/2016 9:38:20 ......number of copies: 2 24/01/2016 9:38:20 ......copy number list: 1 2 24/01/2016 9:38:20 ......found image copy: 1 on: PureDisk 24/01/2016 9:38:21 ......found image copy: 2 on: Tape 24/01/2016 9:38:21 24/01/2016 9:38:21 ...checking image `4` of `4`, image: macosx10-10-4-test_1436462856 24/01/2016 9:38:21 ......number of copies: 2 24/01/2016 9:38:21 ......copy number list: 1 2 24/01/2016 9:38:22 ......found image copy: 1 on: PureDisk 24/01/2016 9:38:22 ......found image copy: 2 on: Tape 24/01/2016 9:38:22 24/01/2016 9:38:22 ...finished evaluating images... 24/01/2016 9:38:22 ...evaluated images: 4 24/01/2016 9:38:22 ...copies to zap: 2 24/01/2016 9:38:22 ...done... 24/01/2016 9:38:22 24/01/2016 9:38:22 Script exiting... 24/01/2016 9:38:22 ...script started: 24/01/2016 9:37:49 24/01/2016 9:38:22 ...script ended: 24/01/2016 9:38:22 E:\sdo\Scripts\expire-oldest-images> type expire-oldest-images.zap REM bpexpdate -backupid win7pro_1434315549 -cn 1 -d 0 -force REM bpexpdate -backupid win7pro_1434316888 -cn 1 -d 0 -force
01-24-2016 07:36 AM
01-24-2016 12:44 PM
Now I'm nervous... :) I'll PM you with it. Would love to have it tested by a proper NBU guru! Please?
01-27-2016 04:27 AM
Hi, sorry for late reply. We kinds resolve the space crunch by deleting few images which were about to expire/not needed.
But I really want to try out the script.
01-27-2016 07:44 AM
Hi Vamika - ok - I'll see if I can post it as a "download" here in the forum, and I'll send you a link if/when it gets published.