Forum Discussion

Vamika's avatar
Vamika
Level 3
9 years ago

SLP Duplication issue

Hi, We recenty have a space crunch issue on our disk pool. So as a temporary resolution we want to expire the images which are on disk and has already been duplicated to tape by SLP.

We take backups on SLP where in we have set the settings as the backups to stay on disk for a month and then duplicate to tape. Now we have changed the setting in SLP to expire after duplication, But i was wondering that there might be images on disk that have already been duplicated to tape but not yet expired because we have specified it to stay on disk for a month.

Please help with your valuable suggestions.

  • If you change a SLP you will get an additional version or revision of SLP (not seen in GUI). New settings will have affect on new backup images. The images which are finished or in progress or waiting in SLP will not changed. There are using the old revision.

     

    As a (short) temporary solution I suggest:

    - pick up the oldest images on disk

    - find the 2nd copy on tape and set it as pimary copy

    - expire the (now not primary) first copy on disk

     

    This will you give free space. The newest images with higher chance of need for restore are still on disk.

    Are the diskpool of type msdp ? If so it will take a while between expiration and freeing space. The effect of freeing space will be (very) slow depending on your dedup-rate.

     

    ciao

    Martin

  • Maybe write a script to:

    1) Use bpimagelist -idonly to list all images to a.txt

    2) Read a.txt to pick out date and imageid, and write b.txt to contain yyyy-mm-dd imageid

    3) sort b.txt into ascending order

    4) manually review the list, and pick out the lines that you want to expire, and save these to c.txt

    5) write another script to read c.txt, and for each backupid do a bpimagelist -backupid to d.txt, and read d.txt and check to see how many copies it has, and where they are and if another copy exists on tape, then make a note of that copy number, make a number of the copy number for the copy on disk, and then promote the copy number which is on tape to primary copy, and expire the copy number which is on disk.

    .

    I would suggest a dry run on paper first, for one image, to understand the process and the nature of the fields inthe files, before starting to code this in a script.

9 Replies

  • If you change a SLP you will get an additional version or revision of SLP (not seen in GUI). New settings will have affect on new backup images. The images which are finished or in progress or waiting in SLP will not changed. There are using the old revision.

     

    As a (short) temporary solution I suggest:

    - pick up the oldest images on disk

    - find the 2nd copy on tape and set it as pimary copy

    - expire the (now not primary) first copy on disk

     

    This will you give free space. The newest images with higher chance of need for restore are still on disk.

    Are the diskpool of type msdp ? If so it will take a while between expiration and freeing space. The effect of freeing space will be (very) slow depending on your dedup-rate.

     

    ciao

    Martin

  • Thanks Martin for your response. Could you please elaborate on how to perform the below steps, as in commands or how to get the outputs,

  • Maybe write a script to:

    1) Use bpimagelist -idonly to list all images to a.txt

    2) Read a.txt to pick out date and imageid, and write b.txt to contain yyyy-mm-dd imageid

    3) sort b.txt into ascending order

    4) manually review the list, and pick out the lines that you want to expire, and save these to c.txt

    5) write another script to read c.txt, and for each backupid do a bpimagelist -backupid to d.txt, and read d.txt and check to see how many copies it has, and where they are and if another copy exists on tape, then make a note of that copy number, make a number of the copy number for the copy on disk, and then promote the copy number which is on tape to primary copy, and expire the copy number which is on disk.

    .

    I would suggest a dry run on paper first, for one image, to understand the process and the nature of the fields inthe files, before starting to code this in a script.

  • Hi Vamika... your question was an interesting one.  Do you still need to expire any copies of images on disk, but only those images which do actually already have a copy on tape?

    If so, I might have a script for you... and I say "might have" because the script that I have developed might not work in your environment.

    Let me know if you want to try the script.

    BTW, the script will be safe to run... as it will NOT actually expire anything anywhere.  What the script does is produce a file containing commands that you might want to run.

    Here's the header of the script.  Have a read through it, and then let me know if you want a copy of the script to test with.

    REM ******************************************************************************************************
    REM *    File:    expire-oldest-images.bat
    REM *    Desc:    Determine oldest images on disk and/or on tape, and...
    REM *        ...optionally choose to retain any copies on tape...
    REM *        ...and thus potentially select only those copies of images residing upon disk...
    REM *        ...and write a ".zap" file which could be used to expire only the selected copies.
    REM *
    REM *    Vers    Date        Who    Description
    REM *    ----    ----        ---    -----------
    REM *    v0.01    23-JAN-2016    sdo    Initial version.
    REM *    v0.02    24-JAN-2016    sdo    Improved copy number loop.  Added more comments.
    REM ******************************************************************************************************
    REM *    !!! WARNING and DISCLAIMER !!!
    REM *    !!! WARNING and DISCLAIMER !!!
    REM *    !!! WARNING and DISCLAIMER !!!
    REM *
    REM *    Before using this script, be aware of the following points:
    REM *    - Use of this script is entirely at the end user's own risk.
    REM *    - Carefully read the entire header section of this script.  Read all of the notes and make sure
    REM *      you have an understanding of the purpose of this script before using it.
    REM *    - Neither the author of this script, nor Veritas, will accept any responsbility for issues or
    REM *      problems or data loss caused by use, or mis-use, of this script, in either its original form or
    REM *      a modified form.
    REM *    - This script is not endorsed by Veritas.
    REM *    - This script is not supported by Veritas.
    REM *    - This script has not been tested by Veritas.
    REM *    - This script may not be suitable for use in any given NetBackup environment.
    REM *    - This script is furnished on an example basis only.
    REM *    - Whilst every effort has been made (by the author of this script) to produce something that is
    REM *      both useful and viable, be aware that bugs and errors may remain which may cause data loss.
    REM ******************************************************************************************************
    REM *    Sharing
    REM *    -------
    REM *    This script is free to share and modify, as long as this entire header section is kept with
    REM *    the script, and maintained and updated appropriately as any changes are made.
    REM ******************************************************************************************************
    REM *    Notes
    REM *    -----
    REM *    1) This script was developed and tested using:
    REM *        Windows 2008 R2, with NetBackup Server v7.7.1
    REM *        Windows 2012 R2, with NetBackup Server v7.7.1
    REM *       ...but should work with any Windows version since 2003 and any v7.* of NetBackup Server.
    REM *    2) This script is a form of "What if?" analysis script.  This script should be safe to run on any
    REM *       system, for several reasons:
    REM *        i)    this script will not actually exire any backup images.  What this script does...
    REM *            is to identify specific copies of backup images which could be expired, and write
    REM *            the commands to achieve this to a separate file.
    REM *        ii)    The commands will be REM'd out, so even an accidental run of the file created by
    REM *            this script will not expire anything either.
    REM *        iii)    The net effect is... the only way that specific copy numbers of backup images
    REM *            could actually be expired is that the administrator has to make a concious effort
    REM *            to edit the the created file and de-REM it, and rename it, so that it can be run.
    REM *    3) This script tries to detect the date format, and also check that the date enterred by the
    REM *       user is a valid date, and also checks that the entered date is less than today's date.
    REM *    4) This script does not check for, nor handle, SLP incomplete backup images.
    REM *    5) The v7.6.0.3 commands manual shows an example date format "mm/dd/yy" but says actual usage
    REM *      is based upon NetBackup's locale setting.  Luckily, we can make several different dummy
    REM *       calls to bpexpdate, because bpexpdate will return a status 227 if a backup image does
    REM *       does not exist, but will return status 20 if one or more of the input parameters is
    REM *       incorrect.  We know that our parameters are correct, except for the date, so... any
    REM *       status 20 must indicate a bad date format.  Thus, multiple dummy calls to bpexpdate can
    REM *       be used to validate its own parameter formats, and depending upon which call generates
    REM *       a status 227, we can thus determine the correct date format for bpexpdate.
    REM *    6) This script could result in making many calls to bpimagelist and bpverify and so could take
    REM *       some time to run on medium to large NetBackup systems.
    REM *    7) This script was written because it can be even more time consuming to manually check many
    REM *       images, and check that any given copy number applies to the desired storage unit type.
    REM *    8) This script will probably fail if you have 'spaces' or any strange characters in any of your
    REM *       client names, or image names, or storage unit details, or disk pool details, or disk volume
    REM *       details, or tape media IDs.  Basically, if your NetBackup environment is in a bit of a mess
    REM *       then you probably wouldn't want to bother attempting to run this script.
    REM ******************************************************************************************************
    REM *    List Of Files Created By This Script
    REM *    ------------------------------------
    REM *    <script>.csv    filtered images in:  YYYY-MM-DD,image-name,copy-number,stu-type,stu-ident,stu-server
    REM *    <script>.det    the bpimagelist of the last image inspected
    REM *    <script>.eva    the list of images to evaluate for copies to potentially expire
    REM *    <script>.ima    the bpimagelist -idonly of all SLP complete images
    REM *    <script>.lis    list of filtered images before selected date in:  YYYY-MM-DD image-name
    REM *    <script>.log    the log of the most recent run of this script
    REM *    <script>.med    list of unique tape media id discovered
    REM *    <script>.srt    the sorted list of filtered images in:  YYYY-MM-DD image-name
    REM *    <script>.stp    list of all NetBackup disk pools
    REM *    <script>.stu    list of all NetBackup storage units
    REM *    <script>.stv    list of last NetBackup disk volume inspected
    REM *    <script>.ver    output of bpverify -pb for the last image copy inspected
    REM *    <script>.xxx    the accumulated log of all completed runs of this script
    REM *    <script>.zap    the purpose of this script, is to write this file, which is a date filtered,
    REM *            and storage unit type filtered, list of commands which could be used to
    REM *            expire the specific copies of images on selected storage unit types.
    REM *            N.B: the commands written to this file are REM'd out.
    REM ******************************************************************************************************
    REM *    Advice About Using This Script
    REM *    ------------------------------
    REM *    If you do want to use this script, then my advice is to run this script several times, and choose
    REM *    different options each time.  Whilst it may take some time to run on medium to large environments,
    REM *    it is useful to run it several times in order to gain an understanding of how it selects copies of
    REM *    images to potentially expire.  So, check the log output, and check to make sure that it really is
    REM *    only selecting the images that you think that you need/want to expire.
    REM ******************************************************************************************************
    REM *    How To Test This Script At Your Site
    REM *    ------------------------------------
    REM *    1) Identify a media server with a little bit of space.
    REM *    2) Create a basic disk storage unit on that media server.
    REM *    3) Use the management console GUI to identify one of the oldest images that you have.
    REM *    4) Use the management console GUI to duplicate that image to your new basic disk storage unit.
    REM *    5) Run this script in an attempt to select only that image for expiry.
    REM *    Remember, this script won't actually expire anything, all it does is write a list of REM'd out
    REM *    commands to the ".zap" file.
    REM ******************************************************************************************************
    REM *    Ideas For Future Development
    REM *    ----------------------------
    REM *    1) Instead of selecting images only by storage unit type, also select only copies of images
    REM *       residing within specific storage units, or specific volumes pools, or specific volume groups.
    REM ******************************************************************************************************
  • ...and here's an example of running the script...

    ...and at the end you will see the contents of the ".zap" file, which shows that the script only selected two copies to expire, for images which do also already have a copy on tape...

    24/01/2016  9:37:49  
    24/01/2016  9:37:49  
    24/01/2016  9:37:49  ********************************************************
    24/01/2016  9:37:49  ********************************************************
    24/01/2016  9:37:49  Script session details...
    24/01/2016  9:37:49  ...script started:  24/01/2016  9:37:49
    24/01/2016  9:37:49  ...script name:     expire-oldest-images
    24/01/2016  9:37:49  ...script path:     E:\sdo\Scripts\expire-oldest-images\
    24/01/2016  9:37:49  ...script log:      E:\sdo\Scripts\expire-oldest-images\expire-oldest-images.log
    24/01/2016  9:37:49  ...computer name:   HSERVER
    24/01/2016  9:37:49  ...username:        Administrator
    24/01/2016  9:37:49  ...client name:     sdo-mbpro.loca
    24/01/2016  9:37:49  ...session name:    RDP-Tcp#0
    24/01/2016  9:37:49  ...done...
    24/01/2016  9:37:49  
    24/01/2016  9:37:49  Checking for presence of NetBackup...
    24/01/2016  9:37:49  ...found NetBackup ok...
    24/01/2016  9:37:49  ...client name:   hserver
    24/01/2016  9:37:49  ...type:          NetBackup Master Server
    24/01/2016  9:37:49  ...path:          C:\Program Files\Veritas\
    24/01/2016  9:37:49  ...done...
    24/01/2016  9:37:49  
    24/01/2016  9:37:49  Determining storage units and types...
    24/01/2016  9:37:49  ...found:  hserver-hcart2  Tape  
    24/01/2016  9:37:49  ...found:  hserver-catalog  Disk  Basic
    24/01/2016  9:37:49  ...found:  hserver-dsu  Disk  Basic
    24/01/2016  9:37:49  ...found:  hserver-msdp  Disk  DiskPool
    24/01/2016  9:37:49  ...done...
    24/01/2016  9:37:49  
    24/01/2016  9:37:49  Listing the `1` discovered disk pools...
    24/01/2016  9:37:49     Storage Unit                Disk Pool Name              Disk Pool Type    Disk Pool Media ID
    24/01/2016  9:37:49     ------------                --------------              --------------    ------------------
    24/01/2016  9:37:49     hserver-msdp                hserver-msdp                PureDisk          aaaal
    24/01/2016  9:37:49  ...done...
    24/01/2016  9:37:49  
    24/01/2016  9:37:49  Determining date format for bpexpdate...
    24/01/2016  9:37:50  ...bpexpdate format is `dd/mm/yyyy`...
    24/01/2016  9:37:50  ...bpexpdate date format is `UK`...
    24/01/2016  9:37:50  ...year has `4` digits...
    24/01/2016  9:37:50  ...done...
    24/01/2016  9:37:50  
    24/01/2016  9:37:50  N.B:  You now have a choice to make:
    24/01/2016  9:37:50  ...enter YES, to     check for duplicates, and only expire images on disk which do     have a copy on tape...
    24/01/2016  9:37:50  ...enter NO,  to NOT check for duplicates, and      expire images on disk which do NOT have a copy on tape...
    24/01/2016  9:37:50  
    24/01/2016  9:37:50  _Check for duplicates on tape?  (YES,NO,exit) [exit]  : 
    24/01/2016  9:37:57  ...user entered `YES`...
    24/01/2016  9:37:57  ...done...
    24/01/2016  9:37:57  
    24/01/2016  9:37:57  _Enter cutoff date in format `YYYY-MM-DD` [exit]  : 
    24/01/2016  9:38:08  ...user entered `2015-09-01`...
    24/01/2016  9:38:08  ...checking whether cuttoff date is a valid date...
    24/01/2016  9:38:08  ...cutoff date of `2015-09-01` appears to be valid...
    24/01/2016  9:38:08  ...comparing to today's date...
    24/01/2016  9:38:08  ...cutoff date appears to be a previous date...
    24/01/2016  9:38:08  ...done...
    24/01/2016  9:38:08  
    24/01/2016  9:38:08  
    24/01/2016  9:38:08  Now choose which image copies to `Expire`...
    24/01/2016  9:38:08     1.   Expire all copies, no matter whether they are on disk or tape
    24/01/2016  9:38:08     2.   Expire only copies that reside upon any form of disk, i.e. BasicDisk, AdvancedDisk, PureDisk, DataDomain
    24/01/2016  9:38:08     3.   Expire only copies that reside upon any storage unit of BasicDisk
    24/01/2016  9:38:08     4.   Expire only copies that reside upon any storage unit of AdvancedDisk
    24/01/2016  9:38:08     5.   Expire only copies that reside upon any storage unit of PureDisk
    24/01/2016  9:38:08     6.   Expire only copies that reside upon any storage unit of DataDomain
    24/01/2016  9:38:08     7.   Expire only copies that reside on any tape
    24/01/2016  9:38:08  
    24/01/2016  9:38:08  _Which option (1-7,EXIT) [exit]  : 
    24/01/2016  9:38:12  ...user entered `3`...
    24/01/2016  9:38:12  ...script will select image copies on storage of `BasicDisk`...
    24/01/2016  9:38:12  ...done...
    24/01/2016  9:38:12  
    24/01/2016  9:38:12  Listing backup images...
    24/01/2016  9:38:12  ...counting images...
    24/01/2016  9:38:12  ...found `60` backup images...
    24/01/2016  9:38:12  ...done...
    24/01/2016  9:38:12  
    24/01/2016  9:38:12  Filtering image list into YYYY-MM-DD order...
    24/01/2016  9:38:12  ...image:  2015-06-13  sdo-mbpro_1434213287
    24/01/2016  9:38:12  ...image:  2015-06-12  zzz-mbpro_1434105353
    24/01/2016  9:38:12  ...image:  2015-06-09  hserver_1433834317
    24/01/2016  9:38:12  ...image:  2015-06-08  hserver_1433767241
    24/01/2016  9:38:12  ...image:  2014-10-26  hserver_1414334444
    24/01/2016  9:38:12  ...image:  2014-08-06  hserver_1407307479
    24/01/2016  9:38:12  ...image:  2014-08-05  hserver_1407235982
    24/01/2016  9:38:12  ...image:  2015-07-09  macosx10-10-4-test_1436462856
    24/01/2016  9:38:12  ...image:  2015-07-09  macosx10-10-4-test_1436451608
    24/01/2016  9:38:12  ...image:  2015-06-14  win7pro_1434316888
    24/01/2016  9:38:12  ...image:  2015-06-14  win7pro_1434315549
    24/01/2016  9:38:12  ...sorting...
    24/01/2016  9:38:12  ...count of images in source list:  60
    24/01/2016  9:38:12  ...count of images before cutoff:   11
    24/01/2016  9:38:12  ...done...
    24/01/2016  9:38:12  
    24/01/2016  9:38:12  Reporting number of copies of each image...
    24/01/2016  9:38:13  ...image:  2014-08-05  hserver_1407235982                  copies: 1
    24/01/2016  9:38:13  ...image:  2014-08-06  hserver_1407307479                  copies: 1
    24/01/2016  9:38:14  ...image:  2014-10-26  hserver_1414334444                  copies: 1
    24/01/2016  9:38:14  ...image:  2015-06-08  hserver_1433767241                  copies: 1
    24/01/2016  9:38:14  ...image:  2015-06-09  hserver_1433834317                  copies: 1
    24/01/2016  9:38:15  ...image:  2015-06-12  zzz-mbpro_1434105353                copies: 1
    24/01/2016  9:38:15  ...image:  2015-06-13  sdo-mbpro_1434213287                copies: 1
    24/01/2016  9:38:16  ...image:  2015-06-14  win7pro_1434315549                  copies: 2
    24/01/2016  9:38:16  ...image:  2015-06-14  win7pro_1434316888                  copies: 2
    24/01/2016  9:38:16  ...image:  2015-07-09  macosx10-10-4-test_1436451608       copies: 2
    24/01/2016  9:38:17  ...image:  2015-07-09  macosx10-10-4-test_1436462856       copies: 2
    24/01/2016  9:38:17  ...images checked:      11
    24/01/2016  9:38:17  ...images to evaluate:  4
    24/01/2016  9:38:17  ...done...
    24/01/2016  9:38:17  
    24/01/2016  9:38:17  Selecting image copies on storage of `BasicDisk`...
    24/01/2016  9:38:17  
    24/01/2016  9:38:17  ...checking image `1` of `4`, image:  win7pro_1434315549
    24/01/2016  9:38:17  ......number of copies:  2
    24/01/2016  9:38:17  ......copy number list:  1 2
    24/01/2016  9:38:18  ......found image copy:  1       on:  BasicDisk
    24/01/2016  9:38:18  ......found image copy:  2       on:  Tape
    24/01/2016  9:38:18  
    24/01/2016  9:38:18  ...checking image `2` of `4`, image:  win7pro_1434316888
    24/01/2016  9:38:19  ......number of copies:  2
    24/01/2016  9:38:19  ......copy number list:  1 2
    24/01/2016  9:38:19  ......found image copy:  1       on:  BasicDisk
    24/01/2016  9:38:20  ......found image copy:  2       on:  Tape
    24/01/2016  9:38:20  
    24/01/2016  9:38:20  ...checking image `3` of `4`, image:  macosx10-10-4-test_1436451608
    24/01/2016  9:38:20  ......number of copies:  2
    24/01/2016  9:38:20  ......copy number list:  1 2
    24/01/2016  9:38:20  ......found image copy:  1       on:  PureDisk
    24/01/2016  9:38:21  ......found image copy:  2       on:  Tape
    24/01/2016  9:38:21  
    24/01/2016  9:38:21  ...checking image `4` of `4`, image:  macosx10-10-4-test_1436462856
    24/01/2016  9:38:21  ......number of copies:  2
    24/01/2016  9:38:21  ......copy number list:  1 2
    24/01/2016  9:38:22  ......found image copy:  1       on:  PureDisk
    24/01/2016  9:38:22  ......found image copy:  2       on:  Tape
    24/01/2016  9:38:22  
    24/01/2016  9:38:22  ...finished evaluating images...
    24/01/2016  9:38:22  ...evaluated images:  4
    24/01/2016  9:38:22  ...copies to zap:     2
    24/01/2016  9:38:22  ...done...
    24/01/2016  9:38:22  
    24/01/2016  9:38:22  Script exiting...
    24/01/2016  9:38:22  ...script started:  24/01/2016  9:37:49
    24/01/2016  9:38:22  ...script ended:    24/01/2016  9:38:22
    
    E:\sdo\Scripts\expire-oldest-images> type expire-oldest-images.zap
    REM bpexpdate -backupid win7pro_1434315549 -cn 1 -d 0 -force
    REM bpexpdate -backupid win7pro_1434316888 -cn 1 -d 0 -force
  • Now I'm nervous... :)     I'll PM you with it.   Would love to have it tested by a proper NBU guru!   Please?

  • Hi, sorry for late reply. We kinds resolve the space crunch by deleting few images which were about to expire/not needed.

    But I really want to try out the script.

  • Hi Vamika - ok - I'll see if I can post it as a "download" here in the forum, and I'll send you a link if/when it gets published.