cancel
Showing results for 
Search instead for 
Did you mean: 

SLP Duplication issue

Vamika
Level 3

Hi, We recenty have a space crunch issue on our disk pool. So as a temporary resolution we want to expire the images which are on disk and has already been duplicated to tape by SLP.

We take backups on SLP where in we have set the settings as the backups to stay on disk for a month and then duplicate to tape. Now we have changed the setting in SLP to expire after duplication, But i was wondering that there might be images on disk that have already been duplicated to tape but not yet expired because we have specified it to stay on disk for a month.

Please help with your valuable suggestions.

2 ACCEPTED SOLUTIONS

Accepted Solutions

tunix2k
Level 5
Partner Accredited

If you change a SLP you will get an additional version or revision of SLP (not seen in GUI). New settings will have affect on new backup images. The images which are finished or in progress or waiting in SLP will not changed. There are using the old revision.

 

As a (short) temporary solution I suggest:

- pick up the oldest images on disk

- find the 2nd copy on tape and set it as pimary copy

- expire the (now not primary) first copy on disk

 

This will you give free space. The newest images with higher chance of need for restore are still on disk.

Are the diskpool of type msdp ? If so it will take a while between expiration and freeing space. The effect of freeing space will be (very) slow depending on your dedup-rate.

 

ciao

Martin

View solution in original post

sdo
Moderator
Moderator
Partner    VIP    Certified

Maybe write a script to:

1) Use bpimagelist -idonly to list all images to a.txt

2) Read a.txt to pick out date and imageid, and write b.txt to contain yyyy-mm-dd imageid

3) sort b.txt into ascending order

4) manually review the list, and pick out the lines that you want to expire, and save these to c.txt

5) write another script to read c.txt, and for each backupid do a bpimagelist -backupid to d.txt, and read d.txt and check to see how many copies it has, and where they are and if another copy exists on tape, then make a note of that copy number, make a number of the copy number for the copy on disk, and then promote the copy number which is on tape to primary copy, and expire the copy number which is on disk.

.

I would suggest a dry run on paper first, for one image, to understand the process and the nature of the fields inthe files, before starting to code this in a script.

View solution in original post

9 REPLIES 9

tunix2k
Level 5
Partner Accredited

If you change a SLP you will get an additional version or revision of SLP (not seen in GUI). New settings will have affect on new backup images. The images which are finished or in progress or waiting in SLP will not changed. There are using the old revision.

 

As a (short) temporary solution I suggest:

- pick up the oldest images on disk

- find the 2nd copy on tape and set it as pimary copy

- expire the (now not primary) first copy on disk

 

This will you give free space. The newest images with higher chance of need for restore are still on disk.

Are the diskpool of type msdp ? If so it will take a while between expiration and freeing space. The effect of freeing space will be (very) slow depending on your dedup-rate.

 

ciao

Martin

Vamika
Level 3

Thanks Martin for your response. Could you please elaborate on how to perform the below steps, as in commands or how to get the outputs,

sdo
Moderator
Moderator
Partner    VIP    Certified

Maybe write a script to:

1) Use bpimagelist -idonly to list all images to a.txt

2) Read a.txt to pick out date and imageid, and write b.txt to contain yyyy-mm-dd imageid

3) sort b.txt into ascending order

4) manually review the list, and pick out the lines that you want to expire, and save these to c.txt

5) write another script to read c.txt, and for each backupid do a bpimagelist -backupid to d.txt, and read d.txt and check to see how many copies it has, and where they are and if another copy exists on tape, then make a note of that copy number, make a number of the copy number for the copy on disk, and then promote the copy number which is on tape to primary copy, and expire the copy number which is on disk.

.

I would suggest a dry run on paper first, for one image, to understand the process and the nature of the fields inthe files, before starting to code this in a script.

sdo
Moderator
Moderator
Partner    VIP    Certified

Hi Vamika... your question was an interesting one.  Do you still need to expire any copies of images on disk, but only those images which do actually already have a copy on tape?

If so, I might have a script for you... and I say "might have" because the script that I have developed might not work in your environment.

Let me know if you want to try the script.

BTW, the script will be safe to run... as it will NOT actually expire anything anywhere.  What the script does is produce a file containing commands that you might want to run.

Here's the header of the script.  Have a read through it, and then let me know if you want a copy of the script to test with.

REM ******************************************************************************************************
REM *    File:    expire-oldest-images.bat
REM *    Desc:    Determine oldest images on disk and/or on tape, and...
REM *        ...optionally choose to retain any copies on tape...
REM *        ...and thus potentially select only those copies of images residing upon disk...
REM *        ...and write a ".zap" file which could be used to expire only the selected copies.
REM *
REM *    Vers    Date        Who    Description
REM *    ----    ----        ---    -----------
REM *    v0.01    23-JAN-2016    sdo    Initial version.
REM *    v0.02    24-JAN-2016    sdo    Improved copy number loop.  Added more comments.
REM ******************************************************************************************************
REM *    !!! WARNING and DISCLAIMER !!!
REM *    !!! WARNING and DISCLAIMER !!!
REM *    !!! WARNING and DISCLAIMER !!!
REM *
REM *    Before using this script, be aware of the following points:
REM *    - Use of this script is entirely at the end user's own risk.
REM *    - Carefully read the entire header section of this script.  Read all of the notes and make sure
REM *      you have an understanding of the purpose of this script before using it.
REM *    - Neither the author of this script, nor Veritas, will accept any responsbility for issues or
REM *      problems or data loss caused by use, or mis-use, of this script, in either its original form or
REM *      a modified form.
REM *    - This script is not endorsed by Veritas.
REM *    - This script is not supported by Veritas.
REM *    - This script has not been tested by Veritas.
REM *    - This script may not be suitable for use in any given NetBackup environment.
REM *    - This script is furnished on an example basis only.
REM *    - Whilst every effort has been made (by the author of this script) to produce something that is
REM *      both useful and viable, be aware that bugs and errors may remain which may cause data loss.
REM ******************************************************************************************************
REM *    Sharing
REM *    -------
REM *    This script is free to share and modify, as long as this entire header section is kept with
REM *    the script, and maintained and updated appropriately as any changes are made.
REM ******************************************************************************************************
REM *    Notes
REM *    -----
REM *    1) This script was developed and tested using:
REM *        Windows 2008 R2, with NetBackup Server v7.7.1
REM *        Windows 2012 R2, with NetBackup Server v7.7.1
REM *       ...but should work with any Windows version since 2003 and any v7.* of NetBackup Server.
REM *    2) This script is a form of "What if?" analysis script.  This script should be safe to run on any
REM *       system, for several reasons:
REM *        i)    this script will not actually exire any backup images.  What this script does...
REM *            is to identify specific copies of backup images which could be expired, and write
REM *            the commands to achieve this to a separate file.
REM *        ii)    The commands will be REM'd out, so even an accidental run of the file created by
REM *            this script will not expire anything either.
REM *        iii)    The net effect is... the only way that specific copy numbers of backup images
REM *            could actually be expired is that the administrator has to make a concious effort
REM *            to edit the the created file and de-REM it, and rename it, so that it can be run.
REM *    3) This script tries to detect the date format, and also check that the date enterred by the
REM *       user is a valid date, and also checks that the entered date is less than today's date.
REM *    4) This script does not check for, nor handle, SLP incomplete backup images.
REM *    5) The v7.6.0.3 commands manual shows an example date format "mm/dd/yy" but says actual usage
REM *      is based upon NetBackup's locale setting.  Luckily, we can make several different dummy
REM *       calls to bpexpdate, because bpexpdate will return a status 227 if a backup image does
REM *       does not exist, but will return status 20 if one or more of the input parameters is
REM *       incorrect.  We know that our parameters are correct, except for the date, so... any
REM *       status 20 must indicate a bad date format.  Thus, multiple dummy calls to bpexpdate can
REM *       be used to validate its own parameter formats, and depending upon which call generates
REM *       a status 227, we can thus determine the correct date format for bpexpdate.
REM *    6) This script could result in making many calls to bpimagelist and bpverify and so could take
REM *       some time to run on medium to large NetBackup systems.
REM *    7) This script was written because it can be even more time consuming to manually check many
REM *       images, and check that any given copy number applies to the desired storage unit type.
REM *    8) This script will probably fail if you have 'spaces' or any strange characters in any of your
REM *       client names, or image names, or storage unit details, or disk pool details, or disk volume
REM *       details, or tape media IDs.  Basically, if your NetBackup environment is in a bit of a mess
REM *       then you probably wouldn't want to bother attempting to run this script.
REM ******************************************************************************************************
REM *    List Of Files Created By This Script
REM *    ------------------------------------
REM *    <script>.csv    filtered images in:  YYYY-MM-DD,image-name,copy-number,stu-type,stu-ident,stu-server
REM *    <script>.det    the bpimagelist of the last image inspected
REM *    <script>.eva    the list of images to evaluate for copies to potentially expire
REM *    <script>.ima    the bpimagelist -idonly of all SLP complete images
REM *    <script>.lis    list of filtered images before selected date in:  YYYY-MM-DD image-name
REM *    <script>.log    the log of the most recent run of this script
REM *    <script>.med    list of unique tape media id discovered
REM *    <script>.srt    the sorted list of filtered images in:  YYYY-MM-DD image-name
REM *    <script>.stp    list of all NetBackup disk pools
REM *    <script>.stu    list of all NetBackup storage units
REM *    <script>.stv    list of last NetBackup disk volume inspected
REM *    <script>.ver    output of bpverify -pb for the last image copy inspected
REM *    <script>.xxx    the accumulated log of all completed runs of this script
REM *    <script>.zap    the purpose of this script, is to write this file, which is a date filtered,
REM *            and storage unit type filtered, list of commands which could be used to
REM *            expire the specific copies of images on selected storage unit types.
REM *            N.B: the commands written to this file are REM'd out.
REM ******************************************************************************************************
REM *    Advice About Using This Script
REM *    ------------------------------
REM *    If you do want to use this script, then my advice is to run this script several times, and choose
REM *    different options each time.  Whilst it may take some time to run on medium to large environments,
REM *    it is useful to run it several times in order to gain an understanding of how it selects copies of
REM *    images to potentially expire.  So, check the log output, and check to make sure that it really is
REM *    only selecting the images that you think that you need/want to expire.
REM ******************************************************************************************************
REM *    How To Test This Script At Your Site
REM *    ------------------------------------
REM *    1) Identify a media server with a little bit of space.
REM *    2) Create a basic disk storage unit on that media server.
REM *    3) Use the management console GUI to identify one of the oldest images that you have.
REM *    4) Use the management console GUI to duplicate that image to your new basic disk storage unit.
REM *    5) Run this script in an attempt to select only that image for expiry.
REM *    Remember, this script won't actually expire anything, all it does is write a list of REM'd out
REM *    commands to the ".zap" file.
REM ******************************************************************************************************
REM *    Ideas For Future Development
REM *    ----------------------------
REM *    1) Instead of selecting images only by storage unit type, also select only copies of images
REM *       residing within specific storage units, or specific volumes pools, or specific volume groups.
REM ******************************************************************************************************

sdo
Moderator
Moderator
Partner    VIP    Certified

...and here's an example of running the script...

...and at the end you will see the contents of the ".zap" file, which shows that the script only selected two copies to expire, for images which do also already have a copy on tape...

24/01/2016  9:37:49  
24/01/2016  9:37:49  
24/01/2016  9:37:49  ********************************************************
24/01/2016  9:37:49  ********************************************************
24/01/2016  9:37:49  Script session details...
24/01/2016  9:37:49  ...script started:  24/01/2016  9:37:49
24/01/2016  9:37:49  ...script name:     expire-oldest-images
24/01/2016  9:37:49  ...script path:     E:\sdo\Scripts\expire-oldest-images\
24/01/2016  9:37:49  ...script log:      E:\sdo\Scripts\expire-oldest-images\expire-oldest-images.log
24/01/2016  9:37:49  ...computer name:   HSERVER
24/01/2016  9:37:49  ...username:        Administrator
24/01/2016  9:37:49  ...client name:     sdo-mbpro.loca
24/01/2016  9:37:49  ...session name:    RDP-Tcp#0
24/01/2016  9:37:49  ...done...
24/01/2016  9:37:49  
24/01/2016  9:37:49  Checking for presence of NetBackup...
24/01/2016  9:37:49  ...found NetBackup ok...
24/01/2016  9:37:49  ...client name:   hserver
24/01/2016  9:37:49  ...type:          NetBackup Master Server
24/01/2016  9:37:49  ...path:          C:\Program Files\Veritas\
24/01/2016  9:37:49  ...done...
24/01/2016  9:37:49  
24/01/2016  9:37:49  Determining storage units and types...
24/01/2016  9:37:49  ...found:  hserver-hcart2  Tape  
24/01/2016  9:37:49  ...found:  hserver-catalog  Disk  Basic
24/01/2016  9:37:49  ...found:  hserver-dsu  Disk  Basic
24/01/2016  9:37:49  ...found:  hserver-msdp  Disk  DiskPool
24/01/2016  9:37:49  ...done...
24/01/2016  9:37:49  
24/01/2016  9:37:49  Listing the `1` discovered disk pools...
24/01/2016  9:37:49     Storage Unit                Disk Pool Name              Disk Pool Type    Disk Pool Media ID
24/01/2016  9:37:49     ------------                --------------              --------------    ------------------
24/01/2016  9:37:49     hserver-msdp                hserver-msdp                PureDisk          aaaal
24/01/2016  9:37:49  ...done...
24/01/2016  9:37:49  
24/01/2016  9:37:49  Determining date format for bpexpdate...
24/01/2016  9:37:50  ...bpexpdate format is `dd/mm/yyyy`...
24/01/2016  9:37:50  ...bpexpdate date format is `UK`...
24/01/2016  9:37:50  ...year has `4` digits...
24/01/2016  9:37:50  ...done...
24/01/2016  9:37:50  
24/01/2016  9:37:50  N.B:  You now have a choice to make:
24/01/2016  9:37:50  ...enter YES, to     check for duplicates, and only expire images on disk which do     have a copy on tape...
24/01/2016  9:37:50  ...enter NO,  to NOT check for duplicates, and      expire images on disk which do NOT have a copy on tape...
24/01/2016  9:37:50  
24/01/2016  9:37:50  _Check for duplicates on tape?  (YES,NO,exit) [exit]  : 
24/01/2016  9:37:57  ...user entered `YES`...
24/01/2016  9:37:57  ...done...
24/01/2016  9:37:57  
24/01/2016  9:37:57  _Enter cutoff date in format `YYYY-MM-DD` [exit]  : 
24/01/2016  9:38:08  ...user entered `2015-09-01`...
24/01/2016  9:38:08  ...checking whether cuttoff date is a valid date...
24/01/2016  9:38:08  ...cutoff date of `2015-09-01` appears to be valid...
24/01/2016  9:38:08  ...comparing to today's date...
24/01/2016  9:38:08  ...cutoff date appears to be a previous date...
24/01/2016  9:38:08  ...done...
24/01/2016  9:38:08  
24/01/2016  9:38:08  
24/01/2016  9:38:08  Now choose which image copies to `Expire`...
24/01/2016  9:38:08     1.   Expire all copies, no matter whether they are on disk or tape
24/01/2016  9:38:08     2.   Expire only copies that reside upon any form of disk, i.e. BasicDisk, AdvancedDisk, PureDisk, DataDomain
24/01/2016  9:38:08     3.   Expire only copies that reside upon any storage unit of BasicDisk
24/01/2016  9:38:08     4.   Expire only copies that reside upon any storage unit of AdvancedDisk
24/01/2016  9:38:08     5.   Expire only copies that reside upon any storage unit of PureDisk
24/01/2016  9:38:08     6.   Expire only copies that reside upon any storage unit of DataDomain
24/01/2016  9:38:08     7.   Expire only copies that reside on any tape
24/01/2016  9:38:08  
24/01/2016  9:38:08  _Which option (1-7,EXIT) [exit]  : 
24/01/2016  9:38:12  ...user entered `3`...
24/01/2016  9:38:12  ...script will select image copies on storage of `BasicDisk`...
24/01/2016  9:38:12  ...done...
24/01/2016  9:38:12  
24/01/2016  9:38:12  Listing backup images...
24/01/2016  9:38:12  ...counting images...
24/01/2016  9:38:12  ...found `60` backup images...
24/01/2016  9:38:12  ...done...
24/01/2016  9:38:12  
24/01/2016  9:38:12  Filtering image list into YYYY-MM-DD order...
24/01/2016  9:38:12  ...image:  2015-06-13  sdo-mbpro_1434213287
24/01/2016  9:38:12  ...image:  2015-06-12  zzz-mbpro_1434105353
24/01/2016  9:38:12  ...image:  2015-06-09  hserver_1433834317
24/01/2016  9:38:12  ...image:  2015-06-08  hserver_1433767241
24/01/2016  9:38:12  ...image:  2014-10-26  hserver_1414334444
24/01/2016  9:38:12  ...image:  2014-08-06  hserver_1407307479
24/01/2016  9:38:12  ...image:  2014-08-05  hserver_1407235982
24/01/2016  9:38:12  ...image:  2015-07-09  macosx10-10-4-test_1436462856
24/01/2016  9:38:12  ...image:  2015-07-09  macosx10-10-4-test_1436451608
24/01/2016  9:38:12  ...image:  2015-06-14  win7pro_1434316888
24/01/2016  9:38:12  ...image:  2015-06-14  win7pro_1434315549
24/01/2016  9:38:12  ...sorting...
24/01/2016  9:38:12  ...count of images in source list:  60
24/01/2016  9:38:12  ...count of images before cutoff:   11
24/01/2016  9:38:12  ...done...
24/01/2016  9:38:12  
24/01/2016  9:38:12  Reporting number of copies of each image...
24/01/2016  9:38:13  ...image:  2014-08-05  hserver_1407235982                  copies: 1
24/01/2016  9:38:13  ...image:  2014-08-06  hserver_1407307479                  copies: 1
24/01/2016  9:38:14  ...image:  2014-10-26  hserver_1414334444                  copies: 1
24/01/2016  9:38:14  ...image:  2015-06-08  hserver_1433767241                  copies: 1
24/01/2016  9:38:14  ...image:  2015-06-09  hserver_1433834317                  copies: 1
24/01/2016  9:38:15  ...image:  2015-06-12  zzz-mbpro_1434105353                copies: 1
24/01/2016  9:38:15  ...image:  2015-06-13  sdo-mbpro_1434213287                copies: 1
24/01/2016  9:38:16  ...image:  2015-06-14  win7pro_1434315549                  copies: 2
24/01/2016  9:38:16  ...image:  2015-06-14  win7pro_1434316888                  copies: 2
24/01/2016  9:38:16  ...image:  2015-07-09  macosx10-10-4-test_1436451608       copies: 2
24/01/2016  9:38:17  ...image:  2015-07-09  macosx10-10-4-test_1436462856       copies: 2
24/01/2016  9:38:17  ...images checked:      11
24/01/2016  9:38:17  ...images to evaluate:  4
24/01/2016  9:38:17  ...done...
24/01/2016  9:38:17  
24/01/2016  9:38:17  Selecting image copies on storage of `BasicDisk`...
24/01/2016  9:38:17  
24/01/2016  9:38:17  ...checking image `1` of `4`, image:  win7pro_1434315549
24/01/2016  9:38:17  ......number of copies:  2
24/01/2016  9:38:17  ......copy number list:  1 2
24/01/2016  9:38:18  ......found image copy:  1       on:  BasicDisk
24/01/2016  9:38:18  ......found image copy:  2       on:  Tape
24/01/2016  9:38:18  
24/01/2016  9:38:18  ...checking image `2` of `4`, image:  win7pro_1434316888
24/01/2016  9:38:19  ......number of copies:  2
24/01/2016  9:38:19  ......copy number list:  1 2
24/01/2016  9:38:19  ......found image copy:  1       on:  BasicDisk
24/01/2016  9:38:20  ......found image copy:  2       on:  Tape
24/01/2016  9:38:20  
24/01/2016  9:38:20  ...checking image `3` of `4`, image:  macosx10-10-4-test_1436451608
24/01/2016  9:38:20  ......number of copies:  2
24/01/2016  9:38:20  ......copy number list:  1 2
24/01/2016  9:38:20  ......found image copy:  1       on:  PureDisk
24/01/2016  9:38:21  ......found image copy:  2       on:  Tape
24/01/2016  9:38:21  
24/01/2016  9:38:21  ...checking image `4` of `4`, image:  macosx10-10-4-test_1436462856
24/01/2016  9:38:21  ......number of copies:  2
24/01/2016  9:38:21  ......copy number list:  1 2
24/01/2016  9:38:22  ......found image copy:  1       on:  PureDisk
24/01/2016  9:38:22  ......found image copy:  2       on:  Tape
24/01/2016  9:38:22  
24/01/2016  9:38:22  ...finished evaluating images...
24/01/2016  9:38:22  ...evaluated images:  4
24/01/2016  9:38:22  ...copies to zap:     2
24/01/2016  9:38:22  ...done...
24/01/2016  9:38:22  
24/01/2016  9:38:22  Script exiting...
24/01/2016  9:38:22  ...script started:  24/01/2016  9:37:49
24/01/2016  9:38:22  ...script ended:    24/01/2016  9:38:22

E:\sdo\Scripts\expire-oldest-images> type expire-oldest-images.zap
REM bpexpdate -backupid win7pro_1434315549 -cn 1 -d 0 -force
REM bpexpdate -backupid win7pro_1434316888 -cn 1 -d 0 -force

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
I want.... please......

sdo
Moderator
Moderator
Partner    VIP    Certified

Now I'm nervous... :)     I'll PM you with it.   Would love to have it tested by a proper NBU guru!   Please?

Vamika
Level 3

Hi, sorry for late reply. We kinds resolve the space crunch by deleting few images which were about to expire/not needed.

But I really want to try out the script.

sdo
Moderator
Moderator
Partner    VIP    Certified

Hi Vamika - ok - I'll see if I can post it as a "download" here in the forum, and I'll send you a link if/when it gets published.