Vaulting and Storage Lifecycle Policies

[elanmbx]

Is there *any* way to use these together.  Here is my scenario:

I need to use SLPs to replicate MSDP data from our primary DC to an offsite DC.  That is working well.  However, I want to batch up writing older MSDP images from disk to tape.  Vault has always worked for this functionality *very* nicely.

So I configured a vault to start rolling images off the MSDP to tape - only to receive the "images failed: Already duplicated" message. 

Is there any way to use vault in this manner?

Thanks,

-Mike-

[mph999]

Should work.

In host properties > master > global attributes, what is teh 'max backup copies' set to.  If this is only = to the number of copies you have after te SLP is run, vault won't be 'allowed' to make any more.

martin

[elanmbx]

"Max Backup Copies = 4" - and the images only appear to have a single copy (I assume that's because the SLP is doing a Replication to the DR site - a separate domain):

sudo bpimagelist -L -backupid dencnft1_1347364409

Client:            dencnft1
Backup ID:         dencnft1_1347364409
Policy:            Denv01d
Policy Type:       VMware (40)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       incr
Schedule Type:     INCR (1)
Retention Level:   1 year (8)
Backup Time:       Tue Sep 11 05:53:29 2012 (1347364409)
Elapsed Time:      754 second(s)
Expiration Time:   Wed Sep 11 05:53:29 2013 (1378900409)
Compressed:        no
Client Encrypted:  no
Kilobytes:         919322
Number of Files:   400
Number of Copies:  1
Number of Fragments:   2
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     yes
Files File Name:   Denv01d_1347364409_INCR.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               0
TIR Info:          0
TIR Expiration:    Wed Dec 31 17:00:00 1969 (0)
Keyword:           Non Production
Ext Security Info: no
File Restore Raw:  yes
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:  Mon Sep 10 02:41:56 2012 (1347266516)
BI Full Time:      Fri Sep 07 16:52:07 2012 (1347058327)
Request Pid:       0
Backup Status:     0
Stream Number:     1908
Backup Copy:       Standard (0)
Files File size:     22235
PFI type:     0
IMAGE_ATTRIBUTE:     0
Primary Copy:      1
Image Type:        0  (Regular)
Job ID:            4323472
Num Resumes:       0
Resume Expiration: Wed Dec 31 17:00:00 1969 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    (none specified)
Storage Lifecycle Policy Version:    0
STL_Completed:      0
Remote Expiration Time: Wed Dec 31 17:00:00 1969 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:      Tue Sep 11 05:52:29 2012 (1347364349)
IR Enabled:      no
Client Character Set:     0
Image On Hold:     0
Indexing Status:   0
Copy number:       1
 Fragment:         TIR (-1)
 Kilobytes:        58
 Remainder:        294
 Media Type:       Disk (0)
 Density:          qscsi (0)
 File Num:         0
 ID:               @aaaah
 Host:             densyma01p
 Block Size:       262144
 Offset:           0
 Media Date:       Wed Dec 31 17:00:00 1969 (0)
 Dev Written On:   -1
 Flags:            0x0
 Media Descriptor:        1;PureDisk;densyma01p;dp_disk_densyma01p;PureDiskVolume;0
 checkpoint:       0
 resume num:       0
 Key tag:          *NULL*
 STL tag:          *NULL*
 Copy on hold:     0
Copy number:       1
 Fragment:         1
 Kilobytes:        919264
 Remainder:        0
 Media Type:       Disk (0)
 Density:          qscsi (0)
 File Num:         0
 ID:               @aaaah
 Host:             densyma01p
 Block Size:       262144
 Offset:           0
 Media Date:       Wed Dec 31 17:00:00 1969 (0)
 Dev Written On:   -1
 Flags:            0x4
 Media Descriptor:        1;PureDisk;densyma01p;dp_disk_densyma01p;PureDiskVolume;0
 Expiration Time:  Wed Sep 11 05:53:29 2013 (1378900409)
 MPX:              0
 retention_lvl:    1 year (8)
 Try to Keep Time:  Wed Dec 31 17:00:00 1969 (0)
 Copy Creation Time:  Tue Sep 11 06:06:03 2012 (1347365163)
 Data Format:      Tar
 checkpoint:       0
 resume num:       0
 Copy on hold:     0

[mph999]

Hmm, just tested it - I get the same thing ...

23:16:50.578 [15401] <8> vltrun@DupValidationManager::LogRejectedImages^2: Rejected for Duplication - IMG=womble_1352415775 REASON=AlreadyDuplicated (2)

I'll have a play about see what I can work out

Martin

[mph999]

Changed the vault policy aroung, but still same error.

Doesn't look like it works after all - anything I have duplicated with SLP is ignored by vault (well, rejected).  I'm a bit surprised, but then, I'd never tried it before.  

I then ran a backup to the same STU that the SLP used, but using a regular policy and vault copied that image fine.

As a work around if you get a list of the backupid, put these as a list in a file and use this with the bpduplicate command to  duplcate them 'manualy'

Martin 

[elanmbx]

Yes - that's not a bad idea.  I was hoping that there was some more elegant way to accomplish this - vault does a pretty good job of "batching" the images so that the tape drives get used efficiently.  I'm not sure how well a manual "bpduplicate" is going to compare... but I think I'll try it out.

If anyone else has any ideas, please keep them coming!

Thanks.

[RLeon]

Edit: I missed the part that you wanted to do this in the source nbu domain, not the target domain. The following is now somewhat irrelevant.

Sounds like another case of SLP not wanting to let go of its toys.

To achieve a "batching" effect (more or less), you could try tweaking LIFECYCLE_PARAMETERS at the target nbu domain. Specifically these entries:

DUPLICATION_SESSION_INTERVAL_MINUTES = Minutes_value (default 5, min 1)
Change this to some big number, so it doesn't check as often. You don't need it to anyway according to your needs.

MIN_GB_SIZE_PER_DUPLICATION_JOB = GB_value (default 7)
Change this to something huge.

MAX_GB_SIZE_PER_DUPLICATION_JOB = GB_value (default 25)
Change this to something huge again; larger than the min gb one.

MAX_MINUTES_TIL_FORCE_SMALL_DUPLICATION_JOB = Minutes_value (default 30)
Change this to very long periods, so you give the min gb param enough time to fill up (possibly reaching max gb); effectively batching more images per duplication job.

[mph999]

It's easy to do ...

Here is a list of images I put into a file (call the bidfile)

womble_1351156009
womble_1351148274
womble_1351146905
womble_1351098562
womble_1351098070
womble_1351098024
womble_1351075851
womble_1351070382

I put these backupids into a file /tmp/imagelist

 I will duplicate to a STU called womble-4mm-robot-tl4-0
The bpduplicate command creates the copy (Bidfile = list of images to duplicate)

bpduplicate -Bidfile /tmp/filelist -dp NetBackup -dstunit womble-4mm-robot-tl4-0
 

... and it is as simple as that 

Martin

[Mark_Solutions]

Just a thought - and i cannot test it at the moment - plus you will have to think about any other consequences ... but if you set the vault profile to make 2 copies what happens? Does it skip one but do the other and so give you what you want?

As i say - total theory but maybe worth a try

[elanmbx]

I know it is pretty easy to use bpduplicate - it's the management of the duplications that vault is particularly good at.  If I just get a list of images for a day and duplicate the bid file it throws the duplications to a single tape drive.  I'd rather not get mired in the management of splitting up bid files into chunks and sending the individual broken-up files to bpduplicate.

I was hoping I was missing something easy and obvious...

[mph999 · Accepted solution]

OK, multiple tests later ...

Configured a basic disk STU and ran SLP that backs up to sp1 and duplicates to sp2 (sp1 and 2 are both advanced disk stu).  Once completed 

I then ran vault to copy this again to 'basic_disk' and it worked ...

IMAGE womble 0 0 9 womble_1352491909 womble_slp_sp1_sp2 0 *NULL* root Full 0 11 1352491909 9 1352578309 0 0 32 6 3 3 0 womble_slp_sp1_sp2_1352491909_FULL.f *NULL* *NULL* 0 1 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 126 2 0 1312 0 0 *NULL* slp_sp1_sp2 3 0 0 0 *NULL* *NULL* 0 13 0 0

HISTO 0 0 0 0 0 0 0 0 0 0

FRAG 1 1 32 0 0 0 0 @aaaad womble 262144 0 0 -1 0 1;AdvancedDisk;womble;ad_disk_sp1;/sp1;0 1352578309 0 65547 0 0 0 6 0 1352491918 3 1 *NULL* *NULL* 0 0

FRAG 2 1 32 0 0 0 0 @aaaae womble 262144 0 0 -1 0 1;AdvancedDisk;womble;ad_disk_sp2;/sp2;0 1352578309 0 65547 0 1 0 6 1352578309 1352492296 1 1 *NULL* *NULL* 0 0

FRAG 3 1 32 0 0 0 0 /netbackup/nbu7503/basic_disk/womble_1352491909_C3_F1_R2 womble 262144 0 0 -1 0 womble 1352578309 0 65547 0 2 0 1 0 1352492877 1 1 *NULL* *NULL* 0 0

... ok - so not sure why things aren't working for you, but at least we have shown that vault can duplicate an SLP image (as you can see above, 3 fragments, 1 is SLP backup, 2 is SLP duplication, 3 is vault duplication ).

I did get a simlar error with what I have now decided was an 'invalid' test - trying to get vault to duplicate another copy of the image on the same stu as either the SLP backup or the SLP duplication (sp1 or sp2)

Not sure why you are getting this error, as you are going from disk to tape - I'll dig round a bit more ...

Perhaps I'm missing something obvious - let me know if you see it   :0)

Martin

[elanmbx]

Yeah - it's really rather strange.  I wonder if it has something to do with the AIR I have configured... although I don't know why it would.

[mph999]

Ahh, one MAJOR thing I forgot to mention ...

I could only play about with ad disk pools / basic_disk etc ... as I have no AIR setup.

This is the closest I could get.

The only conclustion I can reach at the moment, is that this is a limitation in the NBU - but I can't find anyone else reporting the same thing.

I've asked Marianne to have a look, she may very possibly know something I don't ....

Martin

[elanmbx]

Sorry guys - was a misconfig in the vault duplication settings.  I got that sorted out and now I'm vaulting data successfully.

A related question, though - my 1st vault created 4 duplication jobs:

#1 - 14TB - media server 1

#2 - 4.5TB - media server 2

#3 - 2.4TB - media server 3

#4 - 2.0TB - media server 2

Each of these media servers has 3xLTO5 drives attached.  How do I get Vault to break up these jobs into somewhat smaller ones to stream to *all* my tape drives on each media server?  I'd get through things more quickly with more numerous, smaller jobs.

Thanks!

[elanmbx]

THanks to everyone for the assistance!