06-28-2012 10:08 AM
The reason is that we are just implimenting GRT backups of Sharepoint, which requires disk storage. Until we get some new hardware, I have a very small amount of disk storage (enough for one of said backups). So, ideally, I'd like to run the normal backup on a daily schedule and then have it automatically duplicate to tape and delete the copy on disk, so that the backup can run the next day.
So, is there a way that I can automate a process to duplicate image that's on disk and delete the original once the duplication is complete? I was looking at the bpduplicate command (which we already use in batch files called by scheduled tasks to duplicate other images) and the duplication part is easy, but I can't figure out how to get rid of the old image, short of doing it manually each day.
06-28-2012 11:24 AM
Yes this can be done and the best way to do this is using SLP or use the duplicate option in VAULT management. In both these options there is a option where you can delete the original copy once it is duplicated..
06-28-2012 11:48 AM
SLP isn't availabe in BasicDisk is it? And NetbackupVault requires additional licensing?
06-28-2012 12:17 PM
You are right, the only option is to do them manually either using bpexpdate or the other option is in the GUI go to the Catalog options and search with primary copy and copy the contents in Excel, then select copy 2 and copy the content in Excel. Do the Vlookup in excel to find out the images that are duplicated and those images can be expired either using GUI option in Catalog or via bpexpdate.
06-28-2012 12:28 PM
I agree - the only 'automated' methods are SLP and NBU Vault.
If you have DSSU configured, disk will also be cleaned out when High Water mark is reached, provided that backups have been successfully duplicated.
06-28-2012 01:16 PM
what about a script using bpduplicate, then maybe bpexpdate?
Regardless, maybe someone can help me out with this bpduplicate script, at least I can automate that. I'm trying to make 2 copies of the image. Both new copies will reside on the same storage unit, but in different Volume Pools. I've been playing with it, and this is the closest I've gotten to working, where it starts writing one of the copies (to "Full", but I'm not sure if that's coincidence), but the other immediately generates a status 167.
bpduplicate -number_copies 2 -dstunit balt-srvr-nbu-lto,balt-srvr-nbu-lto -dp FULL,offsite_full -policy SharePoint_GRT -hoursago 11 -L c:\Duplicate_Scripts\Dup_SP_GRT_log.txt
Here's the output of the log file.
15:56:22 Duplicate started 6/28/2012 3:56:21 PM
15:56:22 Activity monitor job id = 308698
15:56:22 Copy 1 activity monitor job id = 308699
15:56:22 Copy 2 activity monitor job id = 308700
15:56:26 INF - Destination storage unit balt-srvr-nbu-lto on host balt-srvr-nbu
15:56:26 INF - Destination storage unit *NULL* on host balt-srvr-nbu
15:56:27 INF - Duplicating policy SharePoint_GRT schedule Weekly_Full backup id wtp-sp2010-fe1_1340902715 copy 1 created on 06/28/2012 12:58:35 on source path E:\SharepointBackup
15:56:28 INF - no storage units available for use (213), cannot continue with copy 3
15:56:32 INF - Waiting for mount of media id 000113 on server balt-srvr-nbu for writing.
15:56:34 INF - Beginning duplicate on server wtp-nbumedia2.wtplaw.com of client wtp-sp2010-fe1, reading file E:\SharepointBackup\wtp-sp2010-fe1_1340902715_C1_F1.
06-28-2012 10:04 PM
If you have only BasicDisk, I second Marianne's suggestion to use DSSU - it's automated in a way when HWM is reached. You don't even have to script it. It does make sense because you can still have the disk copy after the duplication, and only get cleaned up when disk is going to fill up. Just need to watch out for duplication error from time to time.
06-28-2012 10:36 PM
> bperror -S 167 -r
cannot find requested volume pool in EMM database
A backup to a nonrobotic drive was tried and the tape manager (bptm) cannot find or add the specified volume pool.
Verify the Media and Device Management volume configuration. Check the NetBackup Problems report for
more information about the error. For detailed troubleshooting information, create a bptm debug log
directory and retry the operation. Check the resulting debug log.
If FULL volume pool is fine, check exact spelling and case of offsite_full.
06-29-2012 12:06 PM
Interesting. I wasn't familiar with it, nor could I find much info. But, I think I got it set up now. Couple of questions/oncerns, which I suppose i can answer through testing, but in case you know the answer:
Thanks in advance. As always, you have some really good info and suggestions.
06-30-2012 01:06 AM
1) You mean making default Low Water Mark (LWM) value lower? Of course you can, the only rule is "HWM > LWM".
2) No, when duplication is done, it won't cleanup/expire/delete your image. It will start to do so when your HWM is reached. For example HWM is 90%, LWM is 70%, you keep doing backup (to DSU) and duplication (to tape) daily, nothing happens, one day the DSU is 90% filled up, the image cleanup job now kicks in to select and cleanup the oldest images. How much data would it select to cleanup? That will be determined by LWM, so cleanup will select up to a point that DSU is back to 70% filled. So the automation is not based on duplication completion, but the disk usage/consumption.
3) Yes, it's a staging kind of setup and yes it's applicable to all type of data.
07-03-2012 12:21 PM
07-06-2012 07:32 AM
I'm having some trouble getting this to behave the way I think it should or would like it to. Any experienced or knowledgable input is appreciated.
This is only a temporary setup, until we get some new Disk Storage (but that could be a little while), but I needed to get it running so we can do GRT backups of Sharepoint until that time. So, no need to recommend more space, etc; as this is all I have to work with for the time being.
I'd like to get it so that Netbackup will free up the space and continue to back up w/o any intervention, but I'm not sure if that's possible.
07-07-2012 01:35 AM
Not sure whether you have gone through all we had posted in regards to your questions earlier...
Anyway, you've given an example (which is good), supposed we can talk about it. However, I should say there is not much we can play around because the storage is just not big enough to hold even 2 full copies of your full backup. 2 full copies would take up 360GB and is already filling up your disk!
- Let's assume you have 1TB of available disk in your basicDisk, and you're only doing full backup for this case, forget about incremental backup for now.
- Day1 : Backup 180GB, assume it's duplicated to tape by end of day. Disk storage usage is now 18%
- Day2 : Backup another 180GB, duplicate to tape again. Usage = 36%, nothing would happen.
- Day3 : Backup another 180GB, duplicate to tape again. Usage = 54%, now... this is where something would happen, since your HWM is 50%, and now disk usage is more than HWM, image cleanup would start.
and how would cleanup knows when to stop, that will be up to the LWM. In your case:
- Cleanup deletes day1 image, and disk usage is reduced to 36% , but still > LWM. That means more cleanup is needed.
- Cleanup deletes day2 image, and disk usage = 18%, still > LWM. More cleanup needed.
- Cleanup deletes day3 image, only then disk usage is < LWM, and cleanup would stop. Now you have an empty disk for have backup run again on Day4.
The LWM value of 10% in this case is not a very good setting as it will cleanup all images - leave you no disk copy, so a 20% would be better IMO. So the value should be set based on your total image size (per day), and its retention (and other factors, but let's keep it simple here).
If you then notice, 341GB is really hard for you set the HWM & LWM here because of the quick usage and no room for 2nd backup. Consider adding more diskspace into it before changing to use this.
07-09-2012 06:22 AM
Yes, I've been following up on each post, which is how I arrived at my last set of questions, which all remain unanswered.
So, my questions are the same.
07-13-2012 05:01 AM
Oh, sorry about that.. guess I got the wrong perception on LWM. I am going to test this in my environment... and hopefully get some insight on it.
07-15-2012 09:40 PM
I tested my basicdisk, much to my surprise, the image cleanup just works like how I understand it - not sure if it's a coincidence. But the image cleanup just got trigger when HWM is exceeded, and it stops once LWM is met! Here is a snippet of my image cleanup job detail:
07/16/2012 11:03:28 - Info bpdm (pid=30546) initial volume /bscdisk: Kbytes total capacity: 50330044, used space: 43095820, free space: 7234224
07/16/2012 11:03:39 - Info bpdm (pid=30546) Removed 3318488 Kbytes
07/16/2012 11:03:39 - Info bpdm (pid=30546) ending volume /bscdisk: Kbytes total capacity: 50330044, used space: 39777332, free space: 10552712
My HWM = 85%, LWM = 82%
07-16-2012 09:42 AM
I do agree with Watson and tested the configuration.
My environment:
NBU 6.5
Basic Disk - used D drive for staging
So, my questions are the same.
As per the policy schedule the data will be written to the basic disk. Once the backup completed the DSSU policy will trigger as per the schedule you have in the storage unit for staging. If for some reason the DSSU policy is not getting triggered the backups will fill the basic disk and eventually fail with no space avilable. The data on the basic disk will not be deleted or expired unless it get duplicated through DSSU policy. The data will remain on the disk as per the policy retention that you have mentioned. If your DSSU policy runs properly it will duplicate the data from the basic disk to the tape but the data will be only deleted from disk depending upon the HWM setting.
2)by "deletion of images" (above), does that mean it will "Expire" the image and delete the associated files, or will it only delete the files, or.....?
Once the data is duplicated to tape through DSSU policy and HWM is triggered it will delete the images from basic disk. Now you only have primary copy on tape and disk data doesn't exist on catalog or anywhere, Its gone completely.
3)If HWM is the only trigger for deletion, why isn't it deleting in my example?
Please test it I have tested it for past 3 days and works like charm. At one point I have set the LWM to 30% and it immediately deleted the last image on the basic disk. If you still think it doesn't work for you please post the output of your policy details, storage unit
4)If Retention time and HWM play a role in the process, how do I create a different retention between the tape and disk copy (the only setting appears to be the retention configured in the backup policy)?
The retention will only be available in the policy. Lets see you have a 100 GB disk and you are doing a retention of 1 week. The backup will run on basic disk and then as per DSSU policy it will be duplicated to tape. Now you have 2 copies, primary on disk and secondary on tape. The primary copy on disk will not be deleted till HWM gets triggered, till that time you can restore from disk instantly. Though the image on disk will be expired after 1 week.
Hope this makes things a bit clear.