Dupe2Tape size issues, Excess amount of Jobs

[brehms]

I just upgraded to 7170 with SP1, and hf16.
I converted all my backups to Policy's and Selection lists. Of which I am not happy because my jobs went from 17 to 40+ (more on this later). Anyways, I do a combination of differential and full backup's during the week, and all fulls every friday to a virtual tape library (HP VLS6800). Then the friday jobs, are supposed to duplicate to a partitioned tape library over the weekend.

My Problem: My linux backups are daily full backups of 200 GB. When friday comes around for the linux duplicate to tape, the size backup'd is ~ 4 TB. Which I figured was a duplicate of the entire month of backups.
So I made some media sets. Daily and Weekly. Yes I know, I probably should of done this before. But since we're only keeping 5 weeks total of backups, regardless of being daily or weekly, I didnt think I'd need to. So now that I have weekly media sets, the duplicate has been reduced in size to ~800 GB. Which is better, and I figured that now it's just a combination of every friday's backup being duplicated at once. So I am thinking that I may have to create a different media set for each friday to make this work. Which creates more jobs.

Jobs:
Before using Policy's / Templates:
    (3) - 1 Job for Linux Fulls. 2 weekly duplicates (servers manually selected)
Initially After Policy / Templates:
    (3) - 1 Job for Linux Fulls. 2 weekly duplicates
After Implementing Media sets (2)
    (4) - 2 Job for Linux Fulls (daily and weekly sets). 2 weekly duplicates
After Implementing Media sets (each friday)
    (7) - 5 Job for Linux Fulls (daily, and each week). 2 weekly duplicates

As you can see, I went from 1 Linux job, to 5 jobs (excluding duplicates).

Now I have 9 different jobs that run, on a daily basis. Exchange, Linux, Cluster file server, Microsoft servers, SQL, GIS, + 3 remote managed media servers. Exchange, SQL, and Linux are all in the same boat with Fulls every day (5 jobs ea). The other servers use DIFF's, with weekly fulls (5 jobs ea) (excluding duplicates).

So all together, I am looking at:
30 Jobs for DIFF/Full servers
15 Jobs for daily FULL servers
18 Jobs for duplicate (twice for 2 different offsite locations)
63 Jobs TOTAL when using Policy and Templates with a media set for each friday.
vs.
12 Jobs for DIFF/Full Servers
3 Jobs for daily FULL servers
2 Jobs for Duplicates
17 Jobs when doing it the old way, Without using Policy and Templates.

So I guess I have two problems.
1. Minimizing the total number of jobs when using Policy / Templates
2. Duplicates only duplicating the previous Fridays job.

Thanks in advance for your help.

Scott

[David_Bond]

Why do you have 5 jobs for the daily and weekly jobs? With the policy you would only need 1 job for the daily full and 1 job for the weekly (with the media sets), and a duplicate job for each of the jobs you want to duplicate. And in the policy if you just selected all the days for the daily and just the friday for the weekly, you would add a template rule that would make the weekly override the daily.

How are you setting up the duplication of the jobs, which jobs are you duplicating? all, just the fridays? If you are just duplicating the friday job, and want it to run right after the backup, create a duplication job in the template, set it to run only according to the rules of this template. If you are duplicating all the backups, just do that for your daily too (if you want it to do it right after it has finished) if not, selecty run according to schedule and run according to the rules for this template, and select when.

But would need more detailed info on what and how you are currently setting up the jobs and how you want it to run to try and help more.

[brehms]

---

@David Bond wrote:

Why do you have 5 jobs for the daily and weekly jobs? With the policy you would only need 1 job for the daily full and 1 job for the weekly (with the media sets), and a duplicate job for each of the jobs you want to duplicate. And in the policy if you just selected all the days for the daily and just the friday for the weekly, you would add a template rule that would make the weekly override the daily.

---

I dont have 5 jobs. I have 2. I have exactly what you describe above. Daily and Weekly media sets. I stated 5 because my duplicates were including Every Friday (800 GB), instead of the friday the night before (200GB). So I was thinking that I might have to have a seperate media set for each friday. 4 fridays a month + 1 daily = 5.

---

@David Bond wrote:

How are you setting up the duplication of the jobs, which jobs are you duplicating? all, just the fridays? If you are just duplicating the friday job, and want it to run right after the backup, create a duplication job in the template, set it to run only according to the rules of this template. If you are duplicating all the backups, just do that for your daily too (if you want it to do it right after it has finished) if not, selecty run according to schedule and run according to the rules for this template, and select when.

---

Idealy, I need all of my jobs to duplicate twice, the Friday FULL backups. I have them scheduled to start duplicating after the last normal job completes. So I am doing both Rules and a schedule.

For simplicity, we can just focus on 1 job. I have my duplicate template pointing to the friday template in the 'Source' section, of which is in a Weekly Media Set. Now when this runs, My duplicate job is 800GB. Which obviously must of included the previous Fridays. since the normal Friday backup is only 200GB.

Hmm I think I may have figured it out. I think I screwed up my template rules. Making some changes, and I'll post back Monday.

Thanks David for the help.

Scott

[brehms]

Hmm...
When I choose within my duplicate template to backup the 'Linux-Fri' job, it creates a Template rule that says
'Duplicate all backup sets that were created by Linux-Fri using Duplicate - 1 as scheduled.'

I dont want 'all' the backup sets, just the backup that was just created. So I removed that rule and added the rule
'After Linux-Fri completes, start Duplicate - 1'

But when I try to click ok and save it; it errors out, because removing the rule, unchecks the associated source job template.

Thoughts?

Scott

[David_Bond]

Are you creating a different policy for the duplications, or creating the duplicate job within the policy that it will apply to?

If you are doing it in the policy for the job it applies to, do the following (not in wizard mode, well probably would be fine but i have never used it :) ), select new template..., Duplicate backup sets template,  in the templates source select your weekly backup set job template. In device and media select the media set you want the tape dupliaction to be asscoiated with and the device you want it to write to. Do the naming of the job and template etc and then in the schedule, select run only according to the rules of this template (this will make it run right after the B2D job).

That should now only make a copy of the latest weekly backup job. To do a second duplcation, just do the same again, it will run right after the first (should, i havent tried).

Hope that helps

David

[brehms]

I've thought about doing it that way. The problem I run into then though, is my maintenance window. I only have 16hrs to get the data off all of the servers. If I start running dupes during that window, my primary jobs would get delayed.
That's one of the reasons I had the duplicates scheduled, so as soon as the last job finished, the dupe's would start.
My total Dupe time runs 19hrs, and I have to dupe twice over the weekend.

[David_Bond]

The duplications shouldnt affect your B2D jobs (may slow it down a little depending on the array speed). The jobs should run in parrallel, you will have to set your B2D locations to allow concurrent accesses if being used by more than 1 backup at a time.

The way that I do my backups is, I have an hourly (for bin/transaction logs for the databases) daily incrementals, weekly and monthly fulls. All these are B2D jobs (except 1, which is the backup server which goes directly to tape). I have set up 2 duplication jobs per policy, one policy per server (as i have different backup locations for each server). The 2 duplication jobs copy the weekly and monthly B2D jobs of its policy to tape, right after the backup has completed.

I have 3 backup servers currently, each backing up around 10 servers, all concurrently, to disk, then on the fridays, the current full backup is copied to tape. For me the duplication job doesnt affect the backup to disk jobs, it just writes away to tape while all the other jobs carry on.

I have a backup to disk location for each of my servers, each of which is set to allow 4 concurrent accesses. If you have only 1 location and you have multiple jobs and only 1 runs at a time. Have you set it to allow more than 1 access at a time in its properties in backup exec ? If not, increase it to how ever many servers you are backing up at the same time plus 1 or 2 (so you can duplicate at the same time as backing up).