How to break up a large backup image

[spitman]

Just for background info:

I'm running NBU 7.7.1 on a RHEL 6 environment, with one master and three media servers. I have a Quantum DXi 6702 disk unit with 80TB of space and a Scalar i500 tape library with 200 licensed slots.

I have a Black Duck client that is running on RHEL 6 also. It has an /app directory on it that is over 3TB and growing.

I'm trying to figure out the best way to break up this backup into "manageable chunks." With other clients I have been able to work with application owners to create a one-off policy that would allow me to break up the backups into 1TB or less chunks--to speed backup (and restore) time.

Because of this application, and the weirdness of its subdirectories, I have been nervous about trying to do that, not knowing how much the file structure will change over time. To be fair, I haven't talked to the application owner yet; but since this is a new thing for us I don't know how much they'll be able to answer the question anyways.

I attempted a policy with the "allow multiple data streams" and just let it go; and it launched a slew of jobs; but /app still always comes up as its own job.

I'm just thinking that I'm not the only person who's ran into a situation like this, and I'm wondering what people's creative ideas are to make this situation better. As it is now, the backup image is almost 4TB, and right now the SLP is trying to copy that monster to tape--and has been running almost two days straight.

Thanks in advance.

[Genericus]

I do this:

Set up OS backups as normal. Policy - LINUX.OS.PROD

Add an exclusion file in /usr/openv/netbackup like exclude_list.LINUX.OS.PROD

Add line in that file /app

Set up a new policy LINUX.OS.APP.PROD

backs up only that client, and selection is /app/*

Use multiple streams.

This is only realy viable for main directories, or if you know you have one main split point.

I do this when I have systems with filesystems over 1TB, so I can run parallel streams to tape.

[watsons]

Enable multiple data stream does not break down the "backup selection" for you. It merely means it will kick off one stream for each backup selection (path) you put in there.

So if you have /app in your backup selection, it will still be backup as a single stream.

If you can break it down based on your research of the application, probably you are able to setup something like this in backup selecton:

/app/log
/app/data/prod1
/app/data/prod2
/app/bin
/app/etc
...

[Marianne]

The * in /app/* with multiple data streams will create a new stream for each subfolder.

[J_MCCOLL]

Another thing to look at is your SLP configuration, as the large data volume appears to be stored in /app/ therefore here are some best practices:

• Mark all disk storage units that are used with SLPs as On demand only.
• Large duplication jobs are more efficient. (Modify the MIN_KB_SIZE_PER_DUPLICATION_JOB)
• Limit the number of SLPs you create.
• Avoid increasing backlog (the number of images waiting to be duplicated).
• Plan for duplication time. Duplication of a backup usually takes longer than writing the initial backup itself.

Not sure if this is the latest SLP Best Practices or if anyone can provide a link to the latest version: http://www.veritas.com/docs/000068194

Cheers,

J

[spitman]

Hi Genericus,

Thanks for the info.

The drill-down to the main dir that takes up over 3T has over 33,000 separate files... and I'd really like to not have that many separate threads...

But, in other situations, hopefully this will help someone who reads this thread.

Susan

[spitman]

Watsons,

This is what I was, in fact, attempting to do;

It was just getting so granular... and I was wondering if there was a way around it...

See my other replies (that I'm just now posting).

Thanks for your info.

Susan

[spitman]

Marianne,

It would end up to be a little further down the tree than that; but yes this solution would work great... 

I'm going to post another final note on this thread.

Thanks again!

Susan

[spitman]

J_M,

- I have a limited number of SLPs.

- The DSU is on demand.

- Our duplication times are windowed to happen during the week (not during weekend backups).

Thanks,

Susan

[spitman]

Thanks to all for your input.

So to finish off the story...

I contacted Black Duck, and they send me their sysadmin guide (see attached).

I also contacted the application owner of this system...

...they had never implemented the suggested database backups. 

!!!

...and the directory containing the slew of files in question should not be being backed up (live database).

He informed me that he had hired a "black duck administrator" who is supposed to start either this week or next week... hope his first order of business is to set up backups correctly...

[Genericus]

There is a reason they say - "Plan, Build, Run"

So many of us get handed something already running, and wonder "Did anyone do any planning?"

I have an application I inherited that places 20 MILLION files in one directory. When I started working on it on HPUX it would actually core dump when you tried to run a "ls" command.

We ended up excluding it from backups entirely as a "compromise" because the application would not revise their structures.  Eventually it was moved to a linux VM and now we FLASH backup the entire disk.

You can do some manual parsing like /apps/a*, /apps/b* etc, but why? Just because you CAN do something does not mean you SHOULD do it. Making your policiy needlessly complicated will mean you will be micromanaging it forever.

You might also check with your Black Duck contacts to make sure there is some kind of cleanup mechanism, I am starting to work with my application groups to have them define their retention on disk to work with my retention within NetBackup.

[Will_Restore]

Protex Admin Guide says:

The standard process for backing up a PostgreSQL database is to dump it to a file. This serves as your backup. You may want to copy your backup files to removable media such as tape or DVD. Should you lose data, or if a database gets corrupted, you can restore the database using this file.

Now you just have to wait for the new Black Duck admin to set that up.