RMAN Netbackup to tape - How to get more than one file per job

[safedba]

We're running a full Exadata 12c Rac. Tape guys complain about the number of jobs saying there's only one file per job. We backup to the flash recovery area and from there to tape. Even changing the filesperset parameter when going to tape doesn't change the number of files per job. Is there any way to get more than one file going per job?

[Marianne]

Check Max Jobs Per Client in Global Properties. Ensure this is more than 1. Value of 4 will allow 4 simultaneous jobs per client.

Check MPX setting in Application schedule. Must be >1.

Define more CHANNELs in rman script.
One job per channel will be created.
Filesperset will simply group more dbs into one backup job.

[safedba]

I appreciate the reply. We're opening up 12 channels, 2 per node. Are any of your suggestions going to increase the number of files instantiated per job? That's what's bothering the tape guys. They say it's taking more time because of the stopping/starting/searching/positioning...and the writing. If each job had as many files as I could put into them, then it could pretty much fly. As it is it's like in the old days when people set their block size too small so reading it was staccatto with lots of starts and stops. Tape guys want a stream of writes, not a bunch of starting and stopping.

[Mouse]

RMAN has a parameter called FILESPERSET and it defines the number of files per backupset. What I would like to warn you about this setting - it really affects deduplication ratio if you're writing it to a deduplicated storage. Fixed block dedupe storages like NBU MSDP or Appliances will have the worst hit, variable block dedupe such as Data Domain will be impacted less but still dedupe rate will be considerably worse than with FILESPERSET=1.

Now you really need to choose your poison...

[Marianne]

Please tell us a bit more.
How many media servers? How many tape drives per media server?
How many simultaneous jobs generated by '12 channels, 2 per node'?
What throughput is seen to each of the drives (Kbytes/sec)?
What is the network speed between Oracle clients/nodes and media server(s)?

'stopping/starting/searching/positioning' is a sign of slow throughput to the drives.
So, yes, more simultaneous jobs per drive could help, but you need to do the maths to work out if there is sufficient bandwidth and media server resources.

[safedba]

I experimented with filesperset at 20 and it still populated 1 per job. Now these were archive logs so I don't know if this has anything to do with anything, but no, by itself setting the filesperset didn't affect anything in the desired vector. (I tried this prior to your reply)

[Marianne]

@safedba
Please try to answer all my questions?

[safedba]

I'm not at work right now. Perhaps I can answer the other questions later if these have anything to do with my goal of multiple files per job.I also have a problem insofar as I have not control over the tape backup guys. I know what I've posted. I have no idea how they've set up the policy on the other end, however I believe I can override their policy if I push parameters. In any case the goal is simple: get multiple files per job. There is no other goal here. Also, we are not backup up from the datafiles in oracle to tape. We are backing up to the flash recovery in ASM on RAC and from there we back up to tape. I'm not an expert in Netbackup obviously. When I check the so-called "best practices" of Oracle or Veritas these appear to presume that one has access to all the bells and whistles. This isn't realistic and it isn't my situation. The control of the tape backups is that we, the oracle client on the linux nodes PUSH the data and we use the recommended rman catalog on a separate machine/db which catalogs all the backup tapes. Netbackup only knows it has tapes. It has no knowlege of anything else.

[Mouse]

I don't think it's a relevant forum for your questions, as NBU just provides a pipe for Oracle through the standard interface, as long as appropriate number of channels can be created - NBU has done its job and the rest is the responsibility of Oracle. Now because the interface is standard, you can replace NBU with BUE, NetWorker or God forbid, CommVault, you will still have the same issue if its root cause in RMAN

To clarify again - NBU don't form backup sets and operate with Oracle logs and datafiles. 

[Marianne]

Hi @safedba

I agree with @Mouse

We are not going to solve your issue here.
You and the NBU admins need to work together as team.
It seems to me that there is 'fingerpointing' happening here that is not going to solve anything.

The filesperset value determines the amount of simultaneous Oracle file that will be grouped in a single stream (one defined channel).
The number of defined channels determine the amount of jobs/streams generated by the Oracle client.

What happens next is controlled by NBU settings, e.g. 
Max Jobs Per client
Number of tape drives
Number of media servers
Media server resources
MPX in Storage Units
MPX in Policy schedule(s)

Plus environment capabilities such as network.

Please show your NBU admins this discussion and ask them to join in.

[safedba]

I'm going to see if I can get the tape guys to clue me in on their policy settings. In any case this is the scripts:

function RUN_RMAN_TO_TAPE {
RSQL="RUN {
SET AUTOLOCATE ON;
ALLOCATE CHANNEL CH01 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup1 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm01-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH02 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup1 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm01-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH03 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup2 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm02-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH04 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup2 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm02-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH05 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup3 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm03-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH06 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup3 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm03-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH07 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup4 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm04-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH08 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup4 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm04-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH09 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup5 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm05-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH10 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup5 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm05-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH11 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup6 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm06-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
ALLOCATE CHANNEL CH12 DEVICE TYPE 'SBT_TAPE' CONNECT /@${ORACLE_BKUP}_bkup6 PARMS='ENV=(NB_ORA_CLIENT=pm01dbadm06-2,NB_ORA_SERV=xx.xx.5.6) SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
BACKUP FILESPERSET 20 RECOVERY AREA NOT BACKED UP 1 TIMES;
DELETE NOPROMPT OBSOLETE;
}"

It isn't pushing multiple files per job. Each job only has 1, so I don't think the suggestion about filesperset is true, or perhaps it doesn't apply to archive logs.

[safedba]

On the other hand it would be useful to note that "fingerpointing" is always and will always be present in every work situation. It is simply unavoidable. One musn't sit one's hands and pout until the pointing stops because it never does. One must simply persist in trying to work past the fingers. Of course it wasn't a problem when I was an IT director because what I said is what happened.

[Marianne]

---

@safedba wrote:

It isn't pushing multiple files per job. Each job only has 1, so I don't think the suggestion about filesperset is true, or perhaps it doesn't apply to archive logs.

---

If this is what you are experiencing, then you need to take it up with Oracle.

Extract from Oracle docs:

https://docs.oracle.com/cd/B28359_01/backup.111/b28270/rcmcncpt.htm#BRADV89486

... you can specify BACKUP ... FILESPERSET to specify the maximum number of files in each backup set.

Wishing you all the best.

[safedba]

No go on the filesperset. It does fine with that parameter when going to the flash recovery area (20 per set) but not when going to tape.

16:10:27 INF - Starting bpbrm
16:10:30 INF - Data socket = nbumedia02-2.IPC:/usr/openv/var/tmp/vnet-11652516637429860659000010331-x8hnD9;f4fb7fec2ee3d25a7a165930b693e6fe;11;3600
16:10:30 INF - Name socket = nbumedia02-2.IPC:/usr/openv/var/tmp/vnet-11654516637429969634000010331-1zYeX9;f02a974355d6c7e4a97b3dc75c684b6d;11;3600
16:10:30 INF - Job id = 446772
16:10:30 INF - Backup id = pm01dbadm01-2_1516637426
16:10:30 INF - Backup time = 1516637426
16:10:30 INF - Policy name = ORACLE_Exadata_PROD
16:10:30 INF - Snapshot = 0
16:10:30 INF - Frozen image = 0
16:10:31 INF - Backup copy = 0
16:10:31 INF - Master server = tweek-2
16:10:31 INF - Media server = nbumedia02-2
16:10:31 INF - Multiplexing = 0
16:10:31 INF - New data socket = nbumedia02-2.IPC:/usr/openv/var/tmp/vnet-11651516637429752684000010331-G7YGj9;7af603c2e64068af1aa8e43aa9bf7626;11;3600
16:10:31 INF - Encrypt = 0
16:10:31 INF - Use shared memory = 0
16:10:31 INF - Compression = 0
16:10:31 INF - Encrypt = 0
16:10:31 INF - Keep logs = 7
16:10:31 INF - Client read timeout = 6000
16:10:31 INF - Media mount timeout = 0
16:10:34 INF - Mounted media id U02499
16:10:36 INF - Beginning backup on server nbumedia02-2 of client pm01dbadm01-2
16:10:54 INF - Server status = 0
16:10:54 INF - Backup by oracle on client pm01dbadm01-2 using policy ORACLE_Exadata_PROD, sched Default-Application-Backup:the requested operation was successfully completed

This is a recent log. Shouldn't multiplexing be greater than zero? If it should is this because it's picking up the policy setting on the tape side? If so, can I or should I override it with a netbackup send, or is that even possible?

[Mouse]

I am sorry mate, but it looks to me you did not get what @Marianne was trying to say.

You need to take it to Oracle if you need to debug the RMAN script: how many "channels" or parallel jobs in NBU you create is irrelevant in this context. When the data stream reaches NBU, it's totally irrelevant if MPX=1 or MPX=8, it won't alter number of files in backupset - NBU is blind about the contents of backup sets. It's Oracle wthat wraps files into backup sets and feeds it to NBU using the RMAN media management layer - NBU settings have nothing to do with that. Now, as it is clear now that NBU is not involved and if you feel that Oracle does not do its job properly or has specific wrapping for files from the FRA and/or log files (ask Oracle if it is the case) - why don't just raise a question how RMAN should look like.

[safedba]

I'm creating an SR w/Oracle. I've searched matalink, but nothing pops. You also can't simultaneously backup to the flash recovery area and tape at the same time (there was a not on this and no update). Of course the free support here has already been better that Oracle's. I've been a dba for years and the VERY last thing I ever want to do is ask Oracle. They want you to wrap up the kitchen sink and send it to them when all you want is an answer to a specific question. It's like the despair.com poster w/regard to support "We're not satisfied until you're not satisfied." :)

[Mouse]

Look mate, totally understand your concerns and would be very happy to help with the issue, had it be related to NBU. Unfortunately this bit of the backup cycle is handled by Oracle exclusively and my DBA skills are a bit too limited to give you a better advice. Let us know if you end up resolving it, or maybe peeps more experienced with Oracle will chip in here.

[safedba]

I will definitely post the solution if I find one....Tks!

[safedba]

Ok...the problem is that as far as netbackup is concerned, it backs up the files it sees in the flash recovery area. So when I open up 12 channels it splits the data its backup up into multiple files so when we back up the recovery area it creates 1 job per file. To "fix" this we have at least for now sacrificed speed of backup to disk by using one channel and this creates as few files as possible. The files are then bigger, and we have fewer jobs as a result.