6.0MP4 runs incrementals and fulls on wrong days and times

[bitminer]

We're trying to debug a system where differential incrementals are scheduled for weekdays and fulls for the weekend.

But the fulls run Wednesday night, not Saturday morning, and the incrementals are only every other day, if at all.

The environment is SUSE Linux 9 SP3 (SLES).  The NBU version is 6.0MP4.

The policy reads

Code:

  Schedule:          Full    Type:            Full Backup    Frequency:       every 3 days    Maximum MPX:     4     Daily Windows:          Saturday   04:30:00  -->  Monday     04:30:00

 

and

Code:

  Schedule:          Incremental    Type:            Differential Incremental Backup    Frequency:       every 1 days    Maximum MPX:     4   Daily Windows:          Tuesday    04:30:00  -->  Tuesday    10:30:00          Wednesday  04:30:00  -->  Wednesday  10:30:00          Thursday   04:30:00  -->  Thursday   10:30:00          Friday     04:30:00  -->  Friday     10:30:00

 

but the results are

Code:

# bpimagelist -U -hoursago 320 -client porsche | sort----------------  ---------- -------- --------  -  ------------ ------------02/27/2008 18:26  03/29/2008    84097 20910177  N  Full Backup  Linux_Backup02/29/2008 18:58  03/14/2008     1575   481457  N  Differential Linux_Backup03/07/2008 16:41  03/21/2008     7360   589437  N  Differential Linux_BackupBacked Up         Expires       Files       KB  C  Sched Type   Policy

 

That's three problems: the frequency is wrong, the dates are wrong, and the times are wrong.

The machine was formerly set to PST/PDT, then changed to UTC.  Right now the time is Sat Mar  8 00:59:19 UTC 2008

We've just restarted all the NBU software (init.d/netbackup stop and start) and are trying to "force" a full backup on the weekend by faking out the incrementals (4 day interval).  fingers are crossed.

Any suggestions on these?

B

[Alexander_Harri]

Have you tried changing the schedules to calendar based (instead of frequency based)? Set the start date for the full backups to every Saturday.

I've found that frequency based tends to result in start time creep because the wait period is based on when the last backup finished.

[TomerG]

Frequency based should work just as well... the only reason there is time creep is because it isn't set correctly... instead of incrementals being every "1 day", they should run every "12 hours" (or sometime larger than the backup window) so that it will run as early as possible the next day (e.g. at 4 or whenever the window opens).

Likewise the full should only run on the day that the window is open (even though they say run every 3 days... since there is no open window on the weekdays, it shouldn't run regardless).

Unless someone is running manual (run now) type backups??? (either in the GUI or via scripts or an external scheduler like cron or Windows Task manager)

[bitminer]

So we have a partial solution:

First, the machine time was UTC, and NBU was Coordinated Universal Time (ETC/Universal), and is now changed to Coordinated Universal time (ETC/UTC).  (File / Set Timezone / ...)

As a result, the incrementals began operating at the scheduled time 04h30 instead of 17h30.  I'm not sure that this timezone is correct or that merely changing it corrected the NBU concept of time, but the change worked.

Since the fulls and incrementals were on the wrong days, we suspected an interlock between the window and schedule and the tape capability.  This is a "manually operated" system, that is, no tape robot.  Once a tape had incrementals on it, NBU would not write fulls onto the same tape, and so the weekends became inactive due to the wrong resources (the wrong type of tape).  We would only change one tape per day, and it's not possible to change mid-weekend to a more suitable tape.

The second fix was to enable mixed full/incremental on tapes.

Code:

nbemmcmd -changesetting -allow_multiple_retentions_per_media yes -machinename %insert-your-server-name-here%

 
We'll find out this weekend if these two changes helped.

The conclusion I draw is that the NBU frequency-based scheduling is a tricky system with many unintended consequences.   We had thought to mix incrementals and fulls without much thought to whether NBU considered these different.  After consideration of the concept of expiry times, backup types, and tape types, it all becomes clear, or at least less muddy.

B

[bitminer]

The fix described above worked, for about a week.  We rebooted the machines (for other reasons), operations performed normally according to plan, then a week later the incorrect timing and events started again. Incrementals at the wrong time of day, and no fulls on the weekend (they do start on Monday when nothing is supposed to happen).

Any suggestions for why the configuration would revert?

[bitminer]

We did eventually solve this.

As it turns out, due to the manual nature of the tape changing, NetBackup has no way to play catchup if it has no tape to write to according to the schedule.  With a real robot, NBU would just load a tape and go.

With manual tape changes, a tape insert was missed.  Meanwhile, the incremental was scheduled, and when a tape is eventually inserted, the backups start almost immediately.  Even at 14:00 in the afternoon when the schedule says for 04:30.  That part is a bug.

When the backup concludes, the tape ejects.  This is fine except for the weekend when we expect the tape to be used for full backups....(remember the plan is that the operator inserts a tape Friday that won't be used until Saturday morning).  Since the operator inserts the tape on Friday and walks away, NBU industriously performs an incremental and ejects the tape.  The operator person does not notice, but NBU has no tape to write to Saturday morning.

The fix is to ask the operator to re-insert the tape Friday night, after the incrementals complete, and then the correct level of backups occur Saturday (because we have enabled mixing fulls and incrementals on the same tape) and following weekdays.  And, at the correct time of 04:30.

So, to summarize our manual backup process:

• 1. setup schedule of fulls and incrementals using windows on weekends and weekdays, specifying 2 weeks expiry for the incrementals and 4 weeks expiry for the fulls
• 2. pre-initialize 8 weeks of tapes (that is, longer than any expiry time plus overhead) with one tape per weekday for 8 weeks.
• 3. enable mixing fulls and incrementals on the same tape
• 4. instruct the operator to change tapes once a weekday, according to a plan indicating which tape is inserted on which calendar day.  For example, TUE3 on April 15.
• 5. instruct the operator to check the tape is still inserted an hour later or before he goes home for the day
• 6. (during installation and configuration) setup the LTO-3 tape and preset the buffering to a higher value of 24 to avoid LTO-3 performance problems
• 7. (during installation and configuration) verify timezone

And that seems to be it. 

B