Multiple data streams not starting

[Meg_G]

Running NBU 7.0.1 on Windows.

I have one large file server that I backup using multiple streams...6 streams in total.  Ran a full backup of this server over the weekend, and only 3 out of the 6 streams ran.  The 3 streams that are "missing" don't even show as unsuccessful, missed, etc.  Not sure if this is behavior by design or something that went wrong?  There are incrementals that run in the same policy and use the same selection list, and with the incrementals things seemed fine...  The next night, 6 jobs ran, 1 failed.  As I have my retry interval to 2, the failed job reran and completed successfully as it should. 

What am I missing in the process?  This is my understanding as to how it should work:
- When the schedule begins, the parent job should activate sub jobs for all the streams
- Depending on the number of active jobs per client, the sub jobs will begin...some active, some queued.  In my case, one job runs at a time.
- If a queued job cannot start within the backup window, it should report error 196 stating the backup window has closed.

I'm not sure if my parent job didn't actually activate all the jobs so they would queue, or if anything missed in the backup window was merely skipped and not reported.  My parent job showed the backup as being successful.  The backups stopped around 6pm, even though the window would have allowed the jobs to run until 10am the next morning so I don't understand how the backup window could be the problem here.

Any thoughts?

[Will_Restore]

please post policy details, bppllist -U <policyname>

[Meg_G]

Policy Name:       ServerName

  Policy Type:         MS-Windows
  Active:              yes
  Effective date:      01/28/2011 21:08:45
  Backup network drvs: no
  Collect TIR info:    yes, with move detection
  Mult. Data Streams:  yes
  Client Encrypt:      no
  Checkpoint:          yes
       Interval:       15
  Policy Priority:     0
  Max Jobs/Policy:     1
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           StorageUnitGroup1
  Volume Pool:         NetBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no

Granular Restore Info:  no
Ignore Client Direct:  no
  HW/OS/Client:  Windows-x86   Windows2003   servername

  Include:  NEW_STREAM
            Shadow Copy Components:\
            C:\
            NEW_STREAM
            D:\Megan
            D:\Shared Resources 1
            NEW_STREAM
            D:\Shared Resources 2
            NEW_STREAM
            D:\Shared Resources 3
            NEW_STREAM
            D:\Shared Resources
            NEW_STREAM
            D:\Shared Resources 4

  Schedule:          Full
    Type:            Full Backup
    Frequency:       every 14 days
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 9 (infinity)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Saturday   08:00:00  -->  Sunday     10:00:00

  Schedule:          Incremental
    Type:            Differential Incremental Backup
    Frequency:       every 1 day
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 9 (infinity)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
      EXCLUDE DATE 0 - 04/21/2011
      EXCLUDE DATE 1 - 04/22/2011
      EXCLUDE DATE 2 - 04/23/2011
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Sunday     21:00:00  -->  Monday     06:00:00
          Monday     21:00:00  -->  Tuesday    06:00:00
          Tuesday    21:00:00  -->  Wednesday  06:00:00
          Wednesday  21:00:00  -->  Thursday   06:00:00
          Thursday   21:00:00  -->  Friday     06:00:00
          Friday     21:00:00  -->  Saturday   06:00:00
          Saturday   21:00:00  -->  Sunday     06:00:00

[Marianne · Accepted solution]

Started manually or automatically?

With frequency based schedules, each stream is evaluated separately - 'has this stream been backed up in the last 14 days?'

Please post output of backups since 2 weekends ago:

bpimagelist -client <servername> -d 04/08/2011 -e 04/25/2011 -U

Any reason why you only allow 1 active stream for this policy? This can result in major 'schedule creeping'.

Please see David Chapa's blog on frequency based schedules and how to avoid 'schedule creeping':
http://blogs.netapp.com/barandgrill/2008/10/frequency-based.html?wwparam=1256285548

[J_H_Is_gone]

If you say backup every 1 day that means 24 hours and if the job did not get finish before 10 pm, then it might be 10 pm before it starts the next time.  So in the same vain if it did not finish before 10 am and the window closes it may never start it.  but I have to say that if it starts 1 of the new streams it normally starts all of them.

The rule of thumb for frequency is bigger than your window, smaller than your requirement.

So looking at your incremental you have a 9 hour window and want it run once a day.  So set your frequency to something between 10 and 20 hours ( I would choose 16).

Same with your full, your full window is only on Saturday night - so it will not run until that window opens.

So set it to 12 or 13 days.  I will not run until the 13 days have passed AND the window is open.

Not sure if that is your issue, but it could be that the frequency is just a little to precise for the window to start it.

[Meg_G]

The backup jobs all initiated automatically.  Only one job is set to run at a time because all of those streams come from the same physical drive, and I backup to disk so my disk targets are limited (limited bugdet).  Short answer: to avoid disk trashing.  If I don't create separate streams it's snapshot hell due to the millions of files on this server....and hundreds of open ones...so nothing ever completes successfully!

Ah, of course....the frequency.  That is indeed the issue here....how did I miss that.  There was a failed attempt at a full backup the week prior (4/17), which 3/5 jobs ran successfully (the missing 3, doh).  However, perhaps off topic, here is where I get a bit lost:

I've been told that a modification to the selection list will force the backups to run as full....regardless of whether or not an incremental is to run or not (seen some incrementals run as fulls because of some silly policy or selection list modification).  Friday we modified the streams, adding one, and moving a few things around.  Hence why we figured that the fulls would run properly Saturday.  In retrospect, I should have fully deleted ALL streams and added them fresh.  I'm sure there's an ID associated with them somehow so despite their modification they didn't get queued up because they weren't supposed to :)  Is there any official documentation that states EXACTLY what changes to a policy/selection list will cause a full backup....even if it's the incremental jobs that runs as a full?

I'll also change the frequency so it's less precise.  Makes sense...

[Marianne]

Remember that each stream gets evaluted individually - if you look on the master in the client folder under netbackup\db\images, you will see the STREAMS files. Incremental backups is based on previous backups for each stream - so, if you change the backup selection in the middle of the week, NBU will have no previous successful backup for the new streams, and thus backup everything in full.

Best to make changes just before a Full is due. If you just change the order around it should not make a difference.

[rizwan84tx]

Very Helpful!!! Thank you so much for sharing this Blog!!!