04-26-2011 06:22 AM
Running NBU 7.0.1 on Windows.
I have one large file server that I backup using multiple streams...6 streams in total. Ran a full backup of this server over the weekend, and only 3 out of the 6 streams ran. The 3 streams that are "missing" don't even show as unsuccessful, missed, etc. Not sure if this is behavior by design or something that went wrong? There are incrementals that run in the same policy and use the same selection list, and with the incrementals things seemed fine... The next night, 6 jobs ran, 1 failed. As I have my retry interval to 2, the failed job reran and completed successfully as it should.
What am I missing in the process? This is my understanding as to how it should work:
- When the schedule begins, the parent job should activate sub jobs for all the streams
- Depending on the number of active jobs per client, the sub jobs will begin...some active, some queued. In my case, one job runs at a time.
- If a queued job cannot start within the backup window, it should report error 196 stating the backup window has closed.
I'm not sure if my parent job didn't actually activate all the jobs so they would queue, or if anything missed in the backup window was merely skipped and not reported. My parent job showed the backup as being successful. The backups stopped around 6pm, even though the window would have allowed the jobs to run until 10am the next morning so I don't understand how the backup window could be the problem here.
Any thoughts?
Solved! Go to Solution.
04-26-2011 09:02 AM
Started manually or automatically?
With frequency based schedules, each stream is evaluated separately - 'has this stream been backed up in the last 14 days?'
Please post output of backups since 2 weekends ago:
bpimagelist -client <servername> -d 04/08/2011 -e 04/25/2011 -U
Any reason why you only allow 1 active stream for this policy? This can result in major 'schedule creeping'.
Please see David Chapa's blog on frequency based schedules and how to avoid 'schedule creeping':
http://blogs.netapp.com/barandgrill/2008/10/frequency-based.html?wwparam=1256285548
04-26-2011 07:24 AM
please post policy details, bppllist -U <policyname>
04-26-2011 08:24 AM
Policy Name: ServerName
Policy Type: MS-Windows
Active: yes
Effective date: 01/28/2011 21:08:45
Backup network drvs: no
Collect TIR info: yes, with move detection
Mult. Data Streams: yes
Client Encrypt: no
Checkpoint: yes
Interval: 15
Policy Priority: 0
Max Jobs/Policy: 1
Disaster Recovery: 0
Collect BMR info: no
Residence: StorageUnitGroup1
Volume Pool: NetBackup
Server Group: *ANY*
Keyword: (none specified)
Data Classification: -
Residence is Storage Lifecycle Policy: no
Granular Restore Info: no
Ignore Client Direct: no
HW/OS/Client: Windows-x86 Windows2003 servername
Include: NEW_STREAM
Shadow Copy Components:\
C:\
NEW_STREAM
D:\Megan
D:\Shared Resources 1
NEW_STREAM
D:\Shared Resources 2
NEW_STREAM
D:\Shared Resources 3
NEW_STREAM
D:\Shared Resources
NEW_STREAM
D:\Shared Resources 4
Schedule: Full
Type: Full Backup
Frequency: every 14 days
Maximum MPX: 1
Synthetic: 0
PFI Recovery: 0
Retention Level: 9 (infinity)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Saturday 08:00:00 --> Sunday 10:00:00
Schedule: Incremental
Type: Differential Incremental Backup
Frequency: every 1 day
Maximum MPX: 1
Synthetic: 0
PFI Recovery: 0
Retention Level: 9 (infinity)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
EXCLUDE DATE 0 - 04/21/2011
EXCLUDE DATE 1 - 04/22/2011
EXCLUDE DATE 2 - 04/23/2011
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Sunday 21:00:00 --> Monday 06:00:00
Monday 21:00:00 --> Tuesday 06:00:00
Tuesday 21:00:00 --> Wednesday 06:00:00
Wednesday 21:00:00 --> Thursday 06:00:00
Thursday 21:00:00 --> Friday 06:00:00
Friday 21:00:00 --> Saturday 06:00:00
Saturday 21:00:00 --> Sunday 06:00:00
04-26-2011 09:02 AM
Started manually or automatically?
With frequency based schedules, each stream is evaluated separately - 'has this stream been backed up in the last 14 days?'
Please post output of backups since 2 weekends ago:
bpimagelist -client <servername> -d 04/08/2011 -e 04/25/2011 -U
Any reason why you only allow 1 active stream for this policy? This can result in major 'schedule creeping'.
Please see David Chapa's blog on frequency based schedules and how to avoid 'schedule creeping':
http://blogs.netapp.com/barandgrill/2008/10/frequency-based.html?wwparam=1256285548
04-26-2011 09:03 AM
If you say backup every 1 day that means 24 hours and if the job did not get finish before 10 pm, then it might be 10 pm before it starts the next time. So in the same vain if it did not finish before 10 am and the window closes it may never start it. but I have to say that if it starts 1 of the new streams it normally starts all of them.
The rule of thumb for frequency is bigger than your window, smaller than your requirement.
So looking at your incremental you have a 9 hour window and want it run once a day. So set your frequency to something between 10 and 20 hours ( I would choose 16).
Same with your full, your full window is only on Saturday night - so it will not run until that window opens.
So set it to 12 or 13 days. I will not run until the 13 days have passed AND the window is open.
Not sure if that is your issue, but it could be that the frequency is just a little to precise for the window to start it.
04-26-2011 09:47 AM
The backup jobs all initiated automatically. Only one job is set to run at a time because all of those streams come from the same physical drive, and I backup to disk so my disk targets are limited (limited bugdet). Short answer: to avoid disk trashing. If I don't create separate streams it's snapshot hell due to the millions of files on this server....and hundreds of open ones...so nothing ever completes successfully!
Ah, of course....the frequency. That is indeed the issue here....how did I miss that. There was a failed attempt at a full backup the week prior (4/17), which 3/5 jobs ran successfully (the missing 3, doh). However, perhaps off topic, here is where I get a bit lost:
I've been told that a modification to the selection list will force the backups to run as full....regardless of whether or not an incremental is to run or not (seen some incrementals run as fulls because of some silly policy or selection list modification). Friday we modified the streams, adding one, and moving a few things around. Hence why we figured that the fulls would run properly Saturday. In retrospect, I should have fully deleted ALL streams and added them fresh. I'm sure there's an ID associated with them somehow so despite their modification they didn't get queued up because they weren't supposed to :) Is there any official documentation that states EXACTLY what changes to a policy/selection list will cause a full backup....even if it's the incremental jobs that runs as a full?
I'll also change the frequency so it's less precise. Makes sense...
04-26-2011 11:37 AM
Remember that each stream gets evaluted individually - if you look on the master in the client folder under netbackup\db\images, you will see the STREAMS files. Incremental backups is based on previous backups for each stream - so, if you change the backup selection in the middle of the week, NBU will have no previous successful backup for the new streams, and thus backup everything in full.
Best to make changes just before a Full is due. If you just change the order around it should not make a difference.
04-26-2011 09:08 PM
Very Helpful!!! Thank you so much for sharing this Blog!!!