07-04-2012 08:21 AM
I have checked, that in case of failure of automatic full backup, cumulative backup do not run and even not quied. I thought that next commutative backup, should start from the last successful full, but this not happens, cause i have successful full backup. Is this is normal NetBackup behavior or bug ?
07-04-2012 08:37 AM
07-04-2012 11:02 PM
C:\Users\Administrator>bppllist rhel6_bmr -U
------------------------------------------------------------
Policy Name: rhel6_bmr
Policy Type: Standard
Active: yes
Effective date: 03/30/2012 12:58:24
Client Compress: no
Follow NFS Mounts: no
Cross Mount Points: no
Collect TIR info: no
Block Incremental: no
Mult. Data Streams: no
Client Encrypt: no
Checkpoint: no
Policy Priority: 0
Max Jobs/Policy: Unlimited
Disaster Recovery: 0
Collect BMR info: no
Residence: STU_dpool1
Volume Pool: NetBackup
Server Group: *ANY*
Keyword: (none specified)
Data Classification: -
Residence is Storage Lifecycle Policy: no
Application Discovery: no
Discovery Lifetime: 28800 seconds
ASC Application and attributes: (none defined)
Granular Restore Info: no
Ignore Client Direct: no
Enable Metadata Indexing: no
Index server name: NULL
Use Accelerator: no
HW/OS/Client: Linux RedHat2.6.18 192.168.64.145
Include: /home/znachkov/7.5.0.3.aix
Schedule: Full
Type: Full Backup
Frequency: every 3 hours
Maximum MPX: 1
Synthetic: 0
Checksum Change Detection: 0
PFI Recovery: 0
Retention Level: 1 (2 weeks)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 0
Schedule indexing: 0
Daily Windows:
Sunday 00:00:00 --> Sunday 24:00:00
Monday 00:00:00 --> Monday 24:00:00
Tuesday 00:00:00 --> Tuesday 24:00:00
Wednesday 00:00:00 --> Wednesday 24:00:00
Thursday 00:00:00 --> Thursday 24:00:00
Friday 00:00:00 --> Friday 24:00:00
Saturday 00:00:00 --> Saturday 24:00:00
Schedule: CummulativeBackup
Type: Cumulative Incremental Backup
Frequency: every 1 hour
Maximum MPX: 1
Synthetic: 0
Checksum Change Detection: 0
PFI Recovery: 0
Retention Level: 1 (2 weeks)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 0
Schedule indexing: 0
Daily Windows:
Sunday 00:00:00 --> Sunday 24:00:00
Monday 00:00:00 --> Monday 24:00:00
Tuesday 00:00:00 --> Tuesday 24:00:00
Wednesday 00:00:00 --> Wednesday 24:00:00
Thursday 00:00:00 --> Thursday 24:00:00
Friday 00:00:00 --> Friday 24:00:00
Saturday 00:00:00 --> Saturday 24:00:00
C:\Users\Administrator>
07-04-2012 11:15 PM
OK, only way to investigate this, is after a Full backup has failed, but before the cumulative should run, run :
nbpemreq -subsystems 1 screen >filename.txt
Look in here for the client /policy pair to see what NBU is planning to run ...
Martin
07-04-2012 11:25 PM
Ok that what i see:
======================================
Root task ID:00000000025214F8 CTX:00000000025214F8 (refs: 5) (Lock ID:00000000024CDAB0 (refs: 2))
PolicyClient::rhel6_bmr/192.168.64.145 Task
Schedule calculation delay: 0:01:00 (60)
Client offline: false
Active: true
Scheduler::rhel6_bmr/192.168.64.145::0
Scheduling calculation made at: 2012/07/05 09:04:35 (1341468275)
Earliest time based on failure history: 2012/07/05 10:03:35 (1341471815)
Stream's earliest times based on failure history
0: 2012/07/05 10:03:35 (1341471815)
Next schedule to run: Full
Due: 2012/07/04 12:11:38 (1341393098)
Start: 2012/07/05 10:03:35 (1341471815)
Window closes: NB_INFINITY (2147483647)
Reasoning:
CummulativeBackup::0: Only/first schedule checked, DT=2012/07/04 11:11:38 (1341389498), RST=2012/07/05 10:03:35 (1341471815), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 10:03:35 (1341471815)
Full::0: Has greater frequency (longer period) than previous, DT=2012/07/04 12:11:38 (1341393098), RST=2012/07/05 10:03:35 (1341471815), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 10:03:35 (1341471815)
======================================
Last Backup Data
Schedule: CummulativeBackup::0
Retention level: 1
Backup type: ST_CINC (4)
Frequency schedule (1:00:00 (3600))
Last backup data as of: 2012/07/04 23:59:36 (1341435576)
Number of files: 0
Image size: 32
Estimate size: 38
Snap time: 0:00:00 (0)
Last backup: 2012/07/04 10:11:38 (1341385898)
Run time: 2012/07/04 10:11:38 (1341385898)
Last dumplevel: 0
Schedule: Full::0
Retention level: 1
Backup type: ST_FULL (0)
Frequency schedule (3:00:00 (10800))
Last backup data as of: 2012/07/05 09:04:35 (1341468275)
Number of files: 7
Image size: 587808
Estimate size: 2180083
Snap time: 0:00:00 (0)
Last backup: 2012/07/04 09:11:38 (1341382298)
Run time: 2012/07/04 09:11:38 (1341382298)
Last dumplevel: 0
======================================
Reuseable scheduler present
Resumeable jobs: 0
Queued for resuming: 0
Task Incomplete, current status PEM_EC_INCOMPLETE (-2)
Task in handler Time Task Handler ID:00000000024E1648 (refs: 4) (Lock ID:00000000024CD0B0 (refs: 1))
Active subtask count: 0
Subtask[0] slot empty
======================================
Sub Task History
Max history length: 20
Deleted history count: 56
======================================
Job: jobid=1402
Pem Job State: PJS_NORMAL_END (6)
Submission type: JBACKTYPE_SCHEDULED (1)
Schedule type: ST_FULL (0)
Start time: 2012/07/05 08:48:03 (1341467283)
Window closes: NB_INFINITY (2147483647)
Last completion: 2012/07/05 09:03:35 (1341468215)
Next Try: 5
Retry at: 2012/07/05 09:03:27 (1341468207)
Copies: 0
Policy Name: rhel6_bmr
Schedule Name: Full
Client Name: 192.168.64.145
Stream Number: 0
Work Unit: rhel6_bmr:192.168.64.145:0
Media Server: nbu1md.sbrf.local
Script Flags: REG_BACKUP_NOTIFY_SCRIPT REG_SESSION_NOTIFY_SCRIPT (19)
Sends Email
Retryable Job
Records failures
BRM Started
User: root
Locale: en
Job is recoverable
Resume/retry Resource Selection: STANDARD_RESOURCE (0)
Resumeable jobs: 0
Queued for resuming: 0
Job Status Strategy (Virtual Parent Default)
Internal -> NBU mapping
Default -> EC_client_aborted (50)
Strategy STRATEGY_PURE_SUCCESS
Applying default strategy
Strategy STRATEGY_PARTIAL_SUCCESS
Applying default strategy
Strategy STRATEGY_SUSPENDED
Applying default strategy
Strategy STRATEGY_RETRYABLE
Applying default strategy
Strategy STRATEGY_INCOMPLETE
Applying default strategy
Strategy STRATEGY_INTERNAL
Applying default strategy
Strategy STRATEGY_ADMINMAIL
Applying default strategy
Strategy STRATEGY_CLIENTMAIL
Applying default strategy
Strategy STRATEGY_DSUFULL
Applying default strategy
Strategy STRATEGY_FAILUREHISTORY
Applying default strategy
Strategy STRATEGY_EXITSCRIPT
Applying default strategy
Strategy STRATEGY_FAILED_ATTEMPT
Applying default strategy
Strategy STRATEGY_CANCELALL
Applying default strategy
Task completed with status EC_files_dont_exist (71) at 2012/07/05 09:03:35 (1341468215 usec=51375)
Active subtask count: 0
BaseJob Class
Backup Type: JTYPE_BACKUP (0)
SubType: Standard (0)
Schedule Type: ST_FULL (0)
Default Resource Allocation Strategy
Initial Resource Selection: STANDARD_RESOURCE (0)
Snap Time: 0:00:00 (0)
Frozen Image Time: 0:00:00 (0)
Last Backup Time: 0:00:00 (0)
Dump level: 0
Max Streams: 1
Policy Priority: 0
Secondary Priority: 000000000000656C = (family) 0000000000000000 + (job) 000000000000656C
Reference BirthTime:0:00:00 (0)
Reference BackupId:
Reference JobId:0
Source/Destination Copy Number:0
NdmpHostName:
Job Method:0
Scheduler
Scheduler::rhel6_bmr/192.168.64.145::0
Scheduling calculation made at: 2012/07/05 07:49:03 (1341463743)
Earliest time based on failure history: 2012/07/05 08:48:03 (1341467283)
Stream's earliest times based on failure history
0: 2012/07/05 08:48:03 (1341467283)
Next schedule to run: Full
Due: 2012/07/04 12:11:38 (1341393098)
Start: 2012/07/05 08:48:03 (1341467283)
Window closes: NB_INFINITY (2147483647)
Reasoning:
CummulativeBackup::0: Only/first schedule checked, DT=2012/07/04 11:11:38 (1341389498), RST=2012/07/05 08:48:03 (1341467283), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 08:48:03 (1341467283)
Full::0: Has greater frequency (longer period) than previous, DT=2012/07/04 12:11:38 (1341393098), RST=2012/07/05 08:48:03 (1341467283), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 08:48:03 (1341467283)
======================================
Last Backup Data
Schedule: CummulativeBackup::0
Retention level: 1
Backup type: ST_CINC (4)
Frequency schedule (1:00:00 (3600))
Last backup data as of: 2012/07/04 23:59:36 (1341435576)
Number of files: 0
Image size: 32
Estimate size: 38
Snap time: 0:00:00 (0)
Last backup: 2012/07/04 10:11:38 (1341385898)
Run time: 2012/07/04 10:11:38 (1341385898)
Last dumplevel: 0
Schedule: Full::0
Retention level: 1
Backup type: ST_FULL (0)
Frequency schedule (3:00:00 (10800))
Last backup data as of: 2012/07/05 07:49:03 (1341463743) (stale)
Number of files: 7
Image size: 587808
Estimate size: 2180083
Snap time: 0:00:00 (0)
Last backup: 2012/07/04 09:11:38 (1341382298)
Run time: 2012/07/04 09:11:38 (1341382298)
Last dumplevel: 0
SimpleJob Class
======================================
Task completion callbacks
ID:00000000024CA0D0 param=0000000000000000, phase=0
ID:00000000024835D0 param=0000000000000000, phase=0
======================================
Job Tasks
ID:00000000024E60E8 CTX:00000000025214F8 (refs: 3) (Lock ID:00000000024CDAB0 (refs: 2))
-------------------------------------------------------------------------------
07-04-2012 11:38 PM
07-05-2012 12:43 AM
Hmm, never noticed this personally.
Full sched - every 3 hours hrs and backup win open all day
Cumulateive sched - every 3 hrs and backup window open all day.
I think NBU uses this logic ... because both schedules have open windows at the same time.
It is treating the Full as more important that the cumulative, within the open window which is still open, it is gong to run the full again next.
I've not seen this exact set of conditions before, but I have seen similar with calendar based schedules and retry after run-day set, which caused what you might think is the wrong schedule to run.
So, I think that the next job to run should be Full, BUT, certainly, if this only runs the next day or 12 hours later then something is up.
Just an idea ...
Martin
Martin
07-05-2012 03:30 AM
I have changed attributes to try 4 times during 1 hour, nothing have changed.
07-05-2012 03:54 AM
I see this in schedule config:
Schedule: Full
Type: Full Backup
Frequency: every 3 hours
Schedule: CummulativeBackup
Type: Cumulative Incremental Backup
Frequency: every 1 hour
After you have changed to 4 tries every 1 hour, did you confirm that this change has been 'implemented'? Some config changes need a restart.
To confirm, run:
bpconfig -L
Also run:
nbpemreq -updatepolicies
(to re-read policy config)
followed by:
nbpemreq -predict -date <mm/dd/yyyy HH:MM:SS>
(make date and time 3 hours from now - this will show us all backup due within next 3 hours)
(I would personally never recommend 4 tries every 1 hour...)