Hmm, never noticed this

Alex_Zn · ‎07-04-2012

I have checked, that in case of failure of automatic full backup, cumulative backup do not run and even not quied. I thought that next commutative backup, should start from the last successful full, but this not happens, cause i have successful full backup. Is this is normal NetBackup behavior or bug ?

Marianne · ‎07-04-2012

Not normal behaviour... Please show us your policy config? bppllist policy-name -U

Handy NetBackup Links

Alex_Zn · ‎07-04-2012

C:\Users\Administrator>bppllist rhel6_bmr -U
------------------------------------------------------------

Policy Name: rhel6_bmr

Policy Type:         Standard
Active:              yes
Effective date:      03/30/2012 12:58:24
Client Compress:     no
Follow NFS Mounts:   no
Cross Mount Points: no
Collect TIR info:    no
Block Incremental:   no
Mult. Data Streams: no
Client Encrypt:      no
Checkpoint:          no
Policy Priority:     0
Max Jobs/Policy:     Unlimited
Disaster Recovery:   0
Collect BMR info:    no
Residence:           STU_dpool1
Volume Pool:         NetBackup
Server Group:        *ANY*
Keyword:             (none specified)
Data Classification:       -
Residence is Storage Lifecycle Policy:    no
Application Discovery:      no
Discovery Lifetime:      28800 seconds
ASC Application and attributes: (none defined)

Granular Restore Info: no
Ignore Client Direct: no
Enable Metadata Indexing: no
Index server name: NULL
Use Accelerator: no
HW/OS/Client: Linux RedHat2.6.18 192.168.64.145

Include: /home/znachkov/7.5.0.3.aix

Schedule:              Full
    Type:                Full Backup
    Frequency:           every 3 hours
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     1 (2 weeks)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     00:00:00 --> Sunday     24:00:00
          Monday     00:00:00 --> Monday     24:00:00
          Tuesday    00:00:00 --> Tuesday    24:00:00
          Wednesday 00:00:00 --> Wednesday 24:00:00
          Thursday   00:00:00 --> Thursday   24:00:00
          Friday     00:00:00 --> Friday     24:00:00
          Saturday   00:00:00 --> Saturday   24:00:00

Schedule:              CummulativeBackup
    Type:                Cumulative Incremental Backup
    Frequency:           every 1 hour
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     1 (2 weeks)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     00:00:00 --> Sunday     24:00:00
          Monday     00:00:00 --> Monday     24:00:00
          Tuesday    00:00:00 --> Tuesday    24:00:00
          Wednesday 00:00:00 --> Wednesday 24:00:00
          Thursday   00:00:00 --> Thursday   24:00:00
          Friday     00:00:00 --> Friday     24:00:00
          Saturday   00:00:00 --> Saturday   24:00:00

C:\Users\Administrator>

mph999 · ‎07-04-2012

OK, only way to investigate this, is after a Full backup has failed, but before the cumulative should run, run :

nbpemreq -subsystems 1 screen >filename.txt

Look in here for the client /policy pair to see what NBU is planning to run ...

Martin

Alex_Zn · ‎07-04-2012

Ok that what i see:
======================================
Root task ID:00000000025214F8 CTX:00000000025214F8 (refs: 5) (Lock ID:00000000024CDAB0 (refs: 2))
PolicyClient::rhel6_bmr/192.168.64.145 Task
Schedule calculation delay: 0:01:00 (60)
Client offline: false
Active: true
    Scheduler::rhel6_bmr/192.168.64.145::0
    Scheduling calculation made at: 2012/07/05 09:04:35 (1341468275)
    Earliest time based on failure history: 2012/07/05 10:03:35 (1341471815)
    Stream's earliest times based on failure history
      0: 2012/07/05 10:03:35 (1341471815)
    Next schedule to run: Full
      Due: 2012/07/04 12:11:38 (1341393098)
      Start: 2012/07/05 10:03:35 (1341471815)
      Window closes: NB_INFINITY (2147483647)
      Reasoning:
        CummulativeBackup::0: Only/first schedule checked, DT=2012/07/04 11:11:38 (1341389498), RST=2012/07/05 10:03:35 (1341471815), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 10:03:35 (1341471815)
        Full::0: Has greater frequency (longer period) than previous, DT=2012/07/04 12:11:38 (1341393098), RST=2012/07/05 10:03:35 (1341471815), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 10:03:35 (1341471815)
    ======================================
    Last Backup Data
      Schedule: CummulativeBackup::0
        Retention level: 1
        Backup type: ST_CINC (4)
        Frequency schedule (1:00:00 (3600))
          Last backup data as of: 2012/07/04 23:59:36 (1341435576)
          Number of files: 0
          Image size: 32
          Estimate size: 38
          Snap time: 0:00:00 (0)
          Last backup: 2012/07/04 10:11:38 (1341385898)
          Run time: 2012/07/04 10:11:38 (1341385898)
          Last dumplevel: 0
      Schedule: Full::0
        Retention level: 1
        Backup type: ST_FULL (0)
        Frequency schedule (3:00:00 (10800))
          Last backup data as of: 2012/07/05 09:04:35 (1341468275)
          Number of files: 7
          Image size: 587808
          Estimate size: 2180083
          Snap time: 0:00:00 (0)
          Last backup: 2012/07/04 09:11:38 (1341382298)
          Run time: 2012/07/04 09:11:38 (1341382298)
          Last dumplevel: 0
    ======================================
Reuseable scheduler present
    Resumeable jobs: 0
    Queued for resuming: 0
Task Incomplete, current status PEM_EC_INCOMPLETE (-2)
Task in handler Time Task Handler ID:00000000024E1648 (refs: 4) (Lock ID:00000000024CD0B0 (refs: 1))
Active subtask count: 0
    Subtask[0] slot empty
======================================
Sub Task History
    Max history length: 20
    Deleted history count: 56
      ======================================
      Job: jobid=1402
      Pem Job State: PJS_NORMAL_END (6)
      Submission type: JBACKTYPE_SCHEDULED (1)
      Schedule type: ST_FULL (0)
      Start time: 2012/07/05 08:48:03 (1341467283)
      Window closes: NB_INFINITY (2147483647)
      Last completion: 2012/07/05 09:03:35 (1341468215)
      Next Try: 5
      Retry at: 2012/07/05 09:03:27 (1341468207)
      Copies: 0
      Policy Name: rhel6_bmr
      Schedule Name: Full
      Client Name: 192.168.64.145
      Stream Number: 0
      Work Unit: rhel6_bmr:192.168.64.145:0
      Media Server: nbu1md.sbrf.local
      Script Flags: REG_BACKUP_NOTIFY_SCRIPT REG_SESSION_NOTIFY_SCRIPT (19)
      Sends Email
      Retryable Job
      Records failures
      BRM Started
      User: root
      Locale: en
      Job is recoverable
      Resume/retry Resource Selection: STANDARD_RESOURCE (0)
        Resumeable jobs: 0
        Queued for resuming: 0
      Job Status Strategy (Virtual Parent Default)
        Internal -> NBU mapping
          Default -> EC_client_aborted (50)
        Strategy STRATEGY_PURE_SUCCESS
          Applying default strategy
        Strategy STRATEGY_PARTIAL_SUCCESS
          Applying default strategy
        Strategy STRATEGY_SUSPENDED
          Applying default strategy
        Strategy STRATEGY_RETRYABLE
          Applying default strategy
        Strategy STRATEGY_INCOMPLETE
          Applying default strategy
        Strategy STRATEGY_INTERNAL
          Applying default strategy
        Strategy STRATEGY_ADMINMAIL
          Applying default strategy
        Strategy STRATEGY_CLIENTMAIL
          Applying default strategy
        Strategy STRATEGY_DSUFULL
          Applying default strategy
        Strategy STRATEGY_FAILUREHISTORY
          Applying default strategy
        Strategy STRATEGY_EXITSCRIPT
          Applying default strategy
        Strategy STRATEGY_FAILED_ATTEMPT
          Applying default strategy
        Strategy STRATEGY_CANCELALL
          Applying default strategy
      Task completed with status EC_files_dont_exist (71) at 2012/07/05 09:03:35 (1341468215 usec=51375)
      Active subtask count: 0
      BaseJob Class
        Backup Type: JTYPE_BACKUP (0)
        SubType: Standard (0)
        Schedule Type: ST_FULL (0)
        Default Resource Allocation Strategy
        Initial Resource Selection: STANDARD_RESOURCE (0)
        Snap Time: 0:00:00 (0)
        Frozen Image Time: 0:00:00 (0)
        Last Backup Time: 0:00:00 (0)
        Dump level: 0
        Max Streams: 1
        Policy Priority: 0
        Secondary Priority: 000000000000656C = (family) 0000000000000000 + (job) 000000000000656C
        Reference BirthTime:0:00:00 (0)
        Reference BackupId:
        Reference JobId:0
        Source/Destination Copy Number:0
        NdmpHostName:
        Job Method:0
        Scheduler
          Scheduler::rhel6_bmr/192.168.64.145::0
          Scheduling calculation made at: 2012/07/05 07:49:03 (1341463743)
          Earliest time based on failure history: 2012/07/05 08:48:03 (1341467283)
          Stream's earliest times based on failure history
            0: 2012/07/05 08:48:03 (1341467283)
          Next schedule to run: Full
            Due: 2012/07/04 12:11:38 (1341393098)
            Start: 2012/07/05 08:48:03 (1341467283)
            Window closes: NB_INFINITY (2147483647)
            Reasoning:
              CummulativeBackup::0: Only/first schedule checked, DT=2012/07/04 11:11:38 (1341389498), RST=2012/07/05 08:48:03 (1341467283), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 08:48:03 (1341467283)
              Full::0: Has greater frequency (longer period) than previous, DT=2012/07/04 12:11:38 (1341393098), RST=2012/07/05 08:48:03 (1341467283), WCT=NB_INFINITY (2147483647), FPE=2012/07/05 08:48:03 (1341467283)
          ======================================
          Last Backup Data
            Schedule: CummulativeBackup::0
              Retention level: 1
              Backup type: ST_CINC (4)
              Frequency schedule (1:00:00 (3600))
                Last backup data as of: 2012/07/04 23:59:36 (1341435576)
                Number of files: 0
                Image size: 32
                Estimate size: 38
                Snap time: 0:00:00 (0)
                Last backup: 2012/07/04 10:11:38 (1341385898)
                Run time: 2012/07/04 10:11:38 (1341385898)
                Last dumplevel: 0
            Schedule: Full::0
              Retention level: 1
              Backup type: ST_FULL (0)
              Frequency schedule (3:00:00 (10800))
                Last backup data as of: 2012/07/05 07:49:03 (1341463743) (stale)
                Number of files: 7
                Image size: 587808
                Estimate size: 2180083
                Snap time: 0:00:00 (0)
                Last backup: 2012/07/04 09:11:38 (1341382298)
                Run time: 2012/07/04 09:11:38 (1341382298)
                Last dumplevel: 0
      SimpleJob Class
======================================
Task completion callbacks
    ID:00000000024CA0D0 param=0000000000000000, phase=0
    ID:00000000024835D0 param=0000000000000000, phase=0
======================================
Job Tasks
ID:00000000024E60E8 CTX:00000000025214F8 (refs: 3) (Lock ID:00000000024CDAB0 (refs: 2))
-------------------------------------------------------------------------------

Marianne · ‎07-04-2012

We see this behaviour due to the Global setting of # tries per ## hours. If a backup fails, this setting causes NBU to ignore schedules until the number of hours have passed. The default is 2 tries every 12 hours. I believe this is a bug. As a workaround, we change the setting to something like 1 try every 2 hours.

Handy NetBackup Links

mph999 · ‎07-05-2012

Hmm, never noticed this personally.

Full sched - every 3 hours hrs and backup win open all day

Cumulateive sched - every 3 hrs and backup window open all day.

I think NBU uses this logic ... because both schedules have open windows at the same time.

It is treating the Full as more important that the cumulative, within the open window which is still open, it is gong to run the full again next.

I've not seen this exact set of conditions before, but I have seen similar with calendar based schedules and retry after run-day set, which caused what you might think is the wrong schedule to run.

So, I think that the next job to run should be Full, BUT, certainly, if this only runs the next day or 12 hours later then something is up.

Just an idea ...

Martin

Alex_Zn · ‎07-05-2012

I have changed attributes to try 4 times during 1 hour, nothing have changed.

Marianne · ‎07-05-2012

I see this in schedule config:

     Schedule:              Full
    Type:                Full Backup
    Frequency:           every 3 hours

    Schedule:              CummulativeBackup
    Type:                Cumulative Incremental Backup
    Frequency:           every 1 hour

After you have changed to 4 tries every 1 hour, did you confirm that this change has been 'implemented'? Some config changes need a restart.

To confirm, run:
bpconfig -L

Also run:
nbpemreq -updatepolicies
(to re-read policy config)

followed by:
nbpemreq -predict -date <mm/dd/yyyy HH:MM:SS>
(make date and time 3 hours from now - this will show us all backup due within next 3 hours)

(I would personally never recommend 4 tries every 1 hour...)

Handy NetBackup Links

VOX

Commulative backups do not run automaticly after failed full