NBU 7.5.0.5 miss some scheduled jobs randomly

[Michele_Nicosia]

Good morning all,

   i've just realized that some tasks schedule, within policies, got missed randomly.

First cases were with CATALOG_BACKUP, some days no catalog without any errors or issues.

Today i've found a SAP FULL DB that never get started within the BU window, again without any real issues around: master up, media up, agent up, and other tasks ongoing normaly.

Where do i start to troubleshoot something like this?

Can i set an alert in OpsCenter to get some alerts, for example?

Thank you.

Kind Regards,

Michele

[Mark_Solutions]

Could we see the output for the catalog and SAP policy please - i am guessing it is related to your window and frequency causing it

bppllist policyname -U

[RamNagalla]

I would start from Verifying the policy configuration.

please post the output of below command

bppllist <policyname> -L

[Michele_Nicosia]

Hi Gents,

 thank you for the support.

Here it is some more info:

------------------------------------------------------------

Policy Name:       MNG_AMS20_SAP_DBPW

  Policy Type:         SAP
  Active:              yes
  Effective date:      08/02/2011 17:47:45
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     100
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           MnG
  Volume Pool:         DataStore
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      0 seconds
ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  Windows-x64   Windows2003   ph300202.ams20.vzbi.caas
                 Windows-x64   Windows2008   ph967702.ams20.vzbi.caas

  Include:  C:\backup-scripts\backup-online_NetBackup.bat

  Schedule:              DAILY_FULL_SAP_DB
    Type:                Automatic Full Backup
    Frequency:           every 1 day
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     1 (2 weeks)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     09:20:00  -->  Sunday     17:20:00
          Monday     08:20:00  -->  Monday     15:20:00
          Tuesday    02:30:00  -->  Tuesday    06:30:00
          Wednesday  02:30:00  -->  Wednesday  06:30:00
          Thursday   02:30:00  -->  Thursday   06:30:00
          Friday     02:30:00  -->  Friday     06:30:00
          Saturday   02:30:00  -->  Saturday   09:30:00

  Schedule:              SAP
    Type:                Application Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     1 (2 weeks)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     00:00:00  -->  Sunday     24:00:00
          Monday     00:00:00  -->  Monday     24:00:00
          Tuesday    00:00:00  -->  Tuesday    24:00:00
          Wednesday  00:00:00  -->  Wednesday  24:00:00
          Thursday   00:00:00  -->  Thursday   24:00:00
          Friday     00:00:00  -->  Friday     24:00:00
          Saturday   00:00:00  -->  Saturday   24:00:00

Policy Name:       MNG_AMS20_SAP_DBPW
Options:           0x0
template:          FALSE
audit_reason:         ?
Names:             (none)
Policy Type:       SAP (17)
Active:            yes
Effective date:    08/02/2011 17:47:45
Mult. Data Stream: no
Perform Snapshot Backup:   no
Snapshot Method:           (none)
Snapshot Method Arguments: (none)
Perform Offhost Backup:    no
Backup Copy:               0
Use Data Mover:            no
Data Mover Type:           2
Use Alternate Client:      no
Alternate Client Name:     (none)
Use Virtual Machine:      0
Hyper-V Server Name:     (none)
Enable Instant Recovery:   no
Policy Priority:   100
Max Jobs/Policy:   Unlimited
Disaster Recovery: 0
Collect BMR Info:  no
Keyword:           (none specified)
Data Classification:       -
Residence is Storage Lifecycle Policy:    no
Client Encrypt:    no
Checkpoint:        no
Residence:         MnG
Volume Pool:       DataStore
Server Group:      *ANY*
Granular Restore Info:  no
Exchange Source attributes:              no
Exchange 2010 Preferred Server: (none defined)
Application Discovery:      no
Discovery Lifetime:      0 seconds
ASC Application and attributes: (none defined)
Generation:      112
Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
Use Accelerator:  no
Client/HW/OS/Pri/DMI:  ph300202.ams20.vzbi.caas Windows-x64 Windows2003 0 0 0 0 ?
Client/HW/OS/Pri/DMI:  ph967702.ams20.vzbi.caas Windows-x64 Windows2008 0 1 0 0 ?
Include:           C:\backup-scripts\backup-online_NetBackup.bat
Schedule:              DAILY_FULL_SAP_DB
  Type:                FULL SSAP (0)
  Frequency:           1 day(s) (86400 seconds)
  Maximum MPX:         1
  Synthetic:           0
  Checksum Change Detection: 0
  PFI Recovery:        0
  Retention Level:     1 (2 weeks)
  u-wind/o/d:          0 0
  Incr Type:           DELTA (0)
  Alt Read Host:       (none defined)
  Max Frag Size:       0 MB
  Number Copies:       1
  Fail on Error:       0
  Residence:           (specific storage unit not required)
  Volume Pool:         (same as policy volume pool)
  Server Group:        (same as specified for policy)
  Residence is Storage Lifecycle Policy:         0
  Schedule indexing:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      009:20:00  017:20:00   009:20:00  017:20:00
   Monday      008:20:00  015:20:00   032:20:00  039:20:00
   Tuesday     002:30:00  006:30:00   050:30:00  054:30:00
   Wednesday   002:30:00  006:30:00   074:30:00  078:30:00
   Thursday    002:30:00  006:30:00   098:30:00  102:30:00
   Friday      002:30:00  006:30:00   122:30:00  126:30:00
   Saturday    002:30:00  009:30:00   146:30:00  153:30:00
Schedule:              SAP
  Type:                UBAK SAP (2)
  Frequency:           7 day(s) (604800 seconds)
  Maximum MPX:         1
  Synthetic:           0
  Checksum Change Detection: 0
  PFI Recovery:        0
  Retention Level:     1 (2 weeks)
  u-wind/o/d:          0 0
  Incr Type:           DELTA (0)
  Alt Read Host:       (none defined)
  Max Frag Size:       0 MB
  Number Copies:       1
  Fail on Error:       0
  Residence:           (specific storage unit not required)
  Volume Pool:         (same as policy volume pool)
  Server Group:        (same as specified for policy)
  Residence is Storage Lifecycle Policy:         0
  Schedule indexing:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      000:00:00  024:00:00   000:00:00  024:00:00
   Monday      000:00:00  024:00:00   024:00:00  048:00:00
   Tuesday     000:00:00  024:00:00   048:00:00  072:00:00
   Wednesday   000:00:00  024:00:00   072:00:00  096:00:00
   Thursday    000:00:00  024:00:00   096:00:00  120:00:00
   Friday      000:00:00  024:00:00   120:00:00  144:00:00
   Saturday    000:00:00  024:00:00   144:00:00  168:00:00
 

---------------------------------------------------

Policy Name:       CATALOG_BACKUP

  Policy Type:         NBU-Catalog
  Active:              yes
  Effective date:      07/21/2011 20:42:26
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     100
  Max Jobs/Policy:     1
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           NBU-CAT
  Volume Pool:         CatalogBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      0 seconds
ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  Linux         Linux         ph469702.ams20.vzbi.caas

  Include:  CATALOG_DRIVEN_BACKUP

  Schedule:              CATALOG_BACKUP
    Type:                Full Backup
    Frequency:           every 1 day
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     1 (2 weeks)
    Number Copies:       1
    Fail on Error:       0
    Residence:           NBU-CAT
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     12:30:00  -->  Sunday     13:10:00
          Monday     13:50:00  -->  Monday     14:30:00
          Tuesday    12:40:00  -->  Tuesday    13:20:00
          Wednesday  12:40:00  -->  Wednesday  13:20:00
          Thursday   12:40:00  -->  Thursday   13:20:00
          Friday     12:40:00  -->  Friday     13:20:00
          Saturday   12:40:00  -->  Saturday   13:20:00

Catalog Disaster Recovery Configuration:
  Email Address:   (none specified)
  Disk Path:       /local/NBUDR
  User Name:       (none specified)
  Pass Word:       (none specified)
  Critical policy: (none specified)

Policy Name:       CATALOG_BACKUP
Options:           0x0
template:          FALSE
audit_reason:         ?
Names:             (none)
Policy Type:       NBU-Catalog (35)
Active:            yes
Effective date:    07/21/2011 20:42:26
Mult. Data Stream: no
Perform Snapshot Backup:   no
Snapshot Method:           (none)
Snapshot Method Arguments: (none)
Perform Offhost Backup:    no
Backup Copy:               0
Use Data Mover:            no
Data Mover Type:           2
Use Alternate Client:      no
Alternate Client Name:     (none)
Use Virtual Machine:      0
Hyper-V Server Name:     (none)
Enable Instant Recovery:   no
Policy Priority:   100
Max Jobs/Policy:   1
Disaster Recovery: 0
Collect BMR Info:  no
Keyword:           (none specified)
Data Classification:       -
Residence is Storage Lifecycle Policy:    no
Client Encrypt:    no
Checkpoint:        no
Residence:         NBU-CAT
Volume Pool:       CatalogBackup
Server Group:      *ANY*
Granular Restore Info:  no
Exchange Source attributes:              no
Exchange 2010 Preferred Server: (none defined)
Application Discovery:      no
Discovery Lifetime:      0 seconds
ASC Application and attributes: (none defined)
Generation:      25
Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
Use Accelerator:  no
Client/HW/OS/Pri/DMI:  ph469702.ams20.vzbi.caas Linux Linux 0 0 0 0 ?
Include:           CATALOG_DRIVEN_BACKUP
Schedule:              CATALOG_BACKUP
  Type:                FULL (0)
  Frequency:           1 day(s) (86400 seconds)
  Maximum MPX:         1
  Synthetic:           0
  Checksum Change Detection: 0
  PFI Recovery:        0
  Retention Level:     1 (2 weeks)
  u-wind/o/d:          0 0
  Incr Type:           DELTA (0)
  Alt Read Host:       (none defined)
  Max Frag Size:       0 MB
  Number Copies:       1
  Fail on Error:       0
  Residence:           NBU-CAT
  Volume Pool:         (same as policy volume pool)
  Server Group:        (same as specified for policy)
  Residence is Storage Lifecycle Policy:         0
  Schedule indexing:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      012:30:00  013:10:00   012:30:00  013:10:00
   Monday      013:50:00  014:30:00   037:50:00  038:30:00
   Tuesday     012:40:00  013:20:00   060:40:00  061:20:00
   Wednesday   012:40:00  013:20:00   084:40:00  085:20:00
   Thursday    012:40:00  013:20:00   108:40:00  109:20:00
   Friday      012:40:00  013:20:00   132:40:00  133:20:00
   Saturday    012:40:00  013:20:00   156:40:00  157:20:00
Catalog Disaster Recovery Configuration:
  Email Address:   (none specified)
  Disk Path:       /local/NBUDR
  User Name:       (none specified)
  Pass Word:       (none specified)
  Critical policy: (none specified)
 

[RamNagalla]

Policy Name:       MNG_AMS20_SAP_DBPW 

   Monday     08:20:00  -->  Monday     15:20:00
          Tuesday    02:30:00  -->  Tuesday    06:30:00

may be this would be the issue...

once after Monday backup tirggred and compleated at 8:20, then next due would be at Tuesday 8:20 but unfortunatly the backup window is closed by 6.30, which is not giving the 1 day frequency.. so you probalby miss the backup on Tuesday.

same case  i am seeing for policy CATALOG_BACKUP

  Monday     13:50:00  -->  Monday     14:30:00

          Tuesday    12:40:00  -->  Tuesday    13:20:00

[Mark_Solutions]

As i suspected - your frequency is too much causing overlap

When using frequency based scheduling remember that the window you set is just a start window and as long as the job gets running during that period it will continue (not if it is just queued though)

It is best to have something like:

Window 08:20 to 12:20 (which is a 4 hour duration) and then a frequency of 5 hours

In this way the frequency is just higher than the window duration so it will only run once during the backup window - but it will always be ready to run immediately when the next window opens

When running jobs daily a frequency of 1 day is too much, but any less means you window needs to be less too

Hope all this makes sense and helps

[Michele_Nicosia]

Hi Nagalla,

 infact i see today (tue) it was missed.

For catalog pretty the same.

The reason about these BU windows depends heavily on the fact i need to avoid overlap for other jobs.

What could be done to have exactly that time range ?

Do i need to change the starting date to have the bu window cover 24hrs then?

Thank you.

Regards,

Michele

[Mark_Solutions]

The question is do you really need a 24 hour window?

It is just a start time not a run time

If you do not have many resources (disk or tape) then you need to stagger your policies to suit your environment

Without an indepth look at your environment it is hard to say exactly what you need

Do you often have jobs queued - if so for how long and why?

Do you use disk or tape to backup to - how many streams can you run at a time and how many jobs do you run per day

The above are the things to look at when deciding what needs to be done with your system but a 24 hour duration window is very rarely needed

Hope this helps and gives food for thought

[RamNagalla]

try to keep the same start window for all days.. that would avoide these issue...

or else,make sure that the next start windows is falling in 24 hours range...

in above case if you have a backup window till 8:30 in tuesday , you would not have miss the backup... 

but better to have the same backup windows for all days.

[RamNagalla · Accepted solution]

alternatly you can try changing the scheule from Freqeucy to Calander based and select all days in recurring days  without changing the backup start times.

[ontherocks]

In place of frequency based schedule go for Calender based schedule.

[Michele_Nicosia]

Thanks to all.

I've got the issue, and now moved from frequency based to calendar.

I already know problems about overlap for different tasks, specifically about SAP ones.

So the necessity to have that kind of BU window, that do not cover 24hrs, it is based on experience on that system.

So, thank you for the tips.

[Marianne]

As the others pointed out, this is your problem:

Frequency:           every 1 day
    ..
    Daily Windows:
          Sunday     09:20:00  -->  Sunday     17:20:00
          Monday     08:20:00  -->  Monday     15:20:00
          Tuesday    02:30:00  -->  Tuesday    06:30:00
          Wednesday  02:30:00  -->  Wednesday  06:30:00
          Thursday   02:30:00  -->  Thursday   06:30:00
          Friday     02:30:00  -->  Friday     06:30:00
          Saturday   02:30:00  -->  Saturday   09:30:00

Frequency of 'every 1 day' means 'once every 24 hours'.

So, when the backup window opens, NBU checks:
'when was the last successful backup?'  (the backup start time is checked)
'has enough time (24 hours) elapsed since the last backup?'
If the answer is NO to above question, the backup will not be submitted.
 

You can see by looking at your backup window that the answer will be NO on some days.

Change the frequency to 'every 12 hours'. 
Backups cannot start every 12 hours because of the backup window. It will merely ensure that 'enough time has elapsed' when backup window opens.

Please see David Chapa's article on Frequency Based Scheduling: 
https://www-secure.symantec.com/connect/articles/netbackup-frequency-based-scheduling-1