Forum Discussion

Katiebee's avatar
Katiebee
Level 4
12 years ago

Backups failing -196 error within backup window Netbackup 7.1

Hi all,

I'm struggling to find out why my exchange backups are failing each week/month end, when the daily backups are working fine. The week/month end backups had been fine up until a few weeks ago and as far as I know, other than finishing migrating users to ex2010, nothing has changed. 

I'm using Netbackup 7.1.0.3 on the Master and Media servers, and the client on my ex servers is 7.1.0.3 as well.

The window that it is using runs from a Friday at 19:00 till Saturday into Sunday midnight.

05/07/2013 19:00:00 - Info nbjm(pid=3212) starting backup job (jobid=2484992) for client DAG1, policy Corp_Applications_EX_C, schedule Monthly 
05/07/2013 19:00:00 - Info nbjm(pid=3212) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=2484992, request id:{1504E3A7-E946-4BB4-A6FC-2291D0274234}) 
05/07/2013 19:00:00 - requesting resource DiskPoolGroup
05/07/2013 19:00:00 - requesting resource NBServer1-v.NBU_CLIENT.MAXJOBS.DAG1
05/07/2013 19:00:00 - requesting resource NBServer1-v.NBU_POLICY.MAXJOBS.Corp_Applications_EX_C
05/07/2013 19:00:00 - requesting resource EXCHANGE_RESOLVER.NBServer1-v.Corp_Applications_EX_C.DAG1
05/07/2013 19:00:05 - granted resource NBServer1-v.NBU_CLIENT.MAXJOBS.DAG1
05/07/2013 19:00:05 - granted resource NBServer1-v.NBU_POLICY.MAXJOBS.Corp_Applications_EX_C
05/07/2013 19:00:05 - granted resource EXCHANGE_RESOLVER.NBServer1-v.Corp_Applications_EX_C.DAG1
05/07/2013 19:00:08 - estimated 136895656 Kbytes needed
05/07/2013 19:00:08 - begin Parent Job
05/07/2013 19:00:08 - begin Unknown Type, Start Notify Script
05/07/2013 19:00:09 - Info RUNCMD(pid=10676) started           
05/07/2013 19:00:09 - Info RUNCMD(pid=10676) exiting with status: 0        
Status 0
05/07/2013 19:00:09 - end Unknown Type, Start Notify Script; elapsed time: 00:00:01
05/07/2013 19:00:09 - begin Unknown Type, Step By Condition
Status 0
05/07/2013 19:00:09 - end Unknown Type, Step By Condition; elapsed time: 00:00:00
05/07/2013 19:00:09 - begin Unknown Type, Read File List
Status 0
05/07/2013 19:00:09 - end Unknown Type, Read File List; elapsed time: 00:00:00
05/07/2013 19:00:09 - begin Unknown Type, Resolver Discovery
05/07/2013 19:00:09 - started process bpbrm (5904)
05/07/2013 19:00:11 - Info bpbrm(pid=5904) DAG1 is the host to restore to     
05/07/2013 19:00:11 - Info bpbrm(pid=5904) reading file list from client       
05/07/2013 19:00:17 - Info bpbrm(pid=5904) client_pid=9792           
05/07/2013 19:00:17 - Info bpbrm(pid=5904) from client DAG1: TRV - BPRESOLVER has executed on server (EXServer) 
05/07/2013 19:01:15 - Info bpresolver(pid=9792) done.  status: 0        
05/07/2013 19:01:20 - Info bpresolver(pid=9792) done. status: 0: the requested operation was successfully completed   
Status 0
05/07/2013 19:01:20 - end Unknown Type, Resolver Discovery; elapsed time: 00:01:11
05/07/2013 19:01:20 - begin Unknown Type, Persist Discovery
Status 0
05/07/2013 19:01:20 - end Unknown Type, Persist Discovery; elapsed time: 00:00:00
05/07/2013 19:01:20 - begin Unknown Type, Policy Execution Manager Preprocessed
Status 196
05/07/2013 19:01:26 - end Unknown Type, Policy Execution Manager Preprocessed; elapsed time: 00:00:06
05/07/2013 19:01:26 - begin Unknown Type, Stop On Error
Status 0
05/07/2013 19:01:26 - end Unknown Type, Stop On Error; elapsed time: 00:00:00
05/07/2013 19:01:26 - begin Unknown Type, End Notify Script
05/07/2013 19:01:26 - Info RUNCMD(pid=8896) started           
05/07/2013 19:01:27 - Info RUNCMD(pid=8896) exiting with status: 0        
Status 0
05/07/2013 19:01:27 - end Unknown Type, End Notify Script; elapsed time: 00:00:01
Status 196
05/07/2013 19:01:27 - end Parent Job; elapsed time: 00:01:19
client backup was not attempted because backup window closed(196)

06/07/2013 20:46:18 - Info nbjm(pid=3212) starting backup job (jobid=2486865) for client DAG1, policy Corp_Applications_EX_C, schedule Monthly 
06/07/2013 20:46:18 - Info nbjm(pid=3212) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=2486865, request id:{48273FDF-9092-4A66-B225-7697B1F414F7}) 
06/07/2013 20:46:18 - requesting resource DiskPoolGroup
06/07/2013 20:46:18 - requesting resource NBServer1-v.NBU_CLIENT.MAXJOBS.DAG1
06/07/2013 20:46:18 - requesting resource NBServer1-v.NBU_POLICY.MAXJOBS.Corp_Applications_EX_C
06/07/2013 20:46:18 - requesting resource EXCHANGE_RESOLVER.NBServer1-v.Corp_Applications_EX_C.DAG1
06/07/2013 20:46:18 - granted resource NBServer1-v.NBU_CLIENT.MAXJOBS.DAG1
06/07/2013 20:46:18 - granted resource NBServer1-v.NBU_POLICY.MAXJOBS.Corp_Applications_EX_C
06/07/2013 20:46:18 - granted resource EXCHANGE_RESOLVER.NBServer1-v.Corp_Applications_EX_C.DAG1
06/07/2013 20:46:18 - estimated 136895656 Kbytes needed
06/07/2013 20:46:18 - begin Parent Job
06/07/2013 20:46:18 - begin Unknown Type, Start Notify Script
06/07/2013 20:46:18 - Info RUNCMD(pid=5164) started           
06/07/2013 20:46:18 - Info RUNCMD(pid=5164) exiting with status: 0        
Status 0
06/07/2013 20:46:18 - end Unknown Type, Start Notify Script; elapsed time: 00:00:00
06/07/2013 20:46:18 - begin Unknown Type, Step By Condition
Status 0
06/07/2013 20:46:18 - end Unknown Type, Step By Condition; elapsed time: 00:00:00
06/07/2013 20:46:18 - begin Unknown Type, Read File List
Status 0
06/07/2013 20:46:18 - end Unknown Type, Read File List; elapsed time: 00:00:00
06/07/2013 20:46:18 - begin Unknown Type, Resolver Discovery
06/07/2013 20:46:19 - Info bpbrm(pid=2116) DAG1 is the host to restore to     
06/07/2013 20:46:19 - Info bpbrm(pid=2116) reading file list from client       
06/07/2013 20:46:19 - started process bpbrm (2116)
06/07/2013 20:46:25 - Info bpbrm(pid=2116) client_pid=14000           
06/07/2013 20:46:25 - Info bpbrm(pid=2116) from client DAG1: TRV - BPRESOLVER has executed on server (EXServer) 
06/07/2013 20:47:16 - Info bpresolver(pid=14000) done.  status: 0        
06/07/2013 20:47:22 - Info bpresolver(pid=14000) done. status: 0: the requested operation was successfully completed   
Status 0
06/07/2013 20:47:22 - end Unknown Type, Resolver Discovery; elapsed time: 00:01:04
06/07/2013 20:47:22 - begin Unknown Type, Persist Discovery
Status 0
06/07/2013 20:47:22 - end Unknown Type, Persist Discovery; elapsed time: 00:00:00
06/07/2013 20:47:22 - begin Unknown Type, Policy Execution Manager Preprocessed
Status 196
06/07/2013 20:47:22 - end Unknown Type, Policy Execution Manager Preprocessed; elapsed time: 00:00:00
06/07/2013 20:47:22 - begin Unknown Type, Stop On Error
Status 0
06/07/2013 20:47:22 - end Unknown Type, Stop On Error; elapsed time: 00:00:00
06/07/2013 20:47:22 - begin Unknown Type, End Notify Script
06/07/2013 20:47:22 - Info RUNCMD(pid=10624) started           
06/07/2013 20:47:22 - Info RUNCMD(pid=10624) exiting with status: 0        
Status 0
06/07/2013 20:47:22 - end Unknown Type, End Notify Script; elapsed time: 00:00:00
Status 196
06/07/2013 20:47:22 - end Parent Job; elapsed time: 00:01:04
client backup was not attempted because backup window closed(196)

This is the first and last attempts of this client for the last weekend. There were 12 failures in total. I have had a look at the Global Attributes on the master server and I can't see anything wrong there or with the storage units (as per related forum discussions).

Other backups have managed to complete (on different policies) to this disk pool and storage unit without any issues.

Can anyone offer any insight?

7.1.x and Earlier
Backup and Recovery
Error messages
NetBackup
Troubleshooting
Windows Server (2003-2008)
  • Mark_Solutions's avatar
    Mark_Solutions
    12 years ago

    Your weekly and monthly schedules are calendar based .. so do bear in mind that your window crossing midnight may not be valid as if it is set to run on a Friday then it only actually has from 7pm Friday to midnight to work - any window after that is not valid as saturday and sunday are different days as far a calendar scheduling goes

    Having said that your jobs seem to fail quicker than that which does suggest a possible schedule corruption

    So i would suggest trying the following:

    1. delete the current schedules

    2. create new schedules that are set to run on 1st Saturday for the Monthly and all other Saturdays for the Weekly backups

    3. Set the start window from 00:10 on Saturday to 23:50 on Saturday (no point in wrapping to ther days with calendar based scheduling unless you allow to run after rundday which just causes other issues)

    See if the jobs now work

11 Replies

Sort By
  • RamNagalla's avatar
    RamNagalla
    Moderator
    12 years ago

    hi,

    your backup job compleated/failed in less then 2 hours..

    1) what is the backup window in the policy Corp_Applications_EX_C schedule

    defined start time and end time in policy scheudle?

    the above one that you posted is parenet job, show us the detail status of the child job also..

  • Katiebee's avatar
    Katiebee
    Level 4
    12 years ago

    As I said, the window for this schedule on this policy is Friday 19:00 till Saturday 00:00 as shown below:

    backup window.GIF

    Part of the problem is that no child jobs are created, so I have nothing more to go on. I have checked the event veiwer on both the EX server with the dag and on the master NBU server and there is nothing that is at the time that these jobs are starting or finishing, or anything else which jumps out at me as being related - no errors & no warnings.

  • RamNagalla's avatar
    RamNagalla
    Moderator
    12 years ago

    06/07/2013 20:47:22 - begin Unknown Type, Policy Execution Manager Preprocessed
    Status 196

    are you using the same storage unit for both week/month end jobs and Daily jobs?

    how it is running when you run the manual backup of the  week/month end schedules  when there is no jobs running on storage units?

    show us the output of below commands

    bpplinfo <policyname> -U

    bpstulist -lable <storageunit name> -L

     

     

  • Katiebee's avatar
    Katiebee
    Level 4
    12 years ago

    The schedules for daily and weekly use the same storage group, the monthy does use the same physical storage group but is on a different slp. I will have to check when I can try a manual monthy backup. These are production servers and in heavy use, but I will let you know the outcomes of the manual attempts and commands as requested when I have them.

    Thanks,

    Katie

  • Katiebee's avatar
    Katiebee
    Level 4
    12 years ago

    bpplinfo corp_applications_ex_c -U returns

    Policy Type:            MS-Exchange-Server
    Active:                 yes
    Effective:              10/20/2008 16:08:11
    Follow NFS Mounts:      no
    Cross Mount Points:     no
    Client Compress:        no
    Collect TIR info:       no
    Policy Priority:        2
    Ext Security Info:      no
    File Restore Raw:       no
    Client Encrypt:         no
    Residence:              Corp_Applications_Daily_C
    Volume Pool:            NetBackup
    Server Group:           *ANY*
    Data Classification:            -
    Residence is storage lifecycle policy:  yes
    Granular Restore:       no
    Ignore Client Direct:   no
    Exchange Source passive db if available: yes
    Exchange 2010 Preferred Server: EXserver names Application Discovery:    no
    Discovery Lifetime:     0 seconds

    bpstulist -label CHMasterServerPool -L returns

    Label:                CHMasterServerPool
    Storage Unit Type:    Disk
    Media Subtype:        DiskPool (6)
    Host Connection:      NBservername
    Concurrent Jobs:      50
    On Demand Only:       yes
    Robot Type:           (not robotic)
    Max Fragment Size:    10240
    Max MPX:              1
    Block Sharing:        no
    File System Export:   no
    Ok On Root:           no
    Disk Pool:            CHMasterServerPool

    The daily backups completed as normal last night, I will see if I can run a manual weekly or monthly full later today, but does the above shed any light on why I should be getting a 196 error 90 seconds after the parent job starts with no child jobs created, all while within the backup window?

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    12 years ago

    Your weekly and monthly schedules are calendar based .. so do bear in mind that your window crossing midnight may not be valid as if it is set to run on a Friday then it only actually has from 7pm Friday to midnight to work - any window after that is not valid as saturday and sunday are different days as far a calendar scheduling goes

    Having said that your jobs seem to fail quicker than that which does suggest a possible schedule corruption

    So i would suggest trying the following:

    1. delete the current schedules

    2. create new schedules that are set to run on 1st Saturday for the Monthly and all other Saturdays for the Weekly backups

    3. Set the start window from 00:10 on Saturday to 23:50 on Saturday (no point in wrapping to ther days with calendar based scheduling unless you allow to run after rundday which just causes other issues)

    See if the jobs now work

  • Katiebee's avatar
    Katiebee
    Level 4
    12 years ago

    Hi thanks Mark and Nagalla,

    We use the recuring weekdays option, have run after runday, and then exclude the days that would otherwise cause the scheduels to clash with each other. I'm going to run a weekend job tonight so I can see what happens, if anything and I'll let you know the results.

    Thanks,

    Katie

  • Katiebee's avatar
    Katiebee
    Level 4
    12 years ago

    Hi again,

    After making no changes I manually ran the weekly backup schedule last night and this completed without any errors, all the child jobs started as they should, took an appropriate amount of time and all the databases and dag were captured. I'm concerned as the error doesn't seem to have much explanation but am happy that it's working...

    Thanks for your help anyway

    Katie

  • RamNagalla's avatar
    RamNagalla
    Moderator
    12 years ago

    so probably need to wait for this weekend schedule backup and see how it goes..

    if it got failed again lets look into the logs

    collet the job logs if it got failed

    vxlogview -p NB -X "jobid=XXXXX"    --> replace XXXXX with job id and post

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    12 years ago

    I would say that it was down to schedule corruption as the jobs failed within a minute of starting so couldn't possibly be outside of their backup window - I do still stand by my advice on the best way to use the calendar scheduling to make it work well and not need any exclude dates adding

    Glad it is solved for you anyway - dont forget to close off the thread if possible by marking the advice that helped the most as the solution to assist other with a similar issue in the future