NDMP backup failing with error code: 196

[Narrybhatia]

Hi,

I have NDMP local backup configuration with netbackup in my environement. Last night after creating DUMP[Pass I] for 7 hours, it failed with error code 196 ie Backup window closed.

Although, I have chekced the policy and it stilll has 4 hours of backup window. I am wondering, how I am getting this error even still I have plenty of backup window. Can anyone please suggest?

Here are my findings:

1. No issue with backup window configured in policy.

2. I have checked the logs of nbjm where I have got the error "BPBRM is not running yet or is shutting down or connection is broken, will close the connection, state = 3" ()Attaching file for the refrence).

3. Vx logs do not have naything to highlight.

[Marianne]

Please show us all text in Job Details. This will hopefully tell us which other logs are needed. And please show us what you see as far as policy config is concerned? bppllist (policy-name) -U

[sdo]

1) Have any of the run window timings been updated recently?

2) What version of NetBackup i sthe master server running?

.

3) Have you seen this:

In very rare circumstances, nbpem may use an old version of a updated policy, which may lead to unexpected results.​

http://www.veritas.com/docs/000097843

...which says:

"This issue affects all NetBackup versions prior to 7.7 GA, including NetBackup Appliance versions prior to 2.7.1."

...but unfortunately does not provide any detail on how to determine whether the symptoms of the issue are occurring or have occured.

[Narrybhatia]

Hi Marrine,

Here is the Detailed status of the job:

24/03/2016 18:00:10 - Info nbjm(pid=12776) starting backup job (jobid=2125201) for client WOKFAS0422, policy NDMP_WOKFAS0422_COMPLIANCE1, schedule Differential-Inc  
24/03/2016 18:00:10 - Info nbjm(pid=12776) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2125201, request id:{F3041B53-135D-4BC5-8DC8-E39D698EA77A})  
24/03/2016 18:00:10 - requesting resource WOKFAS0442_NDMP_STUG
24/03/2016 18:00:10 - requesting resource ukarpwwokpbr01.NBU_CLIENT.MAXJOBS.WOKFAS0422
24/03/2016 18:00:10 - requesting resource ukarpwwokpbr01.NBU_POLICY.MAXJOBS.NDMP_WOKFAS0422_COMPLIANCE1
24/03/2016 18:00:18 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 19:27:44 - awaiting resource WOKFAS0442_NDMP_STUG - No drives are available
24/03/2016 19:29:35 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A

WOKFAS0422 is NDMP filer.

ukarpwwokpbr02 is Media server.

ukarpwwokpbr01 is master server.

I am attaching the restults of bppllist.

Policy is configured for both Monthly full as well as diffrential. I am concerned about diffrential backup. I have multiple seperate streams running for this policy.
    
24/03/2016 21:00:46 - awaiting resource WOKFAS0442_NDMP_STUG - No drives are available
24/03/2016 21:03:58 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 21:07:01 - awaiting resource WOKFAS0442_NDMP_STUG - No drives are available
24/03/2016 21:08:58 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 21:20:32 - awaiting resource WOKFAS0442_NDMP_STUG - No drives are available
24/03/2016 21:21:29 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 21:30:32 - awaiting resource WOKFAS0442_NDMP_STUG - No drives are available
24/03/2016 21:35:29 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 21:38:29 - awaiting resource WOKFAS0442_NDMP_STUG - No drives are available
24/03/2016 21:40:25 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 22:11:02 - awaiting resource WOKFAS0442_NDMP_STUG - Maximum job count has been reached for the storage unit
24/03/2016 22:11:21 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 23:00:04 - awaiting resource WOKFAS0442_NDMP_STUG - No drives are available
24/03/2016 23:02:02 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
24/03/2016 23:10:59 - awaiting resource WOKFAS0442_NDMP_STUG - Maximum job count has been reached for the storage unit
24/03/2016 23:11:08 - awaiting resource WOKFAS0442_NDMP_STUG Reason: Drives are in use, Media Server: ukarpwwokpbr02, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Woking_NDMP, Storage Unit: ukarpwwokpbr02-hcart2-robot-tld-0-WOKFAS0422, Drive Scan Host: N/A
    
client backup was not attempted because backup window closed(196)
25/03/2016 04:48:42 - job 2125201 was restarted as job 2127140

[Narrybhatia]

Hi Sdo,

My all other NDMP backups are running in this window only. Also, I have not changed any timings since last 4 months and it was working fine earlier.

Master server is of 7.6.0.4 version.

[Marianne]

Nothing wrong with your config. Other than clearly a problem with resources. There are 2 possible reasons for this policy having an issue: 1) NBU bug (documented for a previous version) 2) Corruption that creeped into this policy. For no.1, you need to log a Support call with Veritas and be prepared to upload lots of max level logs. For no. 2, you need to recreate the policy. Do not copy existing policy. And lastly look at staggering schedules to make better use of resources.