Status Code 196 - Backup window no longer open

[BackupSysAdmin]

Hello,

I am a newbie to the world on Netbackup.  I have restored exchange mailboxes using Netbackup but am now on side of maintaining backups to tape library. We are currently using Netbackup  version 7.0.1.  I am trying to resolve all the partially successfull and failed backups in our environment.  I am positing the logs for a partially successfull differential Incremental Backup and am looking for advise on how to resolve this issue.  The status code is 196. Should I just adjust the backup window or is there another issue that I may need to address.  Your assistance would be greatly appreciated:

4/17/2012 6:00:00 PM - requesting resource Any
4/17/2012 6:00:00 PM - requesting resource nbumastersvr.NBU_CLIENT.MAXJOBS.clientXXXX
4/17/2012 6:00:00 PM - requesting resource nbumastersvr.NBU_POLICY.MAXJOBS.ORG_Exchange_Mailboxes
4/17/2012 6:00:00 PM - awaiting resource Any - No drives are available
4/17/2012 7:13:39 PM - awaiting resource Any Reason: Drives are in use, Media Server: nbumastersvr,
  Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
  Volume Pool: Exchange, Storage Unit: nbumastersvr-hcart-robot-tld-0, Drive Scan Host: N/A
 
4/17/2012 7:13:39 PM - awaiting resource Any - No drives are available
4/17/2012 10:15:18 PM - awaiting resource Any Reason: Drives are in use, Media Server: nbumastersvr,
  Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
  Volume Pool: Exchange, Storage Unit: nbumastersvr-hcart-robot-tld-0, Drive Scan Host: N/A
 
4/17/2012 10:15:18 PM - awaiting resource Any - No drives are available
4/17/2012 10:17:31 PM - awaiting resource Any Reason: Drives are in use, Media Server: nbumastersvr,
  Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
  Volume Pool: Exchange, Storage Unit: nbumastersvr-hcart-robot-tld-0, Drive Scan Host: N/A
 
4/17/2012 10:17:31 PM - awaiting resource Any - No drives are available
4/17/2012 10:18:13 PM - awaiting resource Any Reason: Drives are in use, Media Server: nbumastersvr,
  Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
  Volume Pool: Exchange, Storage Unit: nbumastersvr-hcart-robot-tld-0, Drive Scan Host: N/A
 
4/17/2012 10:18:13 PM - awaiting resource Any - No drives are available
4/17/2012 10:19:17 PM - awaiting resource Any Reason: Drives are in use, Media Server: nbumastersvr,
  Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
  Volume Pool: Exchange, Storage Unit: nbumastersvr-hcart-robot-tld-0, Drive Scan Host: N/A
 
4/17/2012 10:19:17 PM - awaiting resource Any - No drives are available
4/17/2012 10:19:53 PM - awaiting resource Any Reason: Drives are in use, Media Server: nbumastersvr,
  Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
  Volume Pool: Exchange, Storage Unit: nbumastersvr-hcart-robot-tld-0, Drive Scan Host: N/A
 
4/17/2012 10:26:58 PM - awaiting resource Any - No drives are available
client backup was not attempted because backup window closed(196)

[revarooo]

Were the drives actually in use? If so widen the window, if not check nbrbutil -dump and see if a rogue allocation is holding the drive/s

[Marianne]

Best way to troubleshoot this kind of issue is to be in front of Admin Console to see what is actually happening.

As revaroo mentioned - were all drives really in use at this time?
How many jobs per drive?
What kind of throughput was seen per drive?
If less than documented average throughput - have you tried to increase MPX in STU properties as well as in schedules?

Did this job require a different pool and/or retention level that would prevent it from being multiplexed?

IMHO - there is is nothing that beats being THERE (even if via VPN remote Java/Admin Console) when the majority of jobs are started to SEE the effect on resource allocation.

[BackupSysAdmin]

This is what i received after checking nbrbutil -dump:

D:\Program Files\NetBackup\bin\admincmd>nbrbutil -dump

Allocation Requests
(AllocationRequestSeq )

Allocations
(AllocationSeq )

MDS allocations in EMM:

[Marianne]

Nothing is allocated right now... At least we know there are no orphaned or stuck device or media allocations...

You need to compare the same output with Device Monitor and Activity Monitor when you see queued jobs.

[revarooo]

BackupSysAdmin, 

It's not rogue allocations holding the drives. The drives were in use (or unavailable).

Can you confirm if this policy has worked before or is it new?

[BackupSysAdmin]

Revaroo.....yes that policy has worked in the past.  Not new.  I got the status code 196 on 15 Apr, but it looks like the same policy (incremental backup) ran successfully on the 16th, and 17th. I am going to review all of the jobs is Admin Console.  Is there a Best Practice document that I can compare with what was setup before I was given this position?

[revarooo · Accepted solution]

I would recommend you check the policy and see what storage unit it is using. Check the media server associated to that storage unit and check the drives that are associated to the storage unit type to ensure none are down and you have full resources available in that respect.

Check other jobs that are running that use this storage unit at the time this policy goes active to see how long they run and if they are backing up quickly (kb/s) (as suggested by Marianne before). It could be that the storage unit / drives is just too busy because you have too many jobs running.

[BackupSysAdmin]

Can you tell me if I need to replace expired and expiring media. 

*******************************************************************************
MEDIA SUMMARY FOR NDMP SERVER xxxxxxxx ON 4/19/2012 2:38:39 PM
*******************************************************************************
ACTIVE FULL SUSPENDED FROZEN IMPORTED WORM BE_MEDIA ENCRYPTED
18 80 0  8 0  0 0  0
Number of NON-ACTIVE media that:
3 - are currently EXPIRED
BR5379 expired 12/20/2011 00:00 (FROZEN)
ECL932 expired 12/26/2011 10:55 (FROZEN)
ECL947 expired 02/08/2012 20:48 (FROZEN)
7 - will expire within 1 week
ECL973 expires 04/20/2012 16:00
ECL948 expires 04/20/2012 20:17
BR5709 expires 04/21/2012 21:54
BR5361 expires 04/24/2012 18:00
ECL922 expires 04/24/2012 00:00
ECL916 expires 04/23/2012 11:39
BR5664 expires 04/26/2012 04:05
6 - will expire between 1 and 2 weeks
BR5373 expires 04/27/2012 16:00
BR5674 expires 05/02/2012 02:35 (FROZEN)
ECL953 expires 04/29/2012 01:53
ECL936 expires 04/29/2012 01:53
ECL919 expires 04/29/2012 00:00
BR5767 expires 05/02/2012 02:35
5 - will expire between 2 and 3 weeks
BR5706 expires 05/05/2012 18:00
BR5390 expires 05/09/2012 06:26
ECL939 expires 05/09/2012 00:55
BR5760 expires 05/05/2012 21:11
ECL935 expires 05/09/2012 06:26
6 - will expire between 3 weeks and 1 month
ECL927 expires 05/11/2012 16:00
BR5762 expires 05/12/2012 20:43
BR5365 expires 05/16/2012 07:00
ECL959 expires 05/16/2012 01:18
ECL913 expires 05/15/2012 00:00
BR5677 expires 05/18/2012 16:00
3 - will expire between 1 month and 2 months
ECL931 expires 05/27/2012 00:00
ECL911 expires 05/30/2012 16:00
ECL937 expires 05/28/2012 04:12
1 - will expire between 2 months and 3 months
BR5775 expires 06/29/2012 09:07
57 - will expire in greater than three months
BR5713 expires 07/25/2012 03:11
BR5660 expires 07/25/2012 16:00
BR5383 expires 07/27/2012 21:45
ECL925 expires 07/27/2012 14:03
BR5366 expires 08/31/2012 13:55
ECL974 expires 08/26/2012 00:00
ECL971 expires 08/26/2012 00:00
ECL967 expires 08/29/2012 16:00
ECL966 expires 08/27/2012 04:43
ECL955 expires 08/31/2012 08:24
ECL940 expires 08/15/2012 12:21
ECL938 expires 08/15/2012 10:12
BR5704 expires 07/30/2012 17:16
BR5391 expires 07/29/2012 00:00
BR5387 expires 08/30/2012 21:46
BR5386 expires 07/30/2012 04:24
BR5368 expires 09/23/2012 03:38
BR5371 expires 02/25/2013 04:14
ECL970 expires 10/24/2012 16:00
ECL969 expires 01/24/2013 21:58
ECL965 expires 01/27/2013 00:00
ECL960 expires 10/27/2012 06:21
ECL952 expires 09/24/2012 22:47
ECL945 expires 11/25/2012 01:35 (FROZEN)
ECL944 expires 10/25/2012 18:23
ECL943 expires 11/25/2012 00:53 (FROZEN)
ECL934 expires 09/24/2012 03:08
ECL929 expires 11/28/2012 09:50
ECL924 expires 09/28/2012 08:00
ECL921 expires 02/24/2013 00:00
ECL917 expires 01/28/2013 04:10
ECL914 expires 09/23/2012 12:09
ECL910 expires 01/28/2013 10:44
BR5778 expires 02/25/2013 00:50
BR5777 expires 11/26/2012 16:24
BR5776 expires 12/24/2012 08:57
BR5771 expires 11/25/2012 00:53
BR5769 expires 10/28/2012 00:00
BR5765 expires 09/24/2012 09:56
BR5717 expires 10/24/2012 10:09 (FROZEN)
BR5710 expires 12/30/2012 00:08
BR5678 expires 11/30/2012 18:26
BR5673 expires 10/29/2012 00:47
BR5671 expires 12/24/2012 01:30
BR5663 expires 10/29/2012 04:23
BR5399 expires 01/30/2013 16:00
BR5395 expires 11/30/2012 08:00
BR5382 expires 01/28/2013 10:44
BR5378 expires 12/24/2012 08:57
BR5374 expires 09/26/2012 16:01
BR5385 expires 03/27/2013 16:00
ECL972 expires 02/27/2013 16:00
ECL930 expires 03/25/2013 04:27
BR5700 expires 02/25/2013 04:14
BR5661 expires 03/25/2013 08:54
ECL933 expires 03/27/2013 16:00 (FROZEN)
ECL968 expires 03/31/2013 10:19
Number of ACTIVE media that, as of now:
5 - will expire between 1 and 2 weeks
BR5770 expires 05/02/2012 18:57
BR5363 expires 05/03/2012 03:38
BR5362 expires 05/03/2012 06:04
BR5670 expires 05/03/2012 05:56
BR5377 expires 05/03/2012 05:57
7 - will expire between 3 weeks and 1 month
ECL928 expires 05/15/2012 18:00
BR5384 expires 05/15/2012 18:00
BR5667 expires 05/18/2012 16:00
BR5675 expires 05/16/2012 07:00
BR5715 expires 05/19/2012 20:00
ECL923 expires 05/18/2012 22:21
BR5761 expires 05/20/2012 00:16
1 - will expire between 1 month and 2 months
ECL954 expires 05/28/2012 04:26
5 - will expire in greater than three months
BR5668 expires 08/15/2012 10:12
BR5397 expires 03/29/2013 21:04
BR5370 expires 04/13/2013 01:03
BR5768 expires 03/31/2013 02:26
ECL949 expires INFINITY
Summary by retention level of ALL media
Level   # Media        Megabytes
1         7          4335379.9
3        28         21763063.9
8        70         52526231.7
9         1            38934.1

[Marianne]

@BackupSysAdmin:

ONE issue per discussion, please... I have moved your 'media summary' post to a new discussion as this does not seem to be related to status 196....

[Jeff_Foglietta]

Take a look at the state of your drives. Could be possible you have some drives in a down state unavailable for backups and eating into your backup window.

tpconfig -dl