Lots of 196

[e_win]

Can anyone help me with this.

NBU is 6.5 --> This is ancient and upgrade is still for approval

We have 94 policies, 100 unique clients, and 256 schedules because some policy has daily, weekly and monthly backups.

We are using 1 SL500 (6 drives), and some basic disks for daily_incremental.

During weekdays, all backup runs completely but EVERY weekend I encounter lots of 196 errors.

I tried to explain to them that the schedule backup window has past that's why its failing 196 errors.

my concept is that because there are so many backup schedules running during weekends and only using 6 drives, how can I maximize this?

I am now currently reviewing every policy.. Should I move some schedules during weekdays?

Regards

[Marianne]

Please give us more info before we try and give advice...

Are all 6 drives in use all of the time over weekends?

Do you have multiplexing and multistreaming enabled in STU properties and schedules?

Do you check Activity Monitor while backups are queued to see exact reason for backups being queued?

What is the throughput on your tape drives? Are you able to stream all 6 drives at reasonable speed? Hopefully not all 6 drives attached to single master/media server?

[e_win]

Yes all the drives are used over the weekends,

Some of the jobs took for days.

Multiplexing is enabled in STU, Max concurrent write drives is 6 and maximum streams per drive is 3. Does this mean, 18 jobs should be running simultaneously? On schedule attribute it is not enabled.

Jobs are queued while waiting for the resources or the drive is in use.

All the drives are attached to single master/media server and is running solaris 10.

[Marianne]

On schedule attribute it is not enabled.

This means that you will only have 6 backups active at any point in time.

Unless your backups are running close to 100 Mb/sec, you should enable MPX in schedules as well.

The lower value of MPX in STU properties and schedule will be applied.

You can also have a look at multiple jobs per client - ensure Max Jobs per Client in Master server Host Properties -> Global Attributes is greater than 1. (4 is a good average.)
Depending on Backup Selection, you can then select 'Allow Multiple data streams' in policy attributes.

All of this is explained in NBU Admin Guide I: http://www.symantec.com/docs/DOC5157

Have you confirmed that your master server have enough physical resources to stream 6 drives at +- 100 Mb/sec simultaneously?

PS: only make changes to policies when next Full is due. 

[Dyneshia]

One more thing, what is the exact version of NBU are you running ?

Back in 6.5 there was an issue with 196 and job spanning over midnight .  The fix was in 6.5.3

Fixed in 6.5.3:  (ET1300264) Queued backups with 24/7 windows fail at midnight reporting status code 196
 http://support.veritas.com/docs/305656

[e_win]

we are still looking for solutions, right now we're reviewing our policy, and yesterday we have a meeting  regarding issues on NBU.

NBU is 6.5.6

We have SL500 tape library with extension, and 6 drives are running simultaneosly.

bpconfig -U shows:

Job Retry Delay:              10 minutes
Max Simultaneous Jobs/Client: 99
Backup Tries:                 4 time(s) in 24 hour(s)
Keep Error/Debug Logs:        30 days
Max drives this master:       0
Keep TrueImageRecovery Info:  1 days
Compress Image DB Files:      (not enabled)
Media Mount Timeout:          0 minutes (unlimited)
Shared Media Mount Timeout:0 minutes (unlimited)
Display Reports:              24 hours ago
Preprocess Interval:          4 hours (default)
Maximum Backup Copies:        2
Image DB Cleanup Interval:    12 hours
Image DB Cleanup Wait Time:   60 minutes
Policy Update Interval:       1440 minutes

Have you confirmed that your master server have enough physical resources to stream 6 drives at +- 100 Mb/sec simultaneously? ---- How can I now if the master server can handle 6 drives???

About multiplexing... STU is set to 3, do I need to enable multiplexing to ALL policies on schedule attribute?

[Marianne]

How can I now if the master server can handle 6 drives??? 

Lots of information available, e.g. 
Planning and Performance Guide  http://www.symantec.com/docs/TECH62317 , 
http://www.symantec.com/docs/HOWTO56078 
https://www-secure.symantec.com/connect/sites/default/files/b-whitepaper_nbu_architecture_overview_1...

I have honestly not seen any backup server that can stream 6 drives simultaneously.

The backup server's network card is normally the bottleneck. If the server only has one 1 Gb network card, it can at most receive data from clients at 100 Mb/sec. This will stream one LTO drive.

What kind of throughput are you seeing in Activity Monitor when all 6 drives are active?

do I need to enable multiplexing to ALL policies on schedule attribute? 

MPX setting of 1 in schedule will override MPX setting in STU. Only one backup stream will run to tape. 
So, my answer is yes, enable multiplexing in ALL schedules.

[Dyneshia]

e_win - IT seems this is deinfalty a system (devices vs jobs) resouce issue.   Have you tried to spread out your jobs ? The nbpemreq  is a great command to use to see what is scheduled to run.  See what you can move to another day and test.

[e_win]

were transferring some jobs to other days and hours....i have a question, Is there a command that shows these are my simultaneous jobs running for the given time and date. I mean, for example given the day (Monday) and the time (12 am to 12 pm) -- running jobs are clientA, clientB, clientC,

[Dyneshia]

nbpemreq -predict -date <mm/dd/yyyy HH:MM:SS>

Exmaple :

# ./nbpemreq -predict -date 9/15/2005 16:00:00

Predicted worklist for Thu Sep 15 16:00:00 2005

rl client     policy     schedule   type

-- ---------- ---------- ---------- -----

01 starfury   test-dlt2  FULL       0    

Please see http://www.symantec.com/docs/TECH43573

ttp://www.symantec.com/docs/HOWTO43728

[Marianne]

Is there a command that shows these are my simultaneous jobs running for the given time and date. I mean, for example given the day (Monday) and the time (12 am to 12 pm)

You cannot specify a start and enddate with nbpemreq.

nbpemreq can only predict between 'now' and 'date' specified.

If your status 196 is solved, please select a post that pointed you in the right direction and 'Mark as solution'.

Thanks!

[e_win]

i will, nbpemreq command is good to check future jobs