Possibly Quick NetBackup Question - Error 196

[Jim_Haywood]

Hi there!

I'm after chasing a 'quick kill' on the backup situation we've got here in work. I've been tasked with getting the netbackup solution we've got ticking along nicely. At the moment, there are three policies setup. One for each server. There are other servers that need backing up, but I don't think that adding more policies will help the situation and don't really want to approach the other servers until these two policies are working.

Now, looking at the activity monitor, the three jobs failed with status code 196 which from my reading means that the backup window closed before the job had a chance to start. All three policies failed with this code, so no backups took place in the backup window at all. Which incidently is between midnight and 6AM. Looks like all three jobs just decided to wait until 6 then fail without doing a thing.

That was the night before last's jobs. Last night's jobs are still queued even though the backup window is closed. If it was going to fail, I would have thought that it would have quit with the status code 196 along with the previous days backup. But no, it's just hanging around there.

I'm suspecting that this is a pretty easy one to fix. However, I know hardly anything about netbackup, so I'm struggling to come up with the goods as to where to look to troubleshoot or/and most importantly, how to resolve!

Any help would be greatly received!

Jimbo

[h_m]

I take it other jobs are running at the same time? Are devices free when these backups kick off?

[Jim_Haywood]

Well, the other jobs are scheduled to run at the same time, but they ofcourse are not running when they should!

Netbackup is installed on the fileserver and to an HP ultrium tape drive that has not other uses bar to serve this NBU install. So if NBU isn't using it, it should be free!

[h_m]

You really need to check the schedules for all of your jobs, and look at what the start times and end times are against what the frequency for the schedule is (if frequency based).

Handy hint - if you manually start a backup it can affect the tiem at which the scheduled backup can start.

[Jim_Haywood]

Hmm... Well, looking at the schedules, they are all a bit of a mess so you could well be right there. Seeing as this is pretty much a not working install at the moment, would you recommend clearing down all the schedules and policies and starting again?

Due to business requierments, all backups need to happen between midnight and 6AM - The closing hours for the building.

I'm assuming best practise here would be to have a policy for each server, for example, a filesever policy, DC policy and exchange policy, and to have all of those policies kicked off at midnight by ONE schedule...?

Thanks for your help by the way!

[h_m]

You need a policy per backup type - or backup selection type. You can have many hosts in the same policy if they are going to backup say c:\ - for Windows etc. It really is up to you whether you have a policy per machine or policies covering multiple machines.

I would recommend: - starting all backups at 00:00 and finishing at 06:00 and make the frequency just a little bit longer, say 7 hours. Or look at calendar schedules, again with the syart time at 00:00 and end time at 06:00. this will eliminate frequency issues.

Then look to see if multiplexing is set on all schedules, and also within the storage unit for the storage unit you have configured within each policy (which I assume will be the same).

[Jim_Haywood]

Ok, what I've done for the moment is to set up two policies - One for the file server, and one for the exchange.

They each have thier own schedule that runs from 00:00 to 6:00. Each schedule is a full backup with a 7 hour frequency and a retention of one month.

The tape drive is not an auto changer, we have to change the tape every day. Backups are on weekdays only, and are on a monthy rotation. Therefore 5 tapes a week, for four weeks, equals 20 tapes. Does this all sound about right to you? Is there anything else that I should be checking? I want the backups to work like a charm tonight!!

Multiplexing is set to 1 on each one of the schedules, but I couldn't see anything about multiplexing when looking for storage units...

All good? Any further ideas?

[Chad_Wansing]

One thing I've run into that seems to be a fairly common problem is that people will set their schedules to "1day" on the frequency for a time based schedule. The problem with that is that when a job runs, it takes some amount of time. The job will then not try to run again until the time that it ended the day before. So if you've got a job that runs for 2 hours and an 8 hour window, you're going to work yourself right out of your window in less than a week.

What I suggest is (if you need to set a frequency-based schedule instead of a calendar-based schedule....which you need to do if you're going to let your jobs run across the midnight boundary or else you get yourself into another situation...) to set your policies to do backups every 16 hours, but determine when they actually start through the start window. That way, your job runs for two hours, when the next window rolls around it looks at the policy, says "oh, that hasn't run in 22 hours, let me kick off that backup" and runs like it's suppossed to. This was one of the big issues at the company I just took over the backups for.

[Jim_Haywood]

Riiiiiight... That makes sense - And would tie in with my experiences noted in the first post too...

So, start window is set to midnight, and frequency is set to 7 hours. (One hour over the backup window) And that should probably sort it out!

I'll monitor this thread for the rest of the day incase anyone else has some good ideas, but I'll let you all know how it worked tomorrow and close off the question.

Y'all have been wonderful! (So far - It's not confirmed working yet...!)

[Lance_Hoskins]

Jim,

Can you give us some more details on your clients here? How many machines are you trying to back up and how long do they currently take? What's the network like between the clients and the media servers? What type of a backup device are you trying to backup to?

I don't know for fact this this is the problem, but if you have multiplexing set to 1 for all of the schedules and you have just a single tape drive, then only one client gets backed up at a time. So for example, if you have 5 clients and the first two take 3.5 hours each to back up (one after another), by 7AM when the next 3 clients would even get a chance to start your scheduled start window is closed--resulting in an error 196.

Then comes the idea of what the clients are that you're backing up. Lets say you have windows clients with a C drive for the OS, a D drive with good amounts of data and an E drive with good amounts of data, if you think the network and your tape device can handle it you'll want to turn on "allow multiple data streams" on this policy so that your client can initiate a job for C, D and E all at the same time (as opposed to them backing up one after the other which is pretty inefficient).

Get me some more details about your situation and we'll see what we can fix you up with. Sounds to me like multiplexing (with the same retentions of course) may be of benefit to you, but multiple data streams could also help you out--especially if you only have a 6 hour window to get all of your backups done.

Lance

[Chad_Wansing]

Jim, I would give yourself a bit more of a buffer, but it's really up to you. If you're giving your backups a 6 hour window, also remember that that is just their START window. If they take 10 hours to run, as long as they get started before 6 in your case, they'll run for most of the day. Personally, I'm a fan of the calendar-based scheduling and then controlling when things kick off by the start windows.....but again, be careful of that if your backups straddle midnight, because it confuses the calendar-based stuff and your backups will run kind of crazy, possibly skipping days.

I also definitely agree about the multiplexing. Depending on your particular situation, you may or may not want to enable multiple data streams from the host (you don't want to saturate the connection on the HOST side, just the master server side), but I would definitely run multiplexing.

As an example, where I am now, they were not running multiplexing before I got here and they could barely keep their backups inside of a 12 hour window. After I got some rough multiplexing and a little bit of policy tuning done, the vast majority of our backups ran between 4 and 10 (the others couldn't start till after midnight anyways). Multiplexing really can help that much. Additionally, we just got in a new fibre-channel attatched i500, I disabled multiple data streams but kicked my mulitiplexing up to about 20 jobs per drive (only two drives currently given to the master server...just got our SSO licenses last week and will be doing that as soon as the zoning is done), and last night I was averaging 53% utilization of the gigabit connection on my master server without any noticible effects on the client end. Now THAT is good stuff!

[Jim_Haywood]

Lance,

At the moment, there are two policies configured - One to backup a load of data located on the same server as netbackup, and the other to backup Exchange over a network.

Back when it was working, I've been told as an estimate that the two policies would finish in about 4 - 5 hours. Though I personally haven't witnessed it working!

Connection between the exchange server and the netbackup box will be a standard 100Mbps Ethernet link. Of course the other files backed up are local to the box.

The tape drive is a HP LTO Ultrium SCSI single loader. Takes one LTO (800GB Compressed) per day.

From the setup I'm looking at here, the exchange server is backed up in a flat file style with scripts to stop and start the Exchange services before and after the backups. This historically is something to do with the exchange agent not working properly (Exchange = 5.5 NT box)

So, exchsvr folders on C: D: and E: on one server, and a collection of folders all located on the same 'data' drive on the netbackup server.

Chad,
I haven't looked into calendar based scheduling purely because that's not the way it was set up when I got here. Is it generally excepted to be the better way to implement the solution I'm looking for then?

there is a new separate backup LAN on the way which will be gigabit, but that's not quite yet and I need to get it working in the mean time!


Suppose that leaves two questions,

A, should I switch over to calendar based scheduling based on what I've noted above?

B, Should I implement multiplexing, and if so, to what degree?

Again, many thanks for your help people!

[Lance_Hoskins]

Jim,

The frequency is totally up to you, but for your situation I don't think calendar based is needed. Try if you'd like, but like stated, if you don't know the "tricks" involved in calendar based frequencies, you could be tricked!

So on your master server with the "load" of data is all on one partition huh? How many top-level folders do you have on this machine? It's possible that you could mix them up a little bit with multiple data streams from the top leve to throw more data at the tape drive instead of doing it in one contiguous stream.

What kind of throughput are you getting out of it right now when backing it up and same goes for the Exchange flat file backup--what kind of throughput are you getting from it? I'm assuming your tape drive if LTO1, so maybe start out with a multiplexing of 3-4 and see how that works for your throughput.

Keep in mind that multiplexing will make restores a bit slower, but if the high priority is to get the backups done in a window and it's not currently working, this is a downside that you might just have to live with.

Lastly, with regards to your Exhange server and the flat files backups (ugh, the Exchange agent is very nice and very fast), what is on D and E? Is one partition logs and one data or are there multiple mailbox stores housed on each one with each volume containing flat file backups (***sigh*** again)?

Hope this helps. The more info you can feed us the better we'll be able to help. From what I'm reading though, I don't think frequency is your problem, but rather the limited amount of resources you have availble and the fact that NBU isn't configured yet to work within those limitations.

Lance

P.S. One more thing... If you have one of the clients (either exchange or the "load" of data) that's more important to backup than the other, make sure to start it first. Maybe create a 15-30 minute gap between starting times on their schedules so that you know that your highest priority backup starts first.

[Chad_Wansing]

(you can also adjust the priority a policy receives...since it sounds likeyour breaking this into seperate policies anyways.... by double-clicking the policy, and upping the "Job Priority" field about 2/3 of the way down on the left..... :) )

[Jim_Haywood]

Well, not being versed in calendar trickery, I'll stay away from that option!

From what I understand of frequency, which is admittedly not a lot, 7 hours should suit my purpose. If anyone has some advice that counters my knowledge, I'm more than willing to heed the advice!

Apologies for vagueness. The 'load' of data is about 200GB and all housed in four top level folders on the one partition. So it sounds like they could be mixed up a little from multiple data streams.

There seems to be little in the way of problem logs etc going on with this installation... Maybe been cleared quite recently. From what I can see, the exchange flatfile backup finishes in roughly 3 and a half hours and the 200gb file backup finishes in about 2 hours. Sorry for the lack of info.

I'll try a multiplexing of 3 to see what happens. Though the 'Media Multiplexing' option seems to be grayed out on my schedules... How do I go about changing? Slower restores I don't think will be an issue for us in this case.

The Exchange agent is indeedy the end goal. But the server is in a bad way and needs replacing. I don't think it's to advisable to go playing around with doing things such as reboots it too much until it's replaced. I'll look into the better solution then!

On the exchange server, C: contains the server install, D: contains the data, and E: is logs.

As the exchange server is in a bad way, it'[s more essential that it gets backed up. I'll get that schedule put half an hour earlier!

[Lance_Hoskins]

You should have no problems getting done in the 6 hours given--especially with multiplexing turned on. Make sure you aren't running manual backups or anything else during the day and let the policy and schedules take care of themselves when possible.

As for enabling multiplexing, that's an option in the storage unit, so make sure you have that option checked and make the maximum 3 or greater so that you can set it to that in the schedule.

With regards to your data partition on the master server, you could change the seletions to go in multiple streams rather than doing "all local drives". Maybe try something like:

Shadow Copy Components:\
C:\
D:\* (I think this will work, but in case not you can also just point to the 4 top-level folders)

Then enable multiple data streams and let it roll! This way instead of pulling data off of there contiguously, you can get rolling on 4 different streams at once on your data partition!

Let me know how any/all of this works for you and we can tweak as we go. Where abouts do you live (or in which time region)? I'm in USA Central Standard Time.

Lance

[Jim_Haywood]

Morning!

Last night's backups then. The 200GB worth of files backed up with no issues what so ever - Status exit code was 0. However, the backup did take 6 and a half hours to finish, which seems to be the reason why the Exchange flat file backup failed with status 196. Speed seems to be the issue, is multiplexing the answer?!

The fileserver policy does indeed just have the four folders specified not ALL_LOCAL_DRIVES, so that should be good for the multiplexing.

That was last night, which is a shame as I'm very eager to get an exchange backup done! Today, so far I've ticked 'Enable Multiplexing' in 'Storage Units' and set the 'Maximum streams per drive:' to 4. Do I need to do anything else to get these two policies to multiplex properly?

What I'll do as well, seeing as the Exchange server needs a good backup, is up the priority of that policy to make sure it goes through tonight!

I'm GMT over here... England!

[h_m]

So you're nearly there.Exchange flat file backup? Will you be able to recover from this in the event of a restore? (just wondering).

Multiplexing within a schedule allows the schedule to multiplex at that amount. Therefore the storage unit has to be at least equal to the schedule. But if you want to run 2 schedules in parallel then the storage unit has to be able to allow both schedules to run the backup streams. So work out how many streams each schedule will create, and then work out how many schedules will run at any one time, and configure the storage unit multiplexing for this value. Max is up to 32. You will not break anything by increasing your stu multiplexing in your environment as you only have 1 drive.

Also ensure multistreaming is enabled per policy, and also that the max jobs per cilent (master server settings) is set to match the number of streams which will run from each host.

If you are running flat file backups of excahnge, and can choose separate folders, this will allow the backups to run as multiple streams. Give it a try.

[Jim_Haywood]

So, to get multiplexing working:

My Exchange policy should make three streams, and the Fileserver policy should be four. So I should be setting the Storage Unit -> Maximum Streams per drive to seven. Which I've done. Both schedules will be running at the same time...

In policy -> change... -> I've ticked the box 'Allow multiple data streams'

When looking at the 'Change Schedule' window, is 'Media multiplexing' the option I'm after too? I've changed the field to reflect the number of streams that I can run concurrently. (Either 4 or 3 as mentioned above depending on policy)

I can't find the max jobs per client master server setting though... Any pointers?

As for the ability to restore from an Exchange flat file backup, the Exchange gurus over here suggest that it's a suitable solution until the server gets replaced and we get that nice agent on the go.

Does the above sound good to one and all?

[h_m]

Sounds right, nothing more to do except for the max jobs per client, which is in:-

host properties\master server\
Right click on the master server and choose properties, then the max jobs per client is on the first tab.

If your exchange server is creaky make sure running 3 backup jobs on that server will not kill it, as each job is going to start its own bpbkar32 process with other processes...