Possibly Quick NetBackup Question - Error 196

[Jim_Haywood]

Hi there!

I'm after chasing a 'quick kill' on the backup situation we've got here in work. I've been tasked with getting the netbackup solution we've got ticking along nicely. At the moment, there are three policies setup. One for each server. There are other servers that need backing up, but I don't think that adding more policies will help the situation and don't really want to approach the other servers until these two policies are working.

Now, looking at the activity monitor, the three jobs failed with status code 196 which from my reading means that the backup window closed before the job had a chance to start. All three policies failed with this code, so no backups took place in the backup window at all. Which incidently is between midnight and 6AM. Looks like all three jobs just decided to wait until 6 then fail without doing a thing.

That was the night before last's jobs. Last night's jobs are still queued even though the backup window is closed. If it was going to fail, I would have thought that it would have quit with the status code 196 along with the previous days backup. But no, it's just hanging around there.

I'm suspecting that this is a pretty easy one to fix. However, I know hardly anything about netbackup, so I'm struggling to come up with the goods as to where to look to troubleshoot or/and most importantly, how to resolve!

Any help would be greatly received!

Jimbo

[Jim_Haywood]

Yep... I was looking in the wrong place for the Master Server setting! It's now set to 7.

I reckon the exchange server will stand up to the extra hammering that the new processes will force upon it. Test will be tonight I guess!

So, I've set everything as outlined in your recommendations, I suppose the only thing left is to see if the whole lot works... I'll let you know tomorrow - many many many thanks for your help!

Again, I'll keep monitoring this topic incase Lance has anything to add once his time zone swings around ;)

[Lance_Hoskins]

Ok, it's morning in the USA now! :)

Only thing I'd maybe add is that you probably don't need your multiplexing up to 7 unless you think your clients can feed the data that fast, but even so I don't think your LTO1 library can keep up with that pace. I'd go with a mulitplexing of 3-4 to start and see where that gets you. If you enable multiple data streams on the Exchange policy, you should have four streams total (if your selection is "All Local Drives", you should have System State, C, D, and E). As for your file server (i.e. master server), multiple data streams on that will only give you 3 streams if you have the selection list set up as "All Local Drives" (System State, C, D).

What I'd do for your file server (especially if it's the only client in the policy right now) is set up the selection list like such:

System State (or Shadow Copies):\
C:\
D:\TopLevelFolder1
D:\TopLevelFolder2
D:\TopLevelFolder3
D:\TopLevelFolder4

This way when you start the backup of this client, you'll actually have 6 streams coming off of it (4 of them from your D drive). I think you can also accomplish the same thing by putting D:\*, but I can't remember if that's the exact syntax or not. Be careful with wild cards! ;)

Does this make sense?

Lance

[h_m]

>Only thing I'd maybe add is that you probably don't need your multiplexing up to 7 >unless you think your clients can feed the data that fast, but even so I don't think your >LTO1 library can keep up with that pace.

You always want the bottleneck on the tape drives, all streams will agregate to the speed of the drive, which if you get 2:1 compression then 7 streams * 5MB per stream will get the drive spinning at max throughput.


>What I'd do for your file server (especially if it's the only client in the policy right now) is >set up the selection list like such:
>
>System State (or Shadow Copies):\
>C:\
>D:\TopLevelFolder1
>D:\TopLevelFolder2
>D:\TopLevelFolder3
>D:\TopLevelFolder4

Just make sure you do not get disk thrash if you data is on SCSI arrays etc.

[Lance_Hoskins]

I agree you want the bottleneck to be at the tape, but if his environment is anything like ours, it doesn't matter if your pushing 3 streams at a drive or 20 streams at a drive, the overall throughput is the same. So my way of thinking is, no need to make restores slower than necessary with multiplexing of 7 if you get the same throughput at 3. Just my $0.02.

As for disk thrashing, you could limit number of jobs per client (at the policy level) to 3 on this machien so that not all of them start at the same time, but 4 streams coming from one volume shouldn't be that bad.

[Jim_Haywood]

Riight... OK.

I'm a wee bit confused as to what multiplexing setting I'm setting where now.

I'm backing up 4 different folders on the Fileserver and 3 off the Exchange server.

Both policies have a tick in 'allow multiple data streams'

In the schedules, the exchange server schedule has multiplexing set to 3, and the fileserver's multiplexing setting set to 4.

The Master server properties have a max number of jobs per client set to 4. The reasoning being that the max streams I could want are from the Fileserver, and that has multiplexing set to 4.

The storage unit has it's 'maximum streams per drive' set to seven. Theoretically allowing both policies to run at full multiplexing at the same time when run...? (If the drive can handle it!)

Thoughts to these settings?

I'd expect (Though you guys seem to be the authority) that the Fileserver client at least should be able to feed data that fast as all the data is local to the netbackup install (No transfer over a network, all through SCSI) is thrashing the SCSI an issue here considering we are looking at a brand new HP DL380 which at that time of night is not being used by any clients / users?

Oh, another question, do I need to change 'Maximum concurrent write drives' setting in storage unit settings to get multiplexing working? Currently set to 1

[Lance_Hoskins]

With regards to your question, that setting won't matter since you only have one drive.

My only change to your configuration would be to make multiplexing on each of the schedules the same. Pick 3 or 4 and go with it. If they're different, then they can't use the same tape--which would defeat what you're trying to do here. Also, with that in mind, the retention level needs to be the same, but I'm guessing it already is since you've been backing up both policies to the same tape in prior days (when it worked of course).

Let us know how it goes!

Lance

[zippy]

Jim,

This post was not a quick as you wished :)

http://ftp.support.veritas.com/pub/support/products/NetBackup_Enterprise_Server/276940.pdf

JD

[Lance_Hoskins]

Weak! ;)

[Jim_Haywood]

Aye, looked through that PDF, but I think I needed that little extra bit of hand holding offered by the good people on here!

I've made the multiplexing three on each policy... That should do it! Retention is set to one month on both.

Let's see what happens tonight. Fingers crossed!

Thanks again for all your help gents (And / or possibly ladies!)

Results tomorrow...

[Lance_Hoskins]

Not a problem. I'll look for a status when I the sun comes up in the states!

[h_m]

>
>Thanks again for all your help gents (And / or possibly ladies!)
>.

Thats only on a Sunday... (joke ;) )

[Dennis_Strom]

iF your running netbackup 6.0 make the frequency 1 day vs 7 hours. If your not running 6.0 do not upgrade past 5.1.

[Lance_Hoskins]

No guts, no glory Dennis. My only recommendation is that if you upgrade, wait until January when they slipstream MP4 onto the NBU 6.0 CD. There were some significant bugs with 6.0, no doubt, but you can't live in the past forever!

[Jim_Haywood]

Morning all!

Backups seemed to work last night... Maybe a little too well. My Activity monitor shows the following for last night:

EXCHANGE
00:00 - 1:31
00:30 - 1:30
00:30 - 1:32
00:30 - 1:20

FILESERVER
00:00 - 5:40
00:00 - 5:40
00:00 - 4:54
00:00 - 1:07
00:00 - 2:48


So that'd make a total of nine jobs that ran last night. Is this what I should be expecting with multiplexing turned on, or have I made a booboo somewhere? A couple of the jobs only reported partial success, (Little yellow man!) but looking at the problem report, it's all due to locked files, which I expect is down to users leaving their workstations locked with files open instead of logging off like good users. (Correct me if I'm wrong!)

I suppose the acid test would be to do some test restores?

Using a clean install of netbackup 6. Is it recommended by one and all to move from 7 hours to one day with this being the case?

I'm a wee bit excited over here!

[h_m]

Good work Jim, everything is working as it should be!

Ok, if multiplexing is set to 7 on the STU, then only 7 streams can run at any one time, which means 2 streams had to wait until other backups completed. The big thing to look at is the throughput per stream, add all of the throughputs together and see if they match the speed of the drive. If they do not then think about upping the STU multiplexing.

As you say the little yellow man is due to an open file which has been missed, each job will show the first 10 files missed. The other way to check is to run bperror from the command line and look at the contents of the output. if the files are critical then resolve accordingly.

NBU6.0 - there is currently a bug in scheduling: -

http://seer.support.veritas.com/docs/285529.htm

hence why people have said to set the frequency to a day. read the above alert for reference.

[Lance_Hoskins]

Looks good to me!! If you're running NBU 6.0, make sure you're on MP4 otherwise you're inevitably going to have issues. As for the number of jobs, that looks good, but in my head I was expecting 10 jobs to run. It looks like the 4 I was expecting on the Exchange server ran (Sys State, C, D, E), but it apears you're one short on the FileServer (based on what I was thinking you had - Sys State, C, D (with 4 separate streams coming off of D).

If you open each of the jobs it'll tell you what ran for each of them. Take a look and make sure everything was covered. Lastly, did everything get done in the 6 hour window or were there still things running at 6AM? Do a couple of test restores to ensure the backups worked properly and then it's off to fine tuning!

[Chad_Wansing]

Your backup is only as good as your ability to restore.....before you call this one good, I'd definitely do a test restore and mount of that flat file and verify your ability to get the IS up and consistent with that kind of a backup. I didn't see if you'd said that you're shutting down the Exchange services prior to the flat file backup, but if you're not, your Exchange backups are useless. Just something to think about (I don't mean to be the harbinger of bad news...). Backups can be a tricky business, especially where DB's are concerned, and if you're not planning for them correctly, you'll find your butt in the wind at the worst possible time.