cancel
Showing results for 
Search instead for 
Did you mean: 

job through put lower than expected

Mark_Henrie_2
Level 4
It seems we can only get 16mb/sec on any of our jobs. I have check all our fiber connections and it seems to be a netbackup bottleneck.

We are running NBU 6.0 MP3

Windows master server
Unix Media Server
Unix Clients

Are there any settings In NBU that I should be looking for?

Sorry for being so vague I am just starting at a new job and am not yet familiar with all the systems.

Thanks in advance
28 REPLIES 28

DavidParker
Level 6
Mark,
Here's another thing to check when getting low throughput.
Check all of the networking pieces involved between the master/media server and the client. Specifically the NIC in the master/media and the client as well as the network port on the switches each are connected to.

Make sure that NONE of these things are configured with the 'auto-negotiate speed' setting and that they are ALL 'hard set' to their appropriate speed (100, 1000, etc).

Are the rest of your clients getting good throughput?

Dennis_Strom
Level 6
Using multiplexing can help quite a bit. Look at your individual schedules. Start at 3. You also have to allow mutliple threads on the storage unit properties. In addition try a bit of kernel tuning.
for solaris 8 and 9 this is a good start.

* Message queues
set msgsys:msginfo_msgmax=8192
set msgsys:msginfo_msgmnb=65536
set msgsys:msginfo_msgmni=1024
set msgsys:msginfo_msgtql=1024

* Semaphores
set semsys:seminfo_semmni=2048
set semsys:seminfo_semmns=2048
set semsys:seminfo_semmnu=1024
set semsys:seminfo_semmsl=300
set semsys:seminfo_semopm=128
set semsys:seminfo_semume=64

* Shared memory
set shmsys:shminfo_shmmax=33554432
set shmsys:shminfo_shmmni=512

DavidParker
Level 6
Hang on a sec, Mark.
When you say 16 mb/sec, are you talking about 16,000 KB/s from the Admin Console display?
If so, 16,000 KB/s is actually very good performance from a Windows host ...

Mark_Henrie_2
Level 4
We have multiplexing set at 4 currently. Both on the schedule and the storage unit.

Mark_Henrie_2
Level 4
Yes, 16000KB/sec from the policy job display in the activity monitor. I have seen as high as 30000 it just seems like its bottlenecked somewhere.

DavidParker
Level 6
Have you configured the touch files:
NUMBER_DATA_BUFFERS
SIZE_DATA_BUFFERS

?

16,000KB/s is still pretty darn good if you ask me ...
I'm used to seeing 2-8,000 from most Windows hosts ...

It also depends on the type and size of the data you're backing up; if you're backing up large numbers of small files, your throughput will suffer.

Mark_Henrie_2
Level 4
What numbers are people seeing from Unix vs. Windows Clients?

I realize the number of files will vary im just looking for a base range.

THanks

DavidParker
Level 6
Check out this tech note:
http://seer.entsupport.symantec.com/docs/236103.htm

I know it's for a specific library, but it will at least give you some ranges for the BUFFER files.

Apparently we're not using these on my Windows media server, so maybe someone else who has those configured can throw in some input; I think there was one of those numbers that had to have a little attention paid to it due to a flaw in Windows, but I'm not sure.

Actually, try this one:
http://seer.entsupport.symantec.com/docs/183702.htm

Chia_Tan_Beng
Level 6
> Yes, 16000KB/sec from the policy job display in the
> activity monitor. I have seen as high as 30000 it
> just seems like its bottlenecked somewhere.

Hi Mark,

16MB/s is consider quite good. For those with 30MB/s, are the backup data resides on SAN storage. Size and number of files dose affects the throughput but normally if the data resides on the local disk, this is normally the expected performance.

You can try testing using bpbkar process at client end to backup the data to /dev/null device and see if the local disk or SCSI adapter is the bottleneck. Page 81 of the following document provide the helpful steps:

http://support.veritas.com/docs/281842

Mark_Henrie_2
Level 4
OK, maybe i was a little confused when I asked this question. I have an EMC nas that is attached via fiber to a switch which is attached via fiber to our unix media server. We even bypassed the switch and ran direct fiber from the nas to the Media Server ( UNIX ). They were previously using net vault and were able to reach 80MB/sec. The nic cards in the media server are set to negotiate 2gb.

Maybe this helps a little more. Thanks.

Mark

Lance_Hoskins
Level 6
Mark,

What does the data look like that you're trying to back up? Are you backing up many small files or a few larger files? Also, set up some logging on the client (bpbrm) and on the media server (bptm) and turn the logging level up. Look through those logs for "waited" and see if it's waiting for full or empty buffers. You can also find the butter size in there if you need it and this may lead you to some re-configuring that might need done. Let us know the results of your findings and maybe we can help from there.

Lance

h_m
Level 6
what tape drives are you using?

Mark_Henrie_2
Level 4
The client and the media server are one in the same. The media server is backing up the EMC from /dev/emcpowera I setup some logs but its still running around 15-16000/sec. I didn't find any logs that said waiting. But maybe im not sure what to be looking for in the logs.

Mark_Henrie_2
Level 4
LTO3 in and ADIC Scalar i2000

Chuck_Stevens
Level 6
> Yes, 16000KB/sec from the policy job display in the
> activity monitor. I have seen as high as 30000 it
> just seems like its bottlenecked somewhere.

Heh. I *wish* I saw numbers that high. I've never seen a backup run faster than about 13000 KB/sec, and most run less than half of that speed. I'd love to find a "magic bullet" to fix that, but never have.

Mark_Henrie_2
Level 4
From a SAN through fiber to LTO3. No one has seen high numbers?????

Dennis_Strom
Level 6
30 is high. LTO4 will not be doubling speed since very few installations are able to push LTO3 to capacity. Most sites just do not have the infrustructure and are unwilling to cough up the money to optimize backups, which are quite often a low priority. I have a fairly small site at just shy of 10T and I saturate our gig E network. No SSO.

Chia_Tan_Beng
Level 6
> From a SAN through fiber to LTO3. No one has seen
> high numbers?????

I've only LTO-2, one of my backup clients reach 28MB/s on network backup because the data is mount from SAN storage. Some from SAN through fiber to LTO2 get 22MB/s. All depends on what kind of files its backing up and whether the storage are busy, assumpting the media server is not the bottleneck. I've to say that the numbers range between a few MB or even KB in some slow backup client or with small files.

h_m
Level 6
If you are running an MPX of 4, and have 15-16MB/sec throughput ten this equates to 60MB/sec, so it's not far off 70MB/sec, so you will get a bit of shoe shining. I think ADIC's use IBM drives so it will match 60MB/sec...

So if the above is what you are seeing, and have 4 jobs @ 15Mb/sec, then try reducing the MPX value. The next question is how many HBA's you have in your server and how many drives you've got hanging off the back of it....?