Buffer Size Data for My Site

OS of the media servers?

All 3 that are attached to disk are...

Red Hat Enterprise Linux Server release 6.7 (Santiago)

Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

The master server is Solaris 10.

Current Netbackup version is 7.6.0.2.

Sorry I should have posted all that to begin with.

No worries... IMO, in general, most performance issues nearly always come down to disk (either client side or media server side).   I know you've put a lot of detail up re buffers and settings and config, and that is really appreciated, as it really helps us understand the setup - in fact it really is a nice level of detail.

Now, the odd thing with performance investigations... is... that they are just that... "investigations".  The first step is done (you've done it above) which is establish the high level application view of the base/original/starting point of the configuration.

We now need to dig a little bit deeper - and look at how that "captured config" is actually behaving... i.e. it's one thing to capture the settings... but another to capture the actual behaviour.

I'm not sure how you're going to take this... but I would first do some research on the "iostat" command and/or any other defaut tool that comes with your RHEL installation - and look at only one or two things initially (to rule them in or rule them out - for now)... during times of heavy load, and lighter load (i.e. during the heavy monthly jobs, and the not so heavy weekly jobs)... to specificaly look at disk IO latency, and disk IO queue lengths.  See if you can find some nice examples (in this forum, or on Google) of capturing the iostat numbers into a CSV file and then graphing them in Excel - or maybe there already exists a really nice tool for capturing and graphing?

I don't want to give any specific commands to run, because, TBH, it would be much much better, not only for you yourself, but also for the backup estate/environment and your business... if you research exactly which iostat commands you will use - because once you start digging in to the capabilities of these performance capture tools - then you need to be able to adapt the switches/options of the tools to suit your environment - hence I'm not going to give you any specific advice like... you must run so-and-so command, etc.   It is better if you work out how best to use the peformance reporting tools to suit your needs and to better match the config of the environment.

In short - it is the disk behaviour I would look at first - because you have stated CPU and RAM are all good, and I think you'd kind of already know if you had any glaring network issues.

Does something like this help? This is a report on the LUNs that make up the disk pool on this one media server. This is the one week average response time.

And please please take this the right way... when I see statements like:

"The media servers however have a ton of memory so I dont see any OS performance issues on them at all."

...we need to remember what it is that we mean by "OS" or "media server".  We know what an OS is (a set of binary files which can be executed by a CPU), and we know that an OS is nothing until it is executed... yet, to be executed, a general purpose commodity OS typically needs CPU, RAM and disk.  These are the fundamental building blocks of the modern general purpose sequential processing machine that we call a "computer".  Now, just because the media servers have a ton of memory does not necessarily mean they are capable of performing in an optimal manner.  Give a Cray supercomputer a few disk shelves of storage - and said Cray will never perform to the expected level.

How about this... the best used computer system will be one that is finely scoped/designed/spec'd/built/balanced/tuned such that... the CPU is 100% busy, AND the RAM is 100% used, AND the disks are 100% busy, but still gets the required job done within the required run window.  i.e. you've spent the minimum amount on hardware to get the job done.  Just because a CPU is 100% busy is not necessarily a problem.  If it gets the job done, then is there a problem?  No.

On a heavily loaded system where our "expectations", of getting a job done within a specific time frame, are not being met... then usually at least one of the three basic fundamental building blocks will *always* be a bottleneck.  Now, whether you have cause to notice this bottleneck or not is another matter.  But, note this, there could be two (or more) concurrent bottlenecks, which is better phrased as... there could be two or more elements of the computing system which are both running at 100% at the same time.

This means that if one does not take a holistic view of the entire system, that one can easily be lost chasing down one aspect of performance...and invest time and money in effort and hardware addressing the "discovered limitation"... to later on only find that the situation has not improved at all... because another (second) element was already also at around 100% utilization - and so improving only one area/facet/element resulted in no appreciable benefit.

Ok - re that response time graph...

Firstly, I'm not really sure that I can (or should) say "this is good" or "that is bad"... I don't think that's the way we should play this.  I think it would be better if we think about helping you consider things, i.e. help you make the deductions and decisions.

Secondly, a performance investigation is more like a game of chess.  You have an opponent, and he/she is constantly changing position - and you need to keep an eye on the whole game whilst attacking certain specific areas.  What I'm trying to say is...

1) Smoothed average graphs over such a long period, are great for managers - not good for us.  We need musch much finer detail.  Also, you don't state whether that graph covers a good time, or a bad time, or an indifferent time.  The reason is that the "weekly average" graph is like looking at a mountain from a distance and deciding that there is no gold to be mined, just because we can't see the rich seam from such a great distance.

2) We don't have any idea of how said storage is configured, connected or shared - nor what it really is.   Do you see why I'm so reluctant to actually say anything about that graph?  :)

3) A response time (latency) graph on it's own is not much use (BTW - I'm not asking you for more graphs) - you need the average IO packets sizes (to better understand the application) and IO queue depth (to better understand the OS driver) and the average MB/s (to better understand throughput) - all together - but in much finer detail covering the actual time periods of "accpetable load" and "unacceptable performance".

4) In short you need a raft of different graphs and views at your fingertips whilst you press on.

To be perfectly honest I dont know how any of that helps me resolve my current situation. I am looking for help with tuning my current Netbackup environment based on what objects I have in it. I can provide the information on current CPU and RAM allocated to the media servers and report on their usage if required. I appreciate the responses, but at this time I have not seen much by way of helping resolve the current issue I am facing. Every search I have done for this issue has turned up with buffer related findings.

And I do agree here...

"Now, just because the media servers have a ton of memory does not necessarily mean they are capable of performing in an optimal manner."

What I am saying is that the server is functioning without issue, but the application installed on it is not. I need to tune/correct the issues with the application. In this instance the application installed on it is Netbackup.

Look more closely at disk.  If the CPU is not 100% and the RAM is not 100%, then it's probably disk.  You might be getting 10ms reponse times - but if the IO queue depth is like 2+ then for every IO in flight yopu already have at least another waiting.  Do you see how IO response times on their own can be misleading?

What I am looking for is information on how to collect the required data in order to tune Netbackup to my environment.

I dont as I dont know how to ascertain any part of this sentence...

"You might be getting 10ms reponse times - but if the IO queue depth is like 2+ then for every IO in flight yopu already have at least another waiting."

I am attempting to be very clear and precise here that I am looking for assistance/guidance with how to get the needed information in order to tune my environment.

I am assuming you have been through this process before? Can you share what was done in order to tune your environment or one you have worked on?