cancel
Showing results for 
Search instead for 
Did you mean: 

Ridiculously busy environment

JohnOBrien
Level 3
Partner

Hi All,

Currently trying to get a handle on an enviornment:

Netbackup 6.5.6 - Out of support, plans for upgrade in progress

Master/Media OS: Solaris 10
# of tape drives: 5, single STU
RAM: 32GB
Disk Staging: 22TB across 5 STU

Clients: Mixed UNIX and Windows
Multiplexing set to x 32
limit jobs per policy unchecked
allow multiple data streams checked
# of nightly jobs: 1396

Envornment is creaking badly but was just about managing to get the majority of jobs done in the window.

However something changed last week, windows, unix and network admins say otherwise, but since last weekend jobs that were previously running with moderate speeds have either spiked up to 30mbps or down to .03mbps.

Example A

Client with multiple streams running:

2 streams were granted the resource at the same time 08:00
under the same policy : Windows-Exchange
into the same STU: robot-TLD0
into the same physical tape drive: 123456

however one stream achieves at throughput of 30mbps the other 0.03mb

Previously it was averaging between 5mbps and .3mbps which was slow but meant that all jobs eventually ran to completion.

Example B

Client with multiple streams running:

5 streams were granted the resource at the same time 08:00
under the same policy : Windows-FS
into the same STU: DiskVol1+2 (14TB)
however one stream achieves at throughput of 4mbps the other 4 streams only achieve 0.03mb

Jobs running @ .03mbps will never complete.

Any suggestions on how to better balance how the streams are using the bandwidth between the clients and the Master/Media.

Thanks,

John

4 REPLIES 4

frank_butler1
Level 3

Sounds like someone implemented QOS on the network

As a test create 2 policies (windows OS backups)  make 1 policy set backup selections to C:\ and make the other policy backup selection be D:\ for the same client in each policy.

Launch them both and see if throughput it limited.

Or just call the network team and see if they are using QOS between the client and the media server.

 

SymTerry
Level 6
Employee Accredited

Really this will come down to perfomance tuning. Something may have changed in the env, but tuning can help. Try a file copy from the client to the media server to see if its the pipe the data is going across.

Review the NetBackup 6.5 Backup Planning and Performance Tuning Guide (TECH62317)

 

Michael_G_Ander
Level 6
Certified

Sounds to like you saturate the connection with the first stream, so the rest is just trickling along.

Have you tried to change a backup to just run with 1 stream ?

Other than that I would get together with a network admin and run iperf between clients and media servers to see what happened with 1 stream, 2 streams and so on

Disagree with Terry, don't think you can help by performance tuning Netbackup if the underlying infrastructure has the bottleneck

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

Nicolai
Moderator
Moderator
Partner    VIP   

Look like a network issues to me. Test with iperf from the master/media server to different sources. You you see a similarity to Iperf performance and "general" backup speed.

http://www.symantec.com/docs/HOWTO64302

While you are looking a low hanging fruits -  Did you configure NUMBER_DATA:BUFFERS and SIZE_DATA_BUFFERS.

Take a look in /usr/openv/netbackup/db/config

While buffer tuning will not make the underlying HW faster it will allow the software to make a better usage of it. 

See : http://www.mass.dk/netbackup/guides/49-netbackup-buffer-tuning.html as well as the Backup planing and performance tuning guide as SymTerry linked to.