Can multiplexing cause slow backup

[Jecintha]

Hello,

We have Netbackup 7.5 configured on a Win 2008 R2, 64 bit OS Server. We see backup running at a speed as low as even 1 kbps at times.  We have media multiplexing enabled.  As per my understanding, media multiplexing directs data from multiple clients to one drive.  I need to understand if this can increase the network utilization and end up in reducing the backup speed.

Regards,

Jecintha

[RamNagalla]

hi Jecintha,

did you observe the same client backup is moving fast with out Mulitplexing?

Mulitpexing is generally helpfull ..when tape drive speed is not being utilized upto Max limits.,it can data from mulitple jobs and send it to single tape drive..

we need to check if this process is causing any new bottlenecks in the data flow..

does tape drives are moving with good speed with no mulitplexing is enabled?

when mulitplexing jobs is running data moves from the single media server... does this media server is capable of handlieng the jobs? is that over loaded?

is that mulitplexing jobs are comming from single client or different clients.. if single client ..does client is capable of send the data in multiple jobs( generally when mulitstream is enabled or when multiple policies running)

[Possible]

Multiplexing is generally used to reduce the amount of time that is required to

complete backups

The Enable multiplexing storage unit setting allows multiple backups to multiplex

onto a single drive in a storage unit.

Can u post the job details so that where is the bottleneck....from media server or client server.

Looking for the message...."Waited for empty buffer....Delayed....times..."

[Jecintha]

Hey Nagalla,

Thanks for the comment.  I have disabled multiplexing today and need to observe the things mentioned by you in tonights backup.  However, I need to know if there is a possibility of multiplexing causing the backup speed to get reduced so that I can plan other backup schedules according.

[Jecintha]

The job deails log is as under:

7/25/2013 10:28:00 AM - Info nbjm(pid=6788) starting backup job (jobid=56192) for client "clientname", policy "policyname", schedule Default-Application-Backup 
7/25/2013 10:28:00 AM - Info nbjm(pid=6788) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=56192, request id:{BB2474E0-F85E-4F94-A098-8A46E240FFCD}) 
7/25/2013 10:28:00 AM - requesting resource hydesm001-hcart-robot-tld-1
7/25/2013 10:28:00 AM - requesting resource hydesm001.NBU_CLIENT.MAXJOBS.XXXXXX
7/25/2013 10:28:00 AM - requesting resource hydesm001.NBU_POLICY."policyname"
7/25/2013 10:28:00 AM - granted resource hydesm001.NBU_CLIENT.MAXJOBS.XXXXXXX
7/25/2013 10:28:00 AM - granted resource hydesm001.NBU_POLICY.MAXJOBS.XXXXXXX
7/25/2013 10:28:00 AM - granted resource 0410L4
7/25/2013 10:28:00 AM - granted resource IBM.ULT3580-TD4.001
7/25/2013 10:28:00 AM - granted resource hydesm001-hcart-robot-tld-1
7/25/2013 10:28:00 AM - estimated 0 Kbytes needed
7/25/2013 10:28:00 AM - Info nbjm(pid=6788) started backup (backupid=MHTHPSBBIDB_1374728280) job for client MHTHPSBBIDB, policy XXXXXXXXX, schedule Default-Application-Backup on storage unit hydesm001-hcart-robot-tld-1
7/25/2013 10:28:02 AM - Info bpbrm(pid=4624) XXXXXXXXXXX is the host to backup data from    
7/25/2013 10:28:02 AM - Info bpbrm(pid=4624) telling media manager to start backup on client    
7/25/2013 10:28:02 AM - Info bptm(pid=4964) using 262144 data buffer size       
7/25/2013 10:28:04 AM - Info bptm(pid=4964) using 32 data buffers        
7/25/2013 10:28:08 AM - Info bpbrm(pid=9164) sending bpsched msg: CONNECTING TO CLIENT FOR MHTHPSBBIDB_1374728280    
7/25/2013 10:28:08 AM - Info bpbrm(pid=9164) listening for client connection        
7/25/2013 10:28:08 AM - connecting
7/25/2013 10:28:14 AM - Info bptm(pid=6072) setting receive network buffer to 1049600 bytes     
7/25/2013 10:28:14 AM - Info bpbrm(pid=9164) INF - Client read timeout = 300     
7/25/2013 10:28:14 AM - Info bpbrm(pid=9164) accepted connection from client        
7/25/2013 10:28:14 AM - connected; connect time: 00:00:06
7/25/2013 10:28:14 AM - begin writing
7/25/2013 10:28:15 AM - Info bpbkar32(pid=2200) Backup started          
7/25/2013 10:28:15 AM - Info bpbrm(pid=9164) Sending the file list to the client     
7/25/2013 10:28:15 AM - Info bpbkar32(pid=2200) dbclient(pid=2200) wrote first buffer(size=65536)        
7/25/2013 10:28:17 AM - Info bpbkar32(pid=2200) done. status: 0         
7/25/2013 10:28:20 AM - Info bpbrm(pid=4624) media manager for backup id MHTHPSBBIDB_1374728280 exited with status 0: the requested operation was successfully completed
7/25/2013 10:28:20 AM - Info bpbrm(pid=4624) child done, status 0        
7/25/2013 10:28:20 AM - end writing; write time: 00:00:06
the requested operation was successfully completed(0)

[Marianne]

Maybe show us details of a job that ran slow?

The above job took all of 20 seconds!

The only multiplex setting that can cause slow backup is too high a setting (like 32)  that could possibly overload the network or overload the media server.

As per Nagalla's excellent post - you need to find where exactly the bottleneck is.
First step is to identify clients with slow transfer rate, then determine their ability to generate data streams. Use tools such as bpbkar backup to null device to test.
Next step is to trace the network path and determine if that is the bottleneck. Multiple 'hops' to get to media server? Network cards and/or switch port settings mismatch? What type of network? 1Gb network card in media server can at best receive 100MB/sec. Determine speed of single stream from each client and then decide on multiplexing value in order to ensure constant rate of close to 100MB/sec.
Check switch logs during peak backup time.

As you can see - there is no 'quick fix' for performance tuning.

Best to download the Performance Tuning Guide and work through the suggestions: 
http://www.symantec.com/docs/DOC4483 

[Anshu_Pathak]

Multiplexing will help incase :

1. Slow client

2. Slow Network

3. Do not want to use multiple tapes at the same time for different backup jobs

Multiplexing will cause issue (degrade backup performance) incase:

1. Fast client

2. Fast Network

3. Fast Tape drives

Maximum allowed Mulitplexing value is 32 however recommended value is 10

[Jecintha]

Hello All,

I am not sure if I did not communicate clearly, but I actually meant to say that the data transfer that happens during backup is very slow. The range of data transfer is somewhere as 1 kbps to 20 kbps.  This is what I meant by slow backup.

[Mark_Solutions]

I think we did understand but the log you posted was for a job that only ran for a few seconds to it would not give any proper data throughput rate

Could you show us a log for one that ran for a reasonable amount of time that was slow - and explain what type of backup it is (Windows file system etc)

[Marianne]

We understood you clearly.

This is why I recommended that you should try to pinpoint the bottleneck. 

See my post above.............

[Jecintha]

Sure Marianne.  Got your point.  I'll work on troubleshooting issues from the Network perspective.  In the meanwhile, I'd like to know what is the ideal speed with which backup runs in Netbackup. Is there a standard range or does it differ with environment?

[Mark_Solutions]

Every envorinment has a bottle neck so you cannot put a speed limit on anything

Good disk, backup via fibre to LTO5 / ^ drive and you happily chug along at 200 MB/s+

Bad disk, lots of small files, remote site over slow network and it runs at only kb/s

To get an idea of you have slow rates would need to know far more about your environment - perhaps getting a Symantec Partner to come in and give you a health check or troubleshooting consultancy may help?

[Marianne]

In addition to Mark's excellent post - add up transfer rates of each mpx stream going to the same tape drive.

First step in chasing the bottleneck should be to determine clients' ability to generate data stream. See the Perf Tuning guide for bpbkar test. 
Next step will be to check network between client and media server.

[Jecintha]

Our environment has only one physical server (master server) and the rest all are virtual servers.  On further analysis and investigation, we have identified that the infrastructure itself is very poor with low CPU and RAM.  So also, the master server is serving as media server and I presume that there is an overload on the master server causing the slowness issue and hence I feel adding a media server might resolve slowness issue. Please comment if addition of media server can help us towards resolution. 

Thanks in advance.

[Jecintha · Accepted solution]

We had logged a case with Symantec and based on their advice, we had added a media server and also upgraded the network NIC cards.   This had helped in resolving the issue.