cancel
Showing results for 
Search instead for 
Did you mean: 

Slow backup from media server

philalbert
Level 4

Hi,

 

We are running Netbackup 7.6.1 on Windows 2008 R2 Sp1.

We have 1 master and 6 media servers.

One of our media server is not backing up as fast as the others. We are usually able to backup at 70-90MBps even more, but on this one particular media server we are unable to get more than 20-30MBps.

Any idea why or what could cause the speed difference?

 

Thanks for the help.

Philip

1 ACCEPTED SOLUTION

Accepted Solutions

Nicolai
Moderator
Moderator
Partner    VIP   

All the way from the client ?

what is the NUMBER_DATA_BUFFERS configured on the media server ?

Use at lest 128 or 256 buffers per stream.

What is the media being written to on that media server - if tape is deployed try running a clean tape. Dirty tape drives do a lot of re-write because read after write fail.

Tell me more about the environment and the media server not performing.

This media server, is it located in a switch with a blocking or non-blocking switch port - ask the network admin. A blocking port shared the same bandiwth between multipel switch ports - so if 4 client both send/recive traffic a full load you may see a lot less than the 1GB.  A non-blocking deliver promised bandwith of 1GB - but those blade are a lot expensiver to buy than the blocking ports.

View solution in original post

17 REPLIES 17

revarooo
Level 6
Employee
Have you tried transferring a large file between the client and this media server? Still slow? Backing up to tape or disk? If tape then setup a basic disk storage server....still slow? You could try setting up a null storage server too (google null storage unit netbackup - for instructions) Also did this media server ever backup quickly?

philalbert
Level 4

Have you tried transferring a large file between the client and this media server? Still slow? 

A: Just did and I get around 400-500MB/s transfert rate.

Backing up to tape or disk? If tape then setup a basic disk storage server....still slow? A: Backing up to tape. Can you point me to how-to set that up please?

You could try setting up a null storage server too (google null storage unit netbackup - for instructions) A: I haven't, I'll test it and let you know the results.

Also did this media server ever backup quickly? A: From what I could see in the logs, it never backup as quickly as the others.

 

Thanks for the quick reply.

mph999
Level 6
Employee Accredited

Let the slow backup finish, then look in the Details for that job in activity monitor.

Somewhere near the bottom, it will have a line or maybe two saying 

,,, waited xx times for full buffer, delayed yy times

,,, waited xx times for empty buffer, delayed yy times

If you could paste these line(s) up here ,,,,

Nicolai
Moderator
Moderator
Partner    VIP   

How to configure a "null" Storage Unit for NetBackup 7.5.x and 7.6.x. Media servers.

http://www.veritas.com/docs/000013708

philalbert
Level 4

Here mph999:

2015-11-06 12:50:20 - Info bpbkar32(pid=14460) bpbkar waited 2040612 times for empty buffer, delayed 2084866 times.   
2015-11-06 12:50:23 - Info bpbrm(pid=2240) media manager for backup id sva0182_1446761032 exited with status 0: the requested operation was successfully completed
2015-11-06 12:50:24 - Info bpbrm(pid=2240) child done, status 0         
2015-11-06 12:50:24 - end writing; write time: 19:43:56
the requested operation was successfully completed (0)

revarooo
Level 6
Employee

bpbkar waited 2040612 times for empty buffer, delayed 2084866 times.

 

All the buffers are full on the client, so the client is filling up the buffers, but either the network is too slow passing off the buffers to the media server or the media server is too slow offloading to final storage.

check the network or check the storage unit

 

Set-up the null storage unit: http://www.veritas.com/docs/000013708

Set your policy to use the null storage unit (preferably copy the policy and use the copied policy for testing)

If this is still slow with the Null storage unit you have an issue with your network. If it is fast to the null storage unit, you have a problem with your destination storage unit

Nicolai
Moderator
Moderator
Partner    VIP   

Look really bad.

My first suspicion is network, but before blaming a network admin I would test it out first :)

The tool for the job is iperf

You can download the tool here:

https://iperf.fr/iperf-download.php

Basically do cross test to narrow down where performance drop down to a unacceptable level.

Here are some link on how to use IPERF:

IPERF: How to test network Speed,Performance,Bandwidth

http://www.slashroot.in/iperf-how-test-network-speedperformancebandwidth

Using iPerf Windows Command Line Tool to Measure Bandwidth between Sites

http://www.zerto.com/myzerto/knowledge-base/00000124/

Yes - this is completly outside Netbackup. Software can't make hardware run faster.

philalbert
Level 4

Nicolai: 

I tried iPerf, but the speed is fine. Getting between 1.2-1.5Gbps in both direction.

 

revarooo:

As soon as the media server is done with his backups I'll test the null storage unit. I might have to wait tomorrow, I'll keep you posted.

 

Thanks

Nicolai
Moderator
Moderator
Partner    VIP   

All the way from the client ?

what is the NUMBER_DATA_BUFFERS configured on the media server ?

Use at lest 128 or 256 buffers per stream.

What is the media being written to on that media server - if tape is deployed try running a clean tape. Dirty tape drives do a lot of re-write because read after write fail.

Tell me more about the environment and the media server not performing.

This media server, is it located in a switch with a blocking or non-blocking switch port - ask the network admin. A blocking port shared the same bandiwth between multipel switch ports - so if 4 client both send/recive traffic a full load you may see a lot less than the 1GB.  A non-blocking deliver promised bandwith of 1GB - but those blade are a lot expensiver to buy than the blocking ports.

mph999
Level 6
Employee Accredited

Small correction ... the buffer is on the media server, not the client.  So bpbkar waiting for an empty buffer means the media server is slow emptying the buffers out to storage ???

philalbert
Level 4

The NUMBER_DATA_BUFFERS is set to 512 on all media servers.

The media is written to tape. I understand that a dirty drive could cause errors/delays, but the are all working fine with the other media servers.

This is our environment: 1 Master, 6 media servers. Only 1 media server is having performance issues. The media servers are connected to the tape library via fibre channel 8Gbps. They're also connected to the SAN via network or FC (for some specific volumes).

 

It looks more and more like an issue between the media server and the tape library. Still waiting for the media server to be free to test revaroo suggestion.

jim_dalton
Level 6

Whats the data profile that you are backing up and how does it differ from the others? By which i mean file count versus file size distribution, the extremes of which are large numbers of small files (which is slow) and small numbers of large files (which is fast); the performance differences being due to large numbers of file open/close. Jim

revarooo
Level 6
Employee

And to test Jims differing file sizes and quantities, you can use GEN_DATA directive as the file selection on that specific client (configured in the policy file selection) to enable bpbkar to generate data on the client to send over the network to the media server storage backend,

I would still test this with a NULL storage unit as backend storage to effectively throw away the data but still check the speeds of data transfer between client and media server

GEN_DATA Technote http://www.veritas.com/docs/000091135

jim_dalton
Level 6

I've a dusty thought in the back of my mind that gen_data isnt an option for windows but I could be wrong/ it may have been implemented in later versions. jim.

revarooo
Level 6
Employee

You are right, UNIX only. As the issue is with media server write speed, customer could use a UNIX client. If not then customer must use the live data if they only have Windows clients.

 

Another thought, if this Windows Media server is the only media server with slow backup speed, try backing itself up to it's own storage - still slow?

philalbert
Level 4

We found the issue, well Nicolai did. The NUMBER_DATA_BUFFERS config file was in place, but it had a .txt extension.   The extension was hidden in windows. (Yes I feel stupid)

I changed it and the backups that ran last night were running at 100MB/s.    :)

 

Thanks for all the help everyone!

Nicolai
Moderator
Moderator
Partner    VIP   

LOL - No problem :D

I was glad I could help. Thanks for marking a solution.