08-29-2012 12:38 PM
- Clustered master server with 2 Linux nodes running NBU 7.1.0.3.
- Two Windows Server 2008 R2 media servers on HP ProLiant DL380 G5 (2* E5410 @ 2.33GHz & 24GB RAM each)
- HP MSL8096 tape library with 3 drives (multipath connected, latest drive and library firmware installed), LTO-5 tapes only.
- A mix of Windows 2003 R2 and Windows 2008 R2 clients (VM's and physical systems - approx. 200+ servers). All clients are on NBU Client v7.1.0.3. Physical servers are backed up over the 1GB network. VM's over fiber.
Conducting tests using HP's Library and Tape tools (installed on one of the media servers) it shows the MSL8096 drives are able to backup 30GB of data up to an average speed of approx 310MB/s (unencrypted) and up to 260MB/s (with 2.1 encryption).
Though trying to backup physical systems over the 1GB network this remains between a disappointing 5 ~ 20MB/s. Even if one single backup job is started manually without any other jobs running this does not come close to the magic threshold of 50MB/s. Backing up shares with NDMP on our NetApp sometimes reaches the 40MB/s but still not enough.
Several of our config files with the following values (if applicable):
NUMBER_DATA_BUFFERS = 256
NUMBER_DATA_BUFFERS_DISK = 64
SIZE_DATA_BUFFERS = 524288
SIZE_DATA_BUFFERS_DISK = 524288
SIZE_DATA_BUFFERS_NDMP = 524288
It's very time consuming to get this resolved and quite honestly I'm lost where to look.
Any assistance is appreciated.
Thanks in advance!
Solved! Go to Solution.
08-29-2012 01:22 PM
1 Gb network can at BEST receive data at 100MB/sec.
This means you will never be able to get close to 1 tape drive's capability.
Have you configured Multiplexing in STUs as well as in schedules?
Have you increased Max Jobs per Client?
99.9% of slow backups are due to clients' slow read speed from disk.
For this reason, we use multiplexing to increase number of simultaneous jobs to a tape and so increasing transfer rate.
Enable bptm log on media servers as a starting point.
At the end of backups, look for 'waiting for full/empty buffer'.
'Waiting for full buffers' means that the media server was waiting for data - problem is on client or network.
Please see these URLs for performance tuning tips:
http://www.symantec.com/docs/TECH147296
08-29-2012 01:22 PM
1 Gb network can at BEST receive data at 100MB/sec.
This means you will never be able to get close to 1 tape drive's capability.
Have you configured Multiplexing in STUs as well as in schedules?
Have you increased Max Jobs per Client?
99.9% of slow backups are due to clients' slow read speed from disk.
For this reason, we use multiplexing to increase number of simultaneous jobs to a tape and so increasing transfer rate.
Enable bptm log on media servers as a starting point.
At the end of backups, look for 'waiting for full/empty buffer'.
'Waiting for full buffers' means that the media server was waiting for data - problem is on client or network.
Please see these URLs for performance tuning tips:
http://www.symantec.com/docs/TECH147296
08-29-2012 01:48 PM
do SAN based backups for best perfromance. Or implement SLP.
08-29-2012 02:19 PM
Thanks Marianne!
It looks like setting Multiplexing in schedules solved my problem - it was set to 1 and changed it to 4 for testing purposes and started a backup; whole process ran on approx 70MB/s instead of e.g. 20MB/s
Question now is, what would be the appropriate setting for Multiplexing in schedules? Is it dependant on several factors?
08-29-2012 02:29 PM
A MPX setting of about 4 seems to be generally accepted as a good balance between backup speed and restore speed.
The higher the mpx value, 1. The more memory you need (each stream takes up 'buffer' space, and 2. How long a restore takes.
A mpx backup is of course multipple clients data 'mixed' together, so for a estore, all of it has to be read, even if only restoreing data for one client.
Just occasionally, I see issues where a restore is running slowly, to find that a very high mpx value whas used >12 that soet of value.
I have seen a restore that was taking >36 hours. Re-running the backup with a sensible mpx value allowed exactly the same data to be restoed in a much much short time (can't remember the exact value, but it was single figures).
Martin
08-29-2012 03:12 PM
Thanks for the useful info Martin.
Would you say the buffer sizes are correct to use with a multiplexing setting of 4?
08-29-2012 10:31 PM
You need to check bptm logs to see what the effect is on throughput and the difference in 'waits' at the end of the backup.
The key to finding MPX value that is good in your environment, is to increase in small increments and perform restore tests.
Test restore with MPX value at 4. Record result.
Try MPX of 6 for backups; do test restore.
See if there is significant difference in restore speed/time.
08-29-2012 11:18 PM
Difficult to say, as Marianne has pointed out below you may find better performance with 6, but then again, you have a trade off with restore speed,
In a nutshell, you'll have to do some serious testing, run some backups, then try a restore and see how long it takes - if the time is acceptable to you, then you have the answer.
Be aware, there is nothing that can be done to speed up a restore that is slow due to a high mpx value.
Also, the higher the mpx, the more memory required ...