Duplication performance poor
Our backup server is NBU Enterprise Server, 7.5.0.7 running on OEL 5.10 (x86), on a Dell R620 server and Dell PVTL 2000 (2 x LTO5 drives), each drive connected via 6Gps SAS interface to NBU server.
We backup several items to disk, among them two Exchange 2010 servers. We need the GRT option as we need to recover items at a message level. The disk is an iSCSI volume, presented rw to the OS as mount point /nfsbackup (/dev/mapper/idatap1 on /nfsbackup type ext4 (rw,_netdev)).
The issue I am running into is what I consider to be poor performance during duplication from disk to tape. We can't get the images off disk fast enough (it's a 1.5TB volume) to make room for the next images. The current situation is a great example. They are both Exchange 2010 GRT images, one from each server.
Job #1: Started 12/02/2014 20:30:44; 55% done (217823232KB of 395169442KB done), est throughput: 4731KB/sec
Job #2: Started 12/03/2014 01:08:54; 54% done (202821632KB of 374121233KB done), est throughput: 6889KB/sec
Interfaces
NBU server: eth1, eth2 on 1Gbps (not bonded) interfaces to storage network; jumbo frames enabled
iSCSI storage: multipathed, 2 interfaces @ 1Gbps each, jumbo frames enabled
I'm monitoring the NBU server interface with Solarwinds, and I'm getting maybe 80MB/s (combined), barely breaking 8% utilization. Now, utilization does peak from time to time (see image at end) but then it always settles down to this level.
Storage Unit: BasicDisk
Maximum concurrent jobs: 12
Fragment size: default (524288 MB is greyed out)
High water mark 70% / Low water mark 30%
Enable Temporary Staging Area checked: Staging Schedule at Frequency of 12 hours, no exclusions or Start Window specified
Other NBU touchfiles
NET_BUFFER_SZ: 262144
NET_BUFFER_SZ_REST: 262144
STAGING_JOB_KB_LIMIT: 157286400
SIZE_DATA_BUFFERS: 262144
SIZE_DATA_BUFFERS_DISK: 1048576
NUMBER_DATA_BUFFERS: 256
NUMBER_DATA_BUFFERS_DISK: 512
Performance graphic
Thanks for any insight and help in advance,
Michelle
Marianne / Riaan -
Yes, there is great news to report! After a lot of googling, I found some information which related to iSCSI volume configuration at the disk enclosure (Dell PowerVault MD3000i) and how the OS (OEL 5.10) recognizes the mounted volume on a block level.
Resources used:
- http://en.community.dell.com/techcenter/storage/w/wiki/optimizing-powervault-md3000i-performance
- Performance of read-write throughput with iSCSI: http://www.monperrus.net/martin/performance+of+read-write+throughput+with+iscsi
- Understanding effect of read-ahead on open-iscsi block device: https://groups.google.com/forum/#!msg/open-iscsi/B50Qo9-hpFM/QtFF749WNEAJ
After hurting my brain with all of the above, I adjusted settings on both the PowerVault side AND Linux side to get some screaming performance. I will share the values which worked for me, but for anyone who wants to adjust their settings, please keep these points in mind:
- Your environment is different from mine
- You must understand what you're doing and why.
- And, of course, YMMV.
On the MD3000i side
- Made sure all hardware (disks, controller modules) were up to date with latest firmware
- Increased volume segment size to 256KB
On the NBU master/media server (Oracle Enterprise Linux 5.10)
- Used the blockdev command to understand current disks and their read-ahead/blocksize settings
- Changed all /dev/sdx disk devices(where "x" will be b,c,d...etc) associated with the /dev/dm-n disk device to have a RA (read-ahead) of 4096 (512-byte sectors) and blocksize to 4096 (512-byte sectors).
- Jumbo frames were already set to 9000 on the SAN network.
The goal here is to maximize your throughput without saturation.
