Backup and Restore speed on 10Gb interface

[traghav86]

Hi All

Please advise what will be the backup speed on 2x10gb bonded interface using IBM SAN as msdp on Windows Enironment on below backup/restore types, currenlty am getting not morethan 150Mbps. what is expected speed with above network config. NBU version is 8.1.1 and 8.1.2

-Windows FS

-Vmware

-SQL 

[sdo]

Performance problems are always the hardest to solve... because it's not broken, there's no actual connectivity to confirm is visible / not-visible.

The problem could actually be absolutely anywhere (and I do mean anywhere) along the data path.  There could be two problems in the data path (in which case you have your work cut out).  Gosh, there could be ten problems in the data path (good luck with that).

Have you tried the usual FTP, SCP speed copy tests ?

If you've got the inclination, and time, then here's a test that I might try... try a backup where the only difference is the storage presentation to NetBackup... i.e. do something a bit like this...

...present another LUN from same IBM SAN storage array to the media server... partition and format this LUN as ext4 (or whatever is supported)... mount it... create a plain DSU in NetBackup to point to that new mount... and run the same backup to that new storage unit.

If the new backup is much faster... then the problem is clearly with MSDP dedupe engine... if the backup speed is the same, then the problem could still be anywhere else in the data path.

[sdo]

Here's a thing... I have some big appliances, they all use MSDP, and I can ingest over 800 MB/s over bonded 10Gb, but this is the aggragated inbound total to MSDP for many clients.  However, for any one single client, I can't think that I've ever seen more than 150 MB/s for pure random data from any one single client to MSDP.  So maybe you're hitting an MSDP single stream processing limitation.

For those few special backup jobs which absolutely must run faster, I use an SLP to stage to Adv.Disk first, and then duplicate to MSDP, with immediate expiry from Adv.Disk upon completion of duplication.

[sdo]

I might try some GenData tests if I get time.

[Marianne]

@traghav86 

I am only noticing now that your issue does not seem to be related to NetBackup Appliances. 
Maybe we should move your post to the NetBackup forum for better visibility? 

[sdo]

did some tests:

quietenned MSDP on target: compactoff, rebaseoff, crccheckoff

.

tests t1 to t5 all had client side dedupe enabled

source: 5230 2x10Gb in a bond, 802.3ad

target: 5340 2x10Gb in a bond, 802.3ad

.

t1: source: gendata 1 file, 100GB,  0% random, 99% dedupe -> target: Adv.Disk  averaged  938 MB/s

t2: source: gendata 1 file, 100GB,  0% random, 99% dedupe -> target: MSDP      averaged 1124 MB/s

.

t3: source: gendata 1 file, 100GB, 99% random,  0% dedupe -> target: Adv.Disk  averaged  947 MB/s

t4: source: gendata 1 file, 100GB, 99% random,  0% dedupe -> target: MSDP      averaged  131 MB/s

.

now rerun t4 using three physical clients, the same 5230, the 5340 also as source, a decent spec HP server:

t4,1: source: gendata 1 file, 30GB, 99% random,  0% dedupe -> target: MSDP    averaged  129 MB/s for 5230

t4,2: source: gendata 1 file, 30GB, 99% random,  0% dedupe -> target: MSDP    averaged  127 MB/s for 5340

t4,3: source: gendata 1 file, 30GB, 99% random,  0% dedupe -> target: MSDP    averaged  105 MB/s for HP server

total  361 MB/s for all three

.

now rerun as t5, same three clients, but 20% randon and 80% dedupe :

t5,1: source: gendata 1 file, 30GB, 20% random, 80% dedupe -> target: MSDP    averaged 1038 MB/s for 5230

t5,2: source: gendata 1 file, 30GB, 20% random, 80% dedupe -> target: MSDP    averaged 1045 MB/s for 5340

t5,3: source: gendata 1 file, 30GB, 20% random, 80% dedupe -> target: MSDP    averaged  390 MB/s for HP server

approx 2500 MB/s for all three

.

now copied policy t5 to policy t6 and disabled client-side dedupe, and rerun...

...and now the receicing 2x10Gb bond on the 5340 appliance hit wire max of 12.1 Gb/s...

...and dedupe ingest showed :

t6,1: source: gendata 1 file, 30GB, 20% random, 80% dedupe -> target: MSDP    averaged  795 MB/s for 5230

t6,2: source: gendata 1 file, 30GB, 20% random, 80% dedupe -> target: MSDP    averaged 1015 MB/s for 5340

t6,3: source: gendata 1 file, 30GB, 20% random, 80% dedupe -> target: MSDP    averaged  370 MB/s for HP server

approx 2180 MB/s for all three

.

then I reran again, as test t7, but this time back to 99% random, 0% dedupe, three clients, no client side dedupe, and all three each averaged 130 MB/s to 135 MB/s - so an apparent ingest of 400 MB/s - but with a real combined aggregate arriving at 2x10Gb media server NIC of around 300 MB/s.

.

my conclusion is :

- NetBackup is extremely capable at wide fanout data ingest

- and that single MSDP streams do not like utterly random and thus non-dedupe-able data - and would appear to have a ceiling of around 135 MB/s per individual stream in my tests.

.

@traghav86  Rahul, was your SQL data compressed ?  that would make it look utterly random to the dedupe engine.