Dell DR4100 performance issues

[Patrick099]

Hello

Is there anyone who had some expierence with performance on a Dell DR4100 with 1 additional shelf, so a total of 24 x 3TB SATA 7,2k drives, 56TB. Backup Server Dell DR710, 32GB Memory. All connected same switch with 10Gbit cooper. We had sometimes (I think when lot of jobs running) very bad write performance and the biggest problem read is very bad (for B2tape), I not got more then 1Gbit out of the box even if only 1 job is running. The "container" on the DR4100 for backup is a cifs share. No OST etc. as we don't have a license.

See also attached pdf.

Have now a ticket open at Symantec and Dell but just won't to know if anyone have the similar config and can share some perfromance/experience?

Thanks a lot

Patrick

[CraigV]

...have you made sure that all NICs are hard-coded to 10GB FULL? On the switches, server and storage.

What is your RAID configuration? Are you running any AV scans on the BE services? Check your AV and if they are listed, exclude them from AV scans.

Thanks!

[Patrick099]

Virusscan on BE Server is disabled and all services stopped. Network seems to be ok, ipfer testing shows 7-8GBit/s sending to DR4100 receiving 3-4GBit/s.

Maybe CIFS is slow??

RAID on DR4100 or on the BE Server? DR4100 no clue, setup by Dell consultant.

BE Server, 1 x RAID1 (2 x 136 GB SAS 10k) for OS, BE DB etc, 1 x RAID1 (2x 300GB SAS 10k) for BE Catalogue. 32GB RAM on BE Server. 2 CPU 6 core (Intel(R) Xeon(R) CPU X5650 @ 2.67GHz)

But when duplicate jobs are running system is mainly idle, CPU around 10%, RAM 7-8GB, NIC receiving 400MBit/s to 1Gbit/s, disk usage normal.

I do some additional perf. testing with dell today, iperf and IOMeter. Currently restore job is running with 2,7GB/min.... even from tape I got more speed.

Thanks

Patrick

[pkh]

I believe your device is a dedup appliance. If you use it as an ordinary disk storage, you are taking a double performance hit. Any backup set will be dedup'ed and when you do a verify it is rehydrated before the verification begins. Also, you are not getting a good dedup ratio because there is no steam handler for the backup set. To improve performance, you need to licence Boost on the alliance and the BE dedup option, and use the appliance as an OST device

[Patrick099]

Ok thanks for the input, but that was promised by Dell that this is also possible with this Appliance. Ok my fault didn't ask for a test appliance...

Let's see what Dell support is telling us, if this is really the case we give back the appliance and order 50TB disk shellf to extend the Backup Server and do the backup there, duplicate to tape and cloud from there.

Maybe dedup from Symantec.

Regards
Patrick

[pkh]

If you are going to use the BE dedup option, it would be better to license Boost at the same time and use the appliance as an OST device

[Larry_Fine]

"Boost" is an DataDomain/EMC marketing/license thing and wouldn't apply directly to the Dell DR4100.

I don't know for sure about the licensing of the Dell DR4100, but it may be that all you need is the BE Deduplcation Option license to activate the DR4100 as as OST target.  I think you can activate a 60 trial of the Deduplication Option in BE.  Then you could compare the CIFS performance with the OST performance in your environment and decide which is better for you.

[pkh]

DR 4100 is supported as an OST device by both BE 2012 and BE 2014. You got to check with Dell as to whether any licence is required to use it as an OST device

[Patrick099]

Thanks for all the input.

We did some testing yesterday with Jumbo frames etc, seems the Backup Server is ok as far as network speed. For the DR4100, we only got this speed from the appliance:

DR to Backup Server 3,3-4,5GBit/s, Backup Server to DR 5,7/5,9GBit/s

not bad but I expect more. We also had some issues on our HP Switch with one of the 10Gbit ports, received some strange errors yersterday so now use another Port on the switch also created a LACP trunk.

Waiting now to hear again from Dell.

[Patrick099]

I think in the DR4100 license is all covered but need to check with Sales, regarding OST. Maybe if all is fine with CIFS i spend some time to do OST testing. But right now unforunately no option.

[Dell_SamL]

Hello Patrick099,

When taking data off it rehydrates for tape so there is a performance hit. Now one thing that you can do is it looks like you have both your 10gg & 1 GB connections in the same bond.  What I would try is to create a bond 2 bonds & have the 10gb connects bonded together and then create a second bond with the 1gb connections by using advanced networking options & see if that helps increase your speed.

Please let us know if you have any other questions

[Patrick099]

Dell told us to use Cat6a cable instead of Cat6, ok waiting now for the delivery as I don't have them on stock. Also we do some testing by directly connecting the Dell DR4100 to a different 10GBit port on the Backup Server.

Did someone had experience with Jumbo frames? It was enabled on the Backup Server as we plan to implement iSCSI LUNs directly on the Backup Server. Now turned it of but there is no difference as far as I can say.

Regarding the bond, I have now 2 x 10Gbit connected (for testing only) on the DR4100 so this should not be a problem.

Patrick

[CraigV]

Jumbo frames are going to increase the amount of throughput through the NICs, but at the cost of performance on the media server. You can test it, but keep an eye on performance.

Thanks!

[DellDPDR]

Patrick,

If you would like, please help to send email to "#usscz02-svdc-esc-team@dell.com" with Dell support case number and issue details. We will engage our escalation engineering to work on your issue.

The DR4100 product itself can sustain a single stream CIFS backup of 400-500MB/s no problem. We need to find out what the bottleneck is.

Thanks,

Dell DP DR

[Patrick099]

So after hours of testing here the latest infos:

played around with MTU, seems to have an impact. So now connected the DR4100 directly on the Backup Media Server on a different 10Gbit NIC port with MTU1500 and with Cat6a cable. Setup bond0 for direct connection on DR4100 and bond1 with 1Gbit nic for Mgmt/ AD integration.

Iperf now better, in with multiple tcp streams 9,5 Gbit/s, out around 4 Gbit/s

So now in general better performance, also changed Storage settings on Backup exec to default values according to admin guide. I don't know why these values were different.
max file size: 50GB
block and buffer to auto
auto detect settings to auto

So after that speed in Backup exec is corresponding with iperf. write to DR4100 600-800 MB/s (Backup exec job rate between 30-40 GB/min) .

but still having trouble reading from DR4100 to tape, max out is 200 MB/s (job rate max. 8.5 GB/min) single job, multiple we got to the 200MB/s limit.

So any help is much appreciated...

Regards
Patrick