There are with Symantec products two solutions to implement global deduplication :
- a solution based on appliances 5020;
- a solution based on your own hardware servers (where we will install the PureDisk software appliance) and a mid-range storage disk connected on a SAN (for deduplication data).
Servers based :
Can anyone advice me on the best solution ? What are you experiences with the 5020 appliances, with the PureDisk software appliance ?
Symantec's own admission is that they are working on performance on both the appliance and
We do have serious performance issues with Puredisk and have had to resort to tape. There are other posts in this forum that confirm this.
On the pro/con side
Pro: Monitoring. I'm lead to believe there is hardware monitoring and a dial home service...
Con: Hardware vendor specific agents are not supported, so your typical monitor agents eg. HP Management agents may not work.
The specific performance issue addressed in TECH154166 is rehydration. Rehydration is the process of reassembling data out of deduplication storage to stream to tape.
A lot of performance issues are resolved by following best practices: http://www.symantec.com/docs/TECH158430
For those performance issues not resolved by following Best Practices, they are being addressed in software: Version 126.96.36.199 is the first step, with Appliance updates to follow.
If you have other performance issues not related to rehydration, then please contact support to discuss them.
It wasn't clear in the original post as to how Puredisk was being used. I was giving the O.P. an insight into potential issues we have as a customer, which Symantec also acknowledges they are working on. Looking at the release notes 188.8.131.52 I agree the rehydration issues start to be address. We've not tested it yet. This is a factor that should be taken into consideration when making a purchasing decision to determine whether this is a turnkey solution.
We've been running Puredisk for a year now as part of a NBU solution specified after taking professional services from a Symantec partner. For us the best practice document which appears to have been written in April came too late for us. It seems to suggest to install an Advanced server in the middle for our implementation. This clearly involves additional capital expenditure which clearly wasn't budgeted in the original design.
I'll leave it for other poster can also give pros and cons which was the theme of the original post.
Lee and Todd, thanks for your answer.
The 5020 is installed with sata disk. In the new 5220, they have installed Sas disk.
Does it means there were performance problems with the 5200 and sata disk ?
Is it planned to have an evolution of the 5020 to a new model with sas disk ?
The 5220 is using completely different hardware compared to the 50x0 Series and the 5200.
The form factor of the 5220 is also completely different, and additional storage is via an attached Storage Shelf. This provides additional flexibility when deploying where you may want one 5220 as a master server only (with it's built-in 4TB), and then additional 5220's with Storage Shelves, or 5200's or even 50x0's for storage.
(My personal opinion and preference is that you shouldn't have MSDP on the master server, or load up the master server in general with too much for it to do. Regardless, all of the Appliances we offer are very robust and where performance is concerned, they're pre-tuned to SCREAM.)
Since testing, using and supporting the 50x0 Series is part of my job (in addition to all the other NetBackup Appliances), I can tell you that there aren't any performance issue with the 50x0 Series that I'm aware of.
I can't comment on any unannounced future products or product changes that I'm aware of.
There are just inherent issues with MSDP/PDDO today, that hopefully an update will fix. Taping out, is painfully slow, no amount of "bests practices, " will fix it. It's a hotfix waiting to happen...
There is a nice view that it's an appliance, and it's easier to setup. But you only do that once, so it's not a big deal
With NBU50x0, I'm on the fence of its overpriced hardware from a questionable hardware manufacturer with little North America experience, being serviced by a 3rd party... So roll your own servers for cheaper, with the storage you need today, and grow it out as needed.
Add to that, it seems PD is updated faster than the appliances in regards to fixes, or at least more inline to NBU updates too. Over a year ago at Vision, they talked about putting everything on the same track... So much for believing Product Management.
If you're concerned with performance and being able to stream tape drives in tape-out conditions, you might just want to look into the SD3 appliances. For a similar price, you'll get 3x (or better) performance from a US-based manufacturer who delivers solid products that are designed to last for many years, which is the goal of a backup/recovery appliance in the first place. The SD3 was the first appliance to break the 1TBpm threshold, in 2009, and is way beyond that now. And, they're made by the same company who grew up with NBU (managed by one of the first 12 NBU apostles who worked on release 1.3!). Have a look... you just might be surprised.
So Inquinox is just an OEM vendor to PureDisk... That doesn't solve the inherent issues with PD, you're just swapping out the hardware vendor which is not the culprit to the poor performance seen in the wild.
Not quite... InQuinox is a Symantec reseller who just happens to specialize in building appliances (in Austin, TX) that meet more stringent requirements for performance, while reliably running PureDisk.
The poor rehydration performance of PureDisk can be improved significantly by hardware. Yes, it is the same code, but the underlying hardware can be optimized for PureDisk, which is what we have done. The SD3 was designed from the ground up with restore speeds in mind. We use specialized disk drive technology, built the array to handle the specific way PureDisk accesses data, and then optimized the filesystems placed upon the array. The performance improvement is very significant, as noted by the SD3's backup performance which can attain backup ingestion rates exceeding 1 TB/ minute.
InQuinox SD3 appliances are available as high-end models as well as low end models. The high end models provide High Availabliity, built-in 10 GB Switches, high-performance media servers, and self-healing, high perfromace disk subsystems (using fault detection isolation and repair technology) with rapid rebuilds desined to last years without drive replacements.
The mid-range systems are available as MSDP servers that can also function as FT Media Servers, and as Windows Backup Hosts for VMware, and can optionally house a SanXfer Library server to provide SanXfer capabilty for a degree of data protection not availble anywhere else.
1TB/min? Marketing numbers much? You can't use client-side dedupe numbers to factor that value. You need to use the ingest on raw data, or using target-side dedupe.
I'm sure your "numbers," are accurate, but at the same time, misleading as the box can't really ingest 1TB/min of actual data, you're assuming a dedupe rate of 10x or greater at the client.
If your numbers are true, the NEC HydraStor (claimed fastest in the world), EMC DataDomain's (fastest claimed in north america, they seem to not count NEC in Asia LOL) and Exagrids (Their grid architecture is neat) of the world should just give up on their game.
The 1 TB/min ingest rate of the SD3 is based on a deduplication rate of about 15x. From what we can gather, this is the rate that is used by DataDomain when they publish their 23 TB/hr rate, so this rate is a fair comparison. Ingest rate on raw data is handled differently - it has a much different, less efficient CPU profile and is not a good measure of the ultimate performance. You are only going to get raw data ingest rate on the first backup of new data - and after that there will be duplicates. What I mean to say is that the speed at which a system can ingest raw data is only one measure and it may not be the best measurement. You have to measure the ability of the dedupe appliance to fingerprint the data stream, recognize duplicate data, and store the new data. The processing is different. When you store never-seen-before unique data into PureDisk, you will see a CPU performance profile that differs greatly than if you are sending data that has been seen before. Most backup data being sent to the appliance has been seen before.
That said, speed of the disk is only the first part of achieving a fast ingest rate. We use a proprietary method to ensure that the Content data can be written as fast as possible on the disks (high throughput), and that the metadata is on a disk subsystem capable of high IOPS. Once that is done, re-hydration and backup performance dramatically increase. We've done a lot of work to achieve that 1 TB/min ingest rate. I would welcome a test against anyone - I promise we'll blow them away and then they can all go home. :-)
The numbers are based on deduplicated data streams - that's the point. They are real data streams being stored in the appliance. It cannot digest 1 TB/min of raw data, but you will never send raw data - by that I mean random data that cannot be deduplicated - to a deduplication appliance. We used a dedupe rate of about 15x, which is what DataDomain uses for their published deduplication rate.
At that rate, we can handily digest 1 TB/minute. Raw disk speed is phenomenal on our SD3. On a 6 node (192 TB) SD3 we can write (direct to disk) a little over 5 GB/sec That's 833 MB/sec per node compared to HYDRAstor's 750 MB/sec per node. But I assure you, the tuning does not end with raw disk speed. We carefully configure the disks to optimize throughput specifically for the Content Router data and we optimize IOPs for the metadata.
The reason you don't want to compare the ingest of non-dedupable data in the different solutions (NEC's HYDRAstor, EMC's DataDomain) is because it isn't useful. RAW IO is not proportional to dedupelication ingest rate. You want to measure deduplication input because that is how it will be used and this is a direct measure of the performance of the entire system. PureDisk excels at this once you put it on high performance, tuned hardware. If you look at CPU Usage on the SPA and the nodes during ingestion of data you will see a completely different footprint with non-dedupable data than you will with data that is being deduplicated. PureDisk ramps up far more quickly than the competition as you vary the dedupe rate - because of how it hashes the fingerprints on the Media Servers, as well as many other accelerators.
Hope that helps!
Are you quoting rates based on natively backing up or used with Netbackup in the same manner as Puredisk doing PDDO?
I'd be interested in best/worst case type statistics for rehydration to LTO5 tapes when used with Netbackup.
It's PDDO through and through. Rehydration is faster on the SD3 because of three optimizations:
1. The metadata is stored on disk optimized for high IOPs. This allows PureDisk to look up the references to the data segments constitutiong the data stream very very quickly.
2. Once the location of the data segments are known, they must be retrieved from disk as fast as possible. This operation requires the content data segments be retrieved from a disk subsystem tuned for high throughput.
3. Finally, the rehydrated data needs to be sent as fast as possible to the Media Servers. We use dual 10Gb ethernet bonded interfaces between the PudeDisk nodes and the Media Servers so that the rehydrated data streams can have a large enough pipe to stream multiple tapes.
I have installed two of the SD3 units into our environment, the level of support and dedication that InQuinox places behind the SD3 appliances is by far the best I have ever experienced. Ray worked with me personally on many occasions to resolve an issue when Symantec would provide less than adequate support. The performance numbers are not fluff or "Marketing" but real numbers, the InQuinox SD3 deduplication appliances are fast if not the fastest and the way the company as a whole stands behind its products is second to none. If you are in the market for a deduplication appliance don't overlook InQuinox in your search they will not disappoint.
I know I'm giving you a hardtime, but I will say this. I applaud that you give support above and beyond Symantec's own. Symantec's own support is HORRIBLE! L2 or an escalation manager is the first thing I ask for, even then it's days to weeks to solve. I've had cases open for months too!
Don't worry - I don't blame you at all for questioning how we can take on the big guys on performance and win, and I appreciate the opportunity to respond!
And I also kow that Symantec support is improving as of late. If you have any issues please let them know so they can address it.
Anyone from support listening to this discussion please jump in!