Using Solid State Drive Storage with Veritas Storage Foundation

[Oscar_Wahlberg]

Executive Summary

The performance gap between computing devices and storage devices has been growing for several years. Currently, CPU and DRAM operations are measured in nanoseconds while hard disk drive (HDD) operations are measured in milliseconds – that is several orders of magnitude difference. Solid-state storage has been identified as a technology to fill this performance gap between computing and storage devices. Of particular interest is solid-state disk (SSD) technology which couples solid-state storage with hardware which provides a disk device interface through a physical host interface such as SATA, SAS, FC, and PCI-E.

While the performance benefits of solid-state storage make it extremely attractive for most high-end enterprise SANs, deploying and efficiently managing mixed solid state disk and regular hard disk storage environments comes with a unique set of challenges:

• Visibility: It is increasingly common to offset the cost of solid state disks by replacing a given amount of FC drives with a mix of SSD and SATA drives. Efficiently managing multiple types of storage devices provisioned to a host at minimum requires being able to differentiate between them. If all storage types look identical to the system administrator, the risk of errors during provisioning and routine management operations dramatically increases. Given the big differences in cost and performance between solid state disks and SATA disks, confusing the two can be catastrophic to applications.
• Manageability: Day to day management of the storage consumed by a particular host involves monitoring utilization, adding storage, removing storage, replacing storage, etc. Doing any of these operations in a mixed storage environment is complex. Having the right level of visibility on the host helps minimize errors. Automating these operations so that growing a host volume based on solid state disks automatically happens on solid state storage only (and never on hard disks) and vice versa is the only way to provide 100% protection against allocation errors.
• Data placement policies: The reality is that very few applications deserve to have all their data on SSD. On the other hand, most applications can benefit greatly from having a subset of their data on SSD. So it makes sense for users to be selective in the type of data that should be placed on solid state disk and the type of data that should be left on regular hard disk. Those decisions have to be granular. Doing this at a full LUN level is too coarse.
• Automation: Placing the right data on SSD is important. However, in real production environments, the data that should be on SSD and not on HDD changes over time: today‟s hot data is tomorrow‟s cold data. Being able to simply place the right amount of data on the right type of storage is not enough. Users will need to be able to continuously optimize their mixed SSD and regular disk storage environment and ensure that data that is consuming SSD storage indeed deserves to be there based on its current needs.

Veritas Storage Foundation provides advanced host based storage management functionality that allows system administrators to effectively tackle all of these challenges. This paper walks through these benefits and provides best practices as well as detailed „how to‟ information regarding configuring and using Veritas Storage Foundation in a storage environment containing a mixture of SSD and HDD storage. The operating systems covered are AIX, HP-UX, Linux and Solaris. It is assumed that the reader has some familiarity with Veritas Storage Foundation.

SSD Overview

SSD Performance Considerations

Solid-state storage is a nonvolatile storage medium that employs integrated circuits rather than magnetic or optical media. There are two basic categories of solid-state storage: RAM-based and flash-based. RAM-based solid-state storage uses DRAM or SRAM chips similar to those used as volatile memory in a computer system. Flash-based storage is a collection of flash memory chips; there are several ways of organizing the cells into a flash memory chip. Most of the solid-state disks (SSDs) available today are based on NAND (From the “Not And” logical operation used within integrated circuits) flash technology because of the technology‟s overall price-performance ratios in comparison to other technologies. Therefore the rest of this paper will focus on NAND flash-based SSD.

Flash memory has the concept of an erased cell. An erased cell represents a „1‟ state. A cell programmed with the appropriate charge represents a „0‟ state. An important point in this regards is that changing the cell from a „0‟ to a „1‟ state (often referred to as erasing the cell) is different than changing the cell from a „1‟ to a „0‟ state (referred to as writing the cell). Because of the semiconductor properties involved in flash memory, writes (and erasures) are slower than reads. In NAND flash, cells are organized into pages which are the smallest unit that is readable and writable. In the current generation of SSD‟s 4KB pages are the norm. These pages are grouped together into blocks. Currently, it‟s common to have 128 pages (512KB) or more (e.g., 4MB) in a block. These blocks are the smallest structure that can be erased in a NAND flash. Erasing a block can therefore be a more costly operation than a normal data write.

A flash-based SSD is normally comprised of several integrated circuits (IC). From a performance perspective, these ICs can be accessed in parallel. So, if each IC can provide 20 MB/sec of read bandwidth and an SSD has 10 ICs, the SSD can provide about 200 MB/sec of read bandwidth. Write performance, however, is more problematic. Normally, an entire page must be written to a completely erased area. So, if less than a page is written, the entire page needs to be copied from flash to DRAM memory, the new bits of data must be inserted, and the entire page written back to the flash memory. To make matters worse, the copying may need to involve an entire (erase) block depending on the level of remapping (page versus block) being maintained by the SSD. Also, once a (physical) page has been written, it can‟t be overwritten; it must be erased first before it can be written to again. Because a flash-based SSD can have write performance that is much worse than its read performance, the overall performance mix with reads and writes can be an area for confusion. If doing pure reads is about 10 times faster than doing pure writes, it might be expected that a 50-50 mix of reads and writes would be 5 times faster than doing all writes; however, that is incorrect -- it will be less than twice as fast. Note that even though write performance of an SSD does not match its read performance, it is still orders of magnitude better than an HDD.

SSD Reliability Concerns

There are a couple of reliability issues with flash memory that impact the design and operation of flash-based SSDs. One of particular interest is that flash memory cells have a finite lifecycle, meaning each cell can be written/erased a limited number of times before the cell wears outs and becomes unprogrammable. This is also referred to as the write endurance of the flash memory. It is common for cells to wear out between 10,000 to 1 million erase/write cycles. Even at 1 million cycles, commonly used parts of a file system and other software will easily exceed that many cycles within a relatively short amount of time. So, in order to maximize the life-time of a flashed-based SSD, the flash memory cells need to wear out at roughly the same pace. This can be achieved by moving the contents of erase units around. This technique is called wear-leveling. In its simplest form, a controller within the SSD runs a wear leveling algorithm that ensures writes are evenly distributed amongst all blocks within a flash chip and across all the flash chips in the SSD. The SSD must maintain a (persistent) logical-to-physical block mapping of its storage along with the wear-leveling data. This logical-to-physical block mapping also allows the SSD to get better write performance since “out-of-place” writes can be performed to an already erased block. However, if the SSD were to “fill up”, write performance would go down drastically since every write would require an erase. The approach SSD vendors use to avoid filling up the SSD is to under provision the drive. For example, an SSD with 80GB of flash memory might only advertise 73GB of space available to the end user. While an SSD does not suffer the mechanical failures of HDD's they can fail. Users requiring high availability and reliability will need external forms of redundancy for SSDs, similar to HDDs.

SSD versus HDD

The following lists some of the advantages of a flash-based SSD over an HDD:

• Faster random access for reading. Since an SSD has no read/write head to move, it has access times that are much lower than an HDD.
• SSD access times are independent of the physical location of the data thus file fragmentation has almost no impact on performance.
• SSD storage tend to support more diverse operating environment characteristics:
  • Lower power consumption
  • Minimized heat production
  • Ability to endure more shock and vibration
  • Operates over a larger range of operating temperatures.

The following lists some of the advantages of an HDD over a flash-based SSD:

• HDD prices are currently considerably lower per gigabyte
• HDD capacities are currently considerably higher
• The (random) write performance of HDDs is typically closer to the read performance
• HDD‟s may demonstrate a longer life cycle as SSD‟s have limited write endurance and require wear leveling

Because the cost per bit of SSDs is much higher than HDDs, SSDs are still not cost effective for bulk data storage for all classes of data. Clearly their deployment will likely be focused where the added cost can be justified, such as the most performance sensitive workloads, or where other benefits justify the increased cost. SSDs will therefore be most useful in environments where a subset of data or metadata would benefit greatly from very high IOPS and/or very low latency. Initially users who are currently using enterprise HDDs, but intentionally configure them with less 100% utilization in order to get the highest possible random access performance by minimizing head movements will benefit the most from enterprise SSDs at the least additional cost. As SSD costs come down, SSDs will become cost effective for an increasingly wide range of users, but again predominantly for random access workloads. Because magnetic drives (HDDs) are quite good at sequential workloads, the benefit of SSDs for those workloads is much less for their higher cost. Thus, SSDs will initially be used for specific applications, such as OLTP or paging/swap space, and they will be used as the “tier zero” storage within tiered storage systems.

Flash-based SSDs are starting to appear in the enterprise in a variety of forms:

• integrated into a storage array
• as a stand-alone external disk (or JBOD offering)
• integrated into the server (e.g., as an internal disk drive or PCI-E device)

Storage arrays will initially take advantage of SSDs by allowing them to be used with existing array enclosures and controllers, usually allowing a mixture of SSDs and HDDs within the same enclosure. Because of the extremely high random I/O performance of SSDs, an array designed around the performance of many HDDs may encounter internal bottlenecks earlier when using even a much smaller number of SSDs. Attempts to use more SSDs than the array controller can support may not translate into the expected performance gain, thereby wasting the additional SSDs and the cost paid for them. This necessitates software solutions that allow applications to optimally layout their data on a mixture of SSDs and HDDs. This paper will explain how the Veritas Storage Foundation product can be used to leverage SSDs in mixed storage configurations.

DOWNLOAD THE ATTACHED PDF TO READ THE COMPLETE WHITE PAPER

Authored by: Anindya Banerjee, Principal Software Engineer;Murthy Mamidi, Advisory Engineer;Shashikumar Paul, Development Manager;Niranjan Pendharkar, Distinguished Engineer;Michael Wahl, Sr. Principal Software Engineer;Oscar Wahlberg, Sr. Principal Technical Product Manager

[Terry_Cutler]

I have an SSD in my laptop and in a demo system - truly wonderful little devices. 

More insights\demos\videos on SSDs available at http://www.intel.com/design/flash/nand/ssd-info-center.htm and http://www.intel.com/design/flash/nand/extreme/index.htm

Video interview between Intel and Sun - http://video.intel.com/?fr_story=7462f5f91e38e14060541a3a8750c21c59c4a101&rf=bm 

Whitepaper on hybrid storage pools at http://download.intel.com/design/flash/nand/SolarisZFS_SolutionBrief.pdf