Innovation at the Array Level

The block interface is restricting innovation for Arrays even more than disk drives and we are seeing the rapid commoditization that results from this lack of ability to add meaningful value-add features. A couple years ago array marketers used to talk about segmenting the array market into horizontal tiers based on price, capacity, availability, data services, etc. and they used to have three or more tiers. Today, due to commoditization, this has collapsed into only two tiers, as described to me directly by more than one storage administrator. The top tier still exists with EMC DMX and HDS 9000 and other arrays for the highly paranoid willing to pay these high prices. Below that however, is a single tier of commodity arrays. The sales discussion is pretty simple. "Is it a 2U or 3U box?" "How many disks?" "What capacity?" Then, the next questions are "How cheap is it today and how much cheaper will it be next quarter?"

Some would say this is fine and the natural progression in the storage industry. Arrays become what disk drives where fifteen years ago (the thing that stores the bits as cheaply as possible), and higher level data services move to virtualization engines or back to the host stack.* As an engineer seeking to innovate at the system level however, I can't accept this.

As with disk drives, it's a distributed computing problem and there are improvements that can only be done inside the RAID controller. These include improvements in performance, providing an SLA and improving utilization based on the information being stored, securing the information, and complying with information laws. All this requires knowledge of the information that is stripped away by the block protocol.

Arrays try to do this today the only way they can - through static configuration at the LUN level through management interfaces. One problem with this approach is the granularity is too large (LUNs). Another is it's too static and is difficult to manage, especially when combined with the need to manage switch zoning, host device nodes, etc. Finally they are trying to manage the information by applying properties to the hardware storing the information vs. on the information itself. Take for example, the need to keep a log of any user who accessed a particular legal record. One, you can't use LUNs to store each record (so you can't manage it at the LUN level) and two, information doesn't sit on on one piece of hardware anymore. It gets mirrored locally, maybe remotely, and probably backed-up as well. If you're doing HSM, it might even completely migrate off the original piece of storage hardware. Now remember that the law is to track who accessed the INFORMATION, not a particular copy on one array.

If the storage devices are allowed to keep the record grouped (an object) and app servers and storage agree on a protocol for authentication and a format for the access log, then this becomes a solvable problem. Other ways the array can help manage the data is by storing data on the right class of storage such as RAID 5, mirror, remotely mirrored, or a single non-redundant disk drive. To optimize use of storage, these should be applied at the file, or database record level because required storage service levels change at that granularity.

* Note to array vendors. If this is the direction arrays are going the way to win is clear: follow the example of successful disk companies like Seagate. Build high-volume, high-yield manufacturing capability, get your firmware and interop process fully baked, relentlessly drive down HW costs, and standardize form factors and functionality to enable second-source suppliers.