Seeing What's Next, Fourth and Final Part

The third group that Clayton says to watch for signs of change in an industry are the Non-consumers. In this case, people who have storage and information management problems to solve but are not buying current products because they either can't afford them, or can't adapt them to their unique needs. As I mentioned in Part I, the most interesting group of non-consumers are in High Performance Technical Computing (HPTC). They have both problems. One, their research budgets don't allow them to buy expensive supercomputers and RAID arrays, and two, today's arrays don't meet their needs anyway. To build 'cheap' supercomputers, they cluster together lots of x64 compute nodes running Linux. Then, because they spread their HPTC compute task across so many compute nodes, they need storage that enables sharing the data between all those processors. So, even if they could afford today's block or NAS arrays, they don't support this level of information sharing anyway.

SANs have enabled IT managers to consolidate their storage HARDWARE, but not necessarily their information. Block arrays don't have the capability to effectively share the same information between multiple application servers. Even if it did, most filesystems still assume they use captive DAS disks anyway. So, SAN admins have to partition up their storage through LUN masking and switch zoning so each file system instance thinks it's talking to captive, direct attach storage.

This won't work for highly parallel HPTC tasks. They need a filesystem with the capability to share files between multiple compute servers. Filesystems such as Sun's QFS do this with block storage but it's inefficient. Compute nodes have to spend a lot of time communicating with each other to compare which blocks are used and free, and which nodes have which blocks opened for read or write access. It's much more efficient to let the storage server do this which means, of course, it has to know how data is grouped into files and it has to manage ownership properties with each file. In other words, it needs an object storage interface. NFS is close but won't have the right ownership properties until V4++. So leading HPTC filesystems such as Lustre and Panasas' PanFS invented their own proprietary object storage protocol that runs over IP. Lustre users assemble their own object storage device using Linux on a commodity server. Panasas builds an object storage array providing one of the first commercial products for this market.

One of the reasons this disruption is so interesting is that I've talked to several enterprise datacenter managers who want similar capability. They want to move enterprise applications from large servers to racks of scaleable x64 blades. They want the ability for multiple instances of an application running on separate compute nodes to share information (not just array hardware). They also want quick re-provisioning so an application can migrate to a new compute node without manually reconfiguring LUN masks and switch zoning as required in block SANs today. And, of course, they will need SAN security and access logging with this to comply with information laws. HPTC is evolving this technology, mostly through open source. They are evolving object protocols that allow information sharing AND provide the underlying object protocol on which security and compliance features can be added as sustaining enhancements.

Summary
Like the old guy in The Graduate who's advice to young Dustin Hoffman is I have just one word for you: Plastics, I have just one word for the future of storage: NAS. NAS has been a classic low-end disruption growing among overshot customers. Netapp has grown with customers who don't need the level of performance and availability of a FC SAN, and want the low cost and ease-of-use of an IP network-based filer appliance. In parallel, the sustaining enhancements to give NAS the same performance and availability as FC SANs are in process. These include NFS over RDMA and NFS V4++ enhancements to enable multipathing and client data services. IP-based NAS WILL come up and displace Fibre Channel. No question.

Not only is NAS a low-end disruption, it is also the best candidate to become the next radical up-market sustaining innovation for undershot customers. These customers need storage servers that get data in meaningful groups with properties to let them manage compliance with information laws, manage content, put it on the right tier of storage, etc. Once NAS has the required level of performance and availability, then it just takes a series of straightforward sustaining innovations to give it those capabilities.

Finally, object-based storage on IP is growing as a new-market disruption in HPTC. Since the FC HBA vendors appear to have no interest in growing up to high-performance computing (while they lose the low-end to IP) so these users have moved to IP. Also, they have avoided NFS because they need sharing properties not available in NFS today and they don't need the full functionality of the embedded file system in a NAS server. I predict though, that once those properties, along with RDMA are available in NAS, that someone will build a NAS server optimized for Lustre. Then, the economics of commodity NAS arrays will make those attractive to HPTC.

Down at the spinning rust, data will always be stored in groups optimized for the hardware such as 512-byte blocks. For availability, it will also make sense to group disks together with RAID hardware optimized for those physical blocks but these block RAID arrays will become commodity devices. The questions for these vendors is to either be the winner in that commodity business, or to embed a filesystem, and meaningful data management, and compete as a NAS storage server. As for which players will pursue which strategy and who will win, I don't know. Clayton says one of the natural laws is that companies try to up-level. That says one or more existing disk drive suppliers may try to leverage their existing manufacturing capability and OEM relationships and do RAID trays. It also says existing block RAID vendors will move up to doing NAS servers. EMC has indicated it may go this way by acquiring Rainfinity. In parallel, existing NAS vendors such as Netapp will continue the sustaining innovations to give their products the availability, performance, and data integrity required to compete with Tier 1 block arrays. That will be one of the races to watch to see who gets to enterprise-class NAS first.