What’s Next for Big Data Flash Storage?

Laura Shepard is Senior Director of Product Marketing for DDN Storage.

Last month HPE announced its plans to acquire Nimble and double down on its move into “the fast-growing flash market” for the enterprise. Days later Dell EMC announced it would drop its DSSD flash offering for big data and HPC because the market is too small.

Although Dell EMC “found little market” for DSSD, don’t be deceived about whether or not there’s a market for big data flash storage. There is and it’s growing.  In the HPC space, where DDN Storage plays, we continue to see a clear and growing need for flashed-based innovation. DDN’s Infinite Memory Engine (IME) flash offering is seeing strong demand.

Alongside traditional labs such as the Joint Center for Advanced High Performance Computing (JCAHPC) and Oak Ridge National Laboratory, and the more traditional high-end academic high performance computing (HPC) research, there’s also growing interest within enterprise organizations who want to to speed up their HPC-like workflows.

There’s an evolution coming in how data is managed in these really high-end, performance sensitive HPC-like workflows in the enterprise.  New autonomous vehicles programs for example need an HPC-like solution to manage huge volumes of high-resolution image and sensor data and real-time data processing. More broadly, the intersection of machine learning with standard business intelligence (BI) is causing a huge need for performance on very large pools of data, much of it now being unstructured, to quickly and accurately answer business critical questions. Also, in Financial Services new, more complex trading algorithms are using more and more data to obtain faster results that are needed in shorter times.

This trend is only going to grow – and with it, there is a growing need to eliminate bottlenecks between server and storage and improving performance.  That’s where the right flash solutions can and will help.

In a similar, but not the same, way as all flash arrays have eased standard enterprise bottlenecks, adding a flash layer can solve file system bottlenecks in HPC or HPC-like environments. However, unlike standard flash solutions, this new flash layer is different and is all about divorcing performance from capacity and provides a whole new way to manage data, which can live in the flash layer, can protect data, and allows organizations to manage everything in an environment to make sure it’s in the right place, talks to (and accelerates) an organization’s applications, and can maintain a POSIX interface if required. Vendors who get it right will have:

Whether or working in a traditional HPC environment or one of growing number of HPC-like enterprise environments, don’t let the departure of DSSD dissuade you from exploring the value of flash. The overall flash market was $15 billion last year. IDC predicts it will grow to nearly $20 billion by 2020. This transition to flash is creating a huge opportunity for positive change in the industry. There are a lot of companies, and products, chasing this transition and the most successful ones will be those who take the opportunity to completely change the game in terms of how data is created and managed.

Opinions expressed in the article above do not necessarily reflect the opinions of Data Center Knowledge and Penton.

Industry Perspectives is a content channel at Data Center Knowledge highlighting thought leadership in the data center arena. See our guidelines and submission process for information on participating. View previously published Industry Perspectives in our Knowledge Library.