Big Data vs. Sparse Data

Jerry Gentry is Vice President, IT Program Management at Nemertes Research

In the last few articles I explored possibilities for data futures. We talked about Big Data and Sparse Data. The first is an industry trend that is moving rapidly to center stage for many IT data professionals. It hasn’t built the momentum to carry it far down the food chain of data managers, but you can see that it is coming. In the next one to two years you need to develop a Big Data strategy, regardless of whether you are a large or small enterprise. I’ll get to my reasoning in a little bit.

The second element was Sparse Data. That is a term I coined (actually, my colleague John Burke, suggested it). Sparse Data is an undercurrent. It is already out there, under the surface. It is less visible because no one has seen the need to search for ways to organize it to extract information locked within the data. In much the same way as Big Data, Sparse Data will become important for every enterprise. It is just a matter of time. To paraphrase a noted science fiction author, William Gibson, “the future is already here, it is just not widely distributed.”

Let me answer the fundamental question about why both of these data constructs will become important to all enterprises: they represent how decision making information will be captured in the future. Big Data and Sparse Data will become the vehicles for cloud computing to take the next step. You are being teased with it right now when you look at how Google Maps work. The iOS-6 debacle with Apple’s Maps has brought how difficult it is to develop this type of application to everyone’s attention. I’m not siding with either of these implementations; just recognize that they represent how important this use of Big Data is in providing an advantage in a consumer product area. It is a simple twist to see how those same approaches will change enterprise data.

Both mapping approaches are about Big Data. It isn’t about maps, it is about location, the key data element of a map. Location becomes an extensible notion. The graphical map is an overlay and only important as a representation of information about locations. Just look at the issue with Apple: the locations are not right, so the maps don’t generate correctly. How do they fix that? They add more data using locations. They make the Big Data bigger.

We, as users, do not need to carry large amounts of map data, just a simple application on our devices that can leverage the massive store of data organized by location that Google or Apple have built for us. That is the metaphor I spoke of earlier. Just because your business does not generate data that can be categorized as Big Data or Sparse Data doesn’t mean you won’t find a use or need for it. There will be providers of access to Big and Sparse Data who will tease out the unifying elements underneath the data and give you simple access to it. These could be tools that help you model your asset investments, provide real end-to-end performance at the transaction level, plan travel and logistics real-time at a corporate level to take greatest advantage of hotel and airline savings, or give you a voice recognition system that will access all of your internal documentation from a simple, Siri-like command.

It is not too early to start thinking about this next generation of data store. The implications to mobile device management, end user interfaces and cloud computing are immense. This is one of those game changing moments that will be upon us and we won’t even know it. Is your strategy ready?

Jerry Gentry is a research analyst for Nemertes Research. To get more useful data center management strategies and insight from Nemertes Research download the Q2 Data Center Knowledge Guide to Enterprise Data Center Strategies – Volume 2.