Study Identifies Common Pain Points in Big Data Projects

While more than four out of five companies are taking on Big Data projects this year, almost all (92 percent) of them are still seeing obstacles in their implementation, according to a global study from CA Technologies.

There is significant spending occurring around Big Data, this year being a major tipping point. However, several pain points are being uncovered. Despite these pain points, almost 90 percent of organizations are seeing or anticipating benefits of increased revenue from these projects.

Companies are using Big Data analytics to better understand customers. Organizations are reporting or anticipating increased revenue thanks to improved competitive positioning, ability to provide new products or services and more effective marketing campaigns – all were reported by over 90 percent of respondents noting a positive effect.

The biggest obstacle is around insufficient existing infrastructure (at over 30 percent of respondents) followed by organizational complexity, security and compliance concerns, lack of budget and resources, and lack of visibility into information and processes. All five major obstacles are pain points that Big Data platform providers have attempted to address in recent updates.

A workaround to insufficient existing infrastructure is the use of cloud. MapR and Hortonworks both updated their Hadoop platforms to better handle distributed clusters across any cloud infrastructure. However, with distributed infrastructure, security and compliance becomes complicated, leading to necessary enhancements on this front.

Organizational complexity and visibility issues are being addressed through better centralized administration, improved user interfaces, and customizable views that extend data insight beyond data scientists.

Spending on these projects is expected to increase from 18 to 25 percent over the next three years, according to the study. The amount of data organizations have to deal with is also increasing significantly, by an average of 16 percent in the last two years and predicted to rise almost 25 percent in the next two years. Almost all respondents acknowledged that major investments are required for Big Data projects to work well and more than half seeing scaling existing projects to address more data sources as a major priority.