Businesses and the Big Data Skills Shortage

Rick Delgado is an enterprise tech commentator and writer.

Although Big Data is no longer a foreign concept to businesses they still face significant challenges when implementing it into their organizations. And the problem doesn't stem from understanding its importance. Rather, it lies in finding the right people for the job.

Addressing the shortage

There are many reasons the Big Data skills shortage exists. Put plainly, there are too few true data scientists out there and that problem is only expected to grow if no action is taken. McKinsey & Co. has placed the shortfall of Big Data experts at anywhere from 140,000 to 190,000 by the year 2018.

The technical and IT experts that companies have now, while perfectly capable and intelligent individuals, don’t currently have the skill sets needed to handle the Big Data revolution. For most of their career they've been trained to use more traditional databases, training that has a difficult time being applied to new Big Data techniques.

Without the needed Big Data skills, plenty of jobs go unfilled. Some of the most in-demand job openings are for NoSQL experts (people that have experience with unstructured data systems), Apache Hadoop and Python experts, ETL developers, data warehouse appliance specialists, predictive analytics developers, and information architects.

Confronting the challenge

Companies have responded to the Big Data skill shortage in a number of ways. First, they've identified skills current and future employees need in order to meet the challenges Big Data presents. Some are fairly obvious such as computing and analytical skills. Others are geared more toward a specific expertise in Big Data, particularly being able to understand it, collect it, and preserve it. Knowledge of statistics, mathematics, and data visualization techniques including charting, mapping, and graphing are also needed.

With these skills in mind, companies can narrow their focus in the search for the right candidates, but even then they might find few who match the criteria. That’s when companies look for different but related skills by casting a wider net and looking at people with backgrounds in astrophysics or computational chemistry. While they might not have the exact skills, they have enough of an analytical background where they can be trained to become a data scientist.

Providing the tools for success

Training is another area where companies are working to improve in order to bring up a new generation of Big Data experts. Some companies are looking at taking employees already experienced with relational database management systems (RDBMS) and training them to use big data platforms like Hadoop. Using some of the latest technology, companies can even use some of the techniques related to RDBMS and allow them to be used with Big Data platforms. By training those already with the company, businesses get to keep valuable team members that already have experience with the enterprise while giving them some much-needed skills. Plus, those with skills in SQL or virtualization technology may come up with their own unique solutions to Big Data problems.

Take the team approach

Companies have also responded to the Big Data skills shortage by taking the team approach. A data science team combines people of various areas of expertise and sets them loose on mastering Big Data and making it work for the business. The team can include analytics managers, algorithm scientists, and project managers, each bringing their own set of skills to the table. Together, they can make up for any Big Data skills shortage present in the company.

It starts in the classroom

Perhaps most important of all, businesses are getting help with this skills shortage from universities. Years ago, most colleges weren’t equipped to prepare students with the skills needed to get jobs as data scientists, but that is quickly changing. Many universities all across the world are adding courses and postgraduate degrees that teach students what they need to know to land a career as a Big Data professional.

For example, a university in Australia is offering a Master’s Degree in Data Science. Stateside, Columbia is providing students with its Institute for Data Sciences and Harvard has the Institute for Applied Computational Science, each providing postgraduate masters programs around Big Data. There are also schools with masters programs for business analytics: New York University has an MBA degree with a specialization in analytics and information management, North Carolina State and Drexel have business analytics programs that last only a year, and there’s a business and Big Data program at the University of Tennessee.

These are all different approaches to meeting the demands Big Data is placing on businesses. The benefits can be immense, which is why companies are willing to invest so much in making sure their employees are equipped to handle the challenges. Over time, the hope is that the Big Data skills shortage will shrink until supply meets up with demand.

Industry Perspectives is a content channel at Data Center Knowledge highlighting thought leadership in the data center arena. See our guidelines and submission process for information on participating. View previously published Industry Perspectives in our Knowledge Library.