Splunk Brings Machine Learning to Data Analytics

The developers at the data analytics company Splunk have been keeping busy. On Tuesday the company announced expanded machine learning capabilities across its entire product line, with new releases of five products. The timing of the releases might have something to do with the fact that they were announced at .conf2017 in Washington, D.C. -- otherwise known as the 8th Annual Splunk Conference -- because that's the way the tech world rolls.

Splunk is a San Francisco based software company that's been around since 2003. It's stock-in-trade is software for searching, monitoring, and analyzing machine-generated big data by way of a Web-style interface. Simply put, it makes sense of copious amounts of data in ways that are understandable to mere mortals. That was its purpose from day one. It's flagship product, Splunk Enterprise, was originally designed to analyze big data and generate reports for executives who don't live in an IT environment -- and that remains its primary purpose today.

The software can import data from a variety of user determined sources, from simple logs generated by operating systems or Apache servers, to reams of data collected by big data applications such as Hadoop (with the help of Splunk Hadoop Connect). With machine data growing at a rate that's 50 times faster than the growth of traditional business data, a hefty dosage of machine learning comes in handy.

"Machine learning is critical to customer success and to the evolution of Splunk," said Richard Campione, Splunk's chief product officer, in a statement. "Our seamlessly integrated capabilities open up machine learning to everyone, enabling our customers to better predict future outcomes and more effectively analyze their data.

"Data is a strategic advantage and organizations are looking for the fastest, most efficient way to turn data into answers. With machine learning and metrics advancements that anyone can use, Splunk Enterprise 7.0 and Splunk Cloud powerfully deliver mission-critical answers faster and easier than ever before."

Splunk Cloud is basically the same thing as the company's Enterprise offering, only hosted in the cloud and offered as software-as-a-service.

In the new Splunk Enterprise 7.0, enhanced metrics have accelerated the speed of monitoring and alerting somewhere between 20X and 100X, and optimizations to its core search technology deliver a 3X improvement.

"In fact, the more data that you throw at it, the faster and better return you'll see," said Stephen Luedtke, a technical marketing manager at Splunk, speaking in a company video.

The new release also brings something Splunk's calling "event annotation" to the table, a "little" thing that could end up saving users a lot of time.

"With event annotation, we're allowing you to seamlessly correlate and unify your logs, metrics and annotations all into a single view," Luedtke explained. "For example, before [the new release] you might have had a chart showing a time series graph and you might have asked yourself, 'What caused the dip; what caused the spike?'

"You could have drilled down into that data and tried to correlate your different sources to figure out that answer. Now, with event annotation, we can correlate and unify those disparate data sources and overlay them right on top of your time series charts."

He said the annotation might indicate that a spike was perhaps caused by an error, or that a dip was caused by a crash.

"If you take the latest version of our Machine Learning Toolkit and combine it with the Splunk platform, you now have a one-stop-shop to collect data, search, analyze, clean it and explore it; build your machine learning models; alert on, act on and deploy those results; and finally, share and visualize those results with your organization," he said.

Splunk also announced new machine learning capabilities for its other existing premium-packaged solutions, including:

Splunk IT Service Intelligence 3.0: This new version "revolutionizes" event monitoring by combining service context with machine learning to help identify existing and potential issues, prioritize restoration of business-critical services and deliver analytics-driven IT operations. It applies service context, including dependencies, to events and employs machine learning to reduce the noise of alert fatigue by surfacing only the most critical information.

Splunk User Behavior Analytics 4.0: The new version enables users to create and load their own machine learning models to identify custom anomalies and threats via Splunk UBA’s new SDK. This allows users to detect insider attacks and automate correlation of anomalous behavior into high fidelity threats.

Machine Learning Toolkit: This free toolkit is an application that can be used to predict future IT, security and business outcomes. Recent updates include machine learning model management, which integrates user permissions using an intuitive user interface. In addition, the toolkit now includes public machine learning APIs for open source and proprietary algorithms, and a data prep module to help users prepare and clean their data before initiating machine learning modeling.

Splunk indicated it will be previewing two new technologies-in-the-works at this year's conference. Project Waitomo is an infrastructure monitoring solution that unifies logs and metrics, delivering integrated machine learning for alerts, trends and investigation. Project Nova is an API-based logging-as-a-service solution, targeting developers and DevOps.