Using Data Science to Spot Data Center Failure Before It Happens

Critical Thinking is a weekly column on innovation in data center infrastructure design and management. More about the column and the author here.

There’s an inherent irony at the heart of digital infrastructure. As we have highlighted in this column before, many of the processes and procedures required to keep the lights on in data centers are still not digitized.

Despite the hype around IoT and industrial internet, management of where the digital rubber meets the physical road – power, cooling, physical security – is still built on spreadsheets or even in some cases clipboards of hard-copy checklists.

True, building management systems (BMS) and data center infrastructure management tools (DCIM software) have helped automate monitoring and even some control tasks. But a lot of routine checks and maintenance is still rooted in the same manual processes that were standard practice a decade or more ago.

While those processes work, they are often labor-intensive and costly. There is wasted effort from routine and rote maintenance checks. Also, the less standardized a process is, the more potential for human error – still one of the main contributors to data center downtime.

The more intelligence operators have about their own sites and equipment, the less reliant they are on equipment suppliers or third-party facilities services providers. Knowledge is power when it comes to power and cooling.

Related:This Startup Wants to Do to Data Center Power What VMware Did to Servers

As we have also described, cloud-based monitoring tools that can provide features such as predictive maintenance – so-called data center management as a service (DMaaS) – could help to bring more intelligence, but that space is still very nascent. It also potentially means ceding more control to equipment makers.

There is another class of tools known as Computerized Maintenance and Management software (CMMS) that can help automate some of the manual tasks associated with maintaining facilities equipment. However, adoption is still sporadic, and levels of technical sophistication and functionality vary between different suppliers.

MCIM, developed by software developer Fulcrum Collaborations, is at the upper end of the CMMS technical sophistication curve. In fact, its makers don’t actually classify it as a CMMS tool, referring to it as a “cloud-based platform to centralize, standardize, and automate Facilities Management (FM) operations.”

The MCIM platform is accessible on mobile devices and can provide intelligence across multiple sites. It is also built on top of Salesforce – as are other Fulcrum products. The company says it supports more than 15,000 sites, including Fortune 100 firms, either directly or through FM services partners. Fulcrum believes the total addressable market for its technology is approximately $1 billion -- similar to that for CMMS and DCIM.

Related:A Smart Assistant for Data Center Cooling?

Data Center Knowledge spoke with Michael Dongieux, founder and chief executive of Fulcrum Collaborations, to find out what makes MCIM different and why data science matters in the FM world. Here's our interview:

DCK: What was the genesis behind MCIM?  

Michael Dongieux: I used to be the lead technology project manager for Emcor Facilities Services on behalf of Wachovia/Wells Fargo. Every month we were trying to aggregate data from these incident reports across a portfolio of 70 sites. The typical weapon of choice for that was Excel. We had all of these different macros. It wasn’t that we didn’t know how to steer things to get that data; the actual administration of that was a nightmare.

Didn’t you use DCIM or CMMS tools to automate some of those tasks?

Even CMMS systems weren’t very good at relating data to make it intuitive for the user. The problem is if some data is not captured in the moment correctly -- when the operator is entering it -- it is lost forever. There is only so much you can do to patch the holes after the fact.

So what makes MCIM different from CMMS or DCIM tools?

Our eureka moment was the operators in the field doing rounds who are filling out incident reports, who are filling out change reports, who were dong their CMMS work orders, the knowledge we needed was in their head. So the trick is to get it out their head but do it cleanly right when they are capturing it.

Facilities management teams are typically quite conservative. They need a good reason to invest time in new software. What’s the unique selling proposition for MCIM? 

We are enabling data science. You can’t do data science on an Excel spreadsheet – not at scale anyway – and you certainly can’t do it with data on clipboards. If you really want to get into data science, then you really have to have a clear-cut understanding of the relationships between your different pieces of data. And in our system, everything is related. But we are able to achieve that without burdening them to fill out all sorts of stuff that they wouldn’t normally fill out.

But how can you use that data to practically help operators?

When someone logs an incident report, they are able to associate every asset or assets that were involved in the incident and then say what the source of that failure was. That information is crowdsourced and clustered automatically. That enables us to correlate not only what the asset condition index, or ACI, score is of a particular piece of equipment, but we can also say for example that at 85 percent of their useful life, centrifugal chillers typically start to see an increasing occurrence of a specific kind of failure.

So who are your main target customers now?

We are definitely selling direct, but we are also selling it to the FM service providers. We are not selling it to the equipment providers right now, and the reason for that is that I think that somewhat presents a conflict of interests, when we are presenting performance data, failure data on the equipment, when we are working closely with the provider of the equipment.

Longer term it seems that as suppliers add sophistication to their equipment – through IoT and even DMaaS – CMMS tools as well as the functionality offered by MCIM might be eventually outmoded?

To me a lot of that [DMaaS] is falling into the machine-generated data component, and while it is very much part and parcel of the maintenance schedule, etc., there is still going to be a human-interaction piece of it. Every facility manager wants to enforce global standardization and governance. They want to be able to drive analytics and business intelligence. They want to know how much it costs to operate different pieces of equipment, what’s the cost of maintenance, and all those kinds of things, and what are the opportunities for improvement. And the opportunities for improvement don’t just exist on the machine side. At least for now you are going to need people to interact with equipment. Until we are all replaced by robots, you are going to have maintenance done by individuals in the field.