Big Data's Past Is Messing With Our Future

(Bloomberg Opinion) -- There is an inconvenient open secret that lurks behind many decisions we make every day. We live in a data-empowered era. When we plan a trip, Google estimates our travel time and recommends routes based on both current and past traffic data. Netflix suggests movies and shows we might like based on its data from people with similar taste. The Federal Reserve uses market and macroeconomic data to make projections about inflation and what it should do about interest rates.

The open secret is that data makes us slaves to the past. And when the world changes, data no longer tells us as much about the future, and we can end up making worse decisions. Coming out of the pandemic has caused significant chaos and changed many things about our lives. How it affects the data we use to make future decisions will matter for years to come.

To some extent, we have always relied on data to predict the future. Even when we were fully analogue, our past experiences — our personal “data” — drove how we perceived risk and made choices. But during the last century, computing power meant everyone’s past experience, in the form of large data sets, started influencing our lives. This was especially true in finance, which took securities prices from markets to make projections about where prices would go in the future and how to hedge or insure against risk.  

Related:The Challenges and Opportunities of Big Data in Cybersecurity

Big data big problems

Data meant we could make decisions more scientifically, based on many more experiences than just our own. It offered great promise and more accuracy. Many aspects of our lives became more predictable and efficient. We not only were told the best route for our road trip, our computer knew which shoes we wanted to buy before we did. Companies could better predict demand and spend less maintaining inventories, and airlines could sell more seats because they had a better sense of who would show up.

But this process was always less scientific than we told ourselves.

About 10 years ago, I was tasked with projecting future interest rates for a pension risk model. This problem raised many challenges. Normally in finance we take data from the past; when you look at a fund prospectus it tells you its past average return and volatility. But by 2011, interest rates had been falling for decades, and odds were such a period wouldn’t happen again since a repeat would mean rates would go very negative. Economists normally assume rates go back to their historical mean, but there was no sign that was going to happen anytime soon either. The world had changed, it was more global and there was more demand for US bonds, which drove rates down.

Related:The Lessons of Moneyball for Big Data Analysis

All of these concerns made me question if I could use historical rate data, and if so, what history? If I started in the 1980s, I’d bake into my model a bull bond market that would probably not happen again. If I used data going back to the 1970s, I’d assume rates would go to a higher level – that probably won’t happen, either. In the end, after much debate with senior leadership, we decided to use as much history as possible and worked out a process to update with new data frequently. It was the more conservative choice, but it was a hard call.

A living, breathing data stream

Data isn't a replacement for wisdom. Without good judgment, data will fail you because it ties you to the past. Assuming housing prices wouldn’t fall, simply because they hadn’t in a long while, was a big contributor to the financial crisis. More recently, poor data judgment was one reason why the Fed was so certain inflation was transitory. Fed economists use mean-reversion models that, with the last 40 years of data, predicted inflation would quickly go away on its own. Changing data is one reason why Google searches aren’t as useful as they used to be.

Today, the practice of letting data guide us has become a bigger danger because our lives have become more data-driven. Our online lives generate more data than ever before. And greater computing power and technology enabled use of that data. Suddenly every trip we took, everything we bought and every website we visited added to data’s predictive power and guided our decisions — even if it was all a little creepy.

But even big powerful data is from the past. And now the world has changed and all that data isn’t so predictive anymore. Inflation is back and changing how and what we buy. Interest rates are going up, changing how we save, invest and buy homes. And it’s not just the economy. Traffic patterns from 2019 — when people went to work five days a week — are no longer as useful, nor are projections on what clothes we will need. We watch more movies at home and eat out earlier. Leisure travel is up, but there is still less business travel, which makes it harder for airlines to know how many seats to sell.

Today’s economy is chaotic for many reasons — primarily supply and labor shortages — so it’s hard to know how much our data-powered algorithms are failing us. If the world has truly changed, it may take years to have meaningful data to power our decisions again. Or, if it goes back to normal, which pandemic years should we use going forward?

It’s unusual for the world to change so much in such a short amount of time, but it offers a valuable lesson that we will probably forget: Data is only a guide, and never a replacement for our own judgment. Going forward, we’ll have to take all the predictions, big and small, with some skepticism and balance them with our own personal experience.