Outages Halting Trades Span Globe in Ugly Month for Bourses

Viren Vaghela, Michael O'Boyle and Gearoid Reidy (Bloomberg) -- In August of 1994, trading was halted for 40 minutes on the Nasdaq after a squirrel chewed through a power line and the exchange’s backup power system failed to kick in.

Today, the technology that underpins billions of dollars in daily trades has never been more advanced, meaning winning margins can be measured in milliseconds. But even with millions spent on hardened systems, backups, and upgrades, outages still happen.

“There are redundancies but they don’t always work,” Xavier Rolet, former chief executive officer at London Stock Exchange Group Plc, said on Bloomberg TV. “Sometimes you get a trifecta of software issues and hardware failures together with network failures that make it very difficult to recover.”

Markets hit the trifecta this month, with exchanges on three continents around the world going down; Euronext NV was the latest, following the Mexican Bourse and Tokyo Stock Exchange. The Paris-listed operator, which will handle 25% of all European stock trading after it completes its $5 billion purchase of Borsa Italiana, shuttered markets for three hours on Monday morning. It was the biggest outage to hit the exchange operator in two years.

Euronext

While outages typically stem from a software failure, hardware issue or cyber attack, Euronext cited a “middleware” issue for the crash that halted trading in Paris, Brussels, Amsterdam, Lisbon and Dublin.

Related:How One Piece of Hardware Took Down a $6 Trillion Stock Market

“To use an analogy, the pilot used to have a dashboard and a joystick that moved the rudder but now there is a middle bit where the joystick sends instructions into a computer before the rudder gets moved,” said Niki Beattie, founder at Market Structure Partners. “When middleware goes wrong it is like the joystick falling off.”

Euronext upgraded its technology in 2018 to a proprietary platform called Optiq, which it claims on its website has “high reliability” while meeting “regulators’ demands for fewer incidents.” Euronext declined to say how much it spent on the platform. The company has spent 4.6% of its free cash flow on capital expenditures in the past decade, or $81.2 million, according to data compiled by Bloomberg. Mexico’s Bolsa has spent about 2.3%, or $16 million.

Tokyo Stock Exchange

Japan’s market snafu occurred Oct. 1 in a piece of hardware called Device 1 of the Shared Disk Device which stores data used across the servers that makes up the Tokyo Stock Exchange’s trading system known as Arrowhead. It distributes information such as commands and ID/password combinations for terminals that monitor trades.

A component failure in the device’s memory led to an error, which should have triggered a failover -- an automatic switch to a backup. It didn’t, due to a setting error caused by a discrepancy between the system and its manual, one that had lain hidden since 2015.

Japan’s Financial Services Agency plans to conduct a probe of the TSE and may call for administrative action.

Mexico Exchange

None of the redundancies in place helped Mexico recover from the longest blackout in the Bolsa Mexicana de Valores’ recent history, according to current and former exchange officials, who spoke on condition of anonymity because they were not authorized to speak on the matter.

The Bolsa said its 10-hour outage on Friday, Oct. 9 was due to an error at a technology provider. It denied rumors of a cyber attack, but has not provided further details.

An official at BMV told Bloomberg News that a problem with main servers at the exchange’s data center provider, Kio Networks, caused the outage. A spokesperson for Kio, which is backed by the family office of beer heiress Maria Asuncion Aramburuzabala, declined to comment.

In the wake of the halt, the exchange could not fire up its backup system, which had been tested by the local regulator. That raises questions about the soundness of the back up system and regulators’ checks, the people said.

Regulator CNBV did not respond to a request to comment.

“Being down for a day will wake up people of the need to connect to the alternative market, and pressure regulators and the exchange itself,” said Larry Tabb, head of market structure research at Bloomberg Intelligence.