How One of the World’s Largest Games Was Crippled by the Server Shortage

Final Fantasy XIV fans rejoiced last week when Square Enix opened up a data center in the Oceania region that is intended to take some of the load off the company’s other data centers, meaning users can once again buy the game.

In December, the title – which counts 25 million registered players – experienced a surge in popularity with the launch of its latest expansion, called Endwalker.

Anecdotally, the massively multiplayer role-playing game offered the sort of escapism that proved especially alluring amid the latest wave of Omicron infections and the accompanying lockdown and social distancing measures.

Excitement for a new Final Fantasy entry is not unusual, but Square Enix underestimated just how high the demand on its servers was going to be. The launch broke the game, overloading the company’s systems and resulting in hour-long queues for countless customers.

After a week, one of Japan’s largest and most successful game developers took a highly controversial step: it stopped the sales of the game to anyone who wasn’t an existing subscriber, until it could fix its IT infrastructure woes.

“Players are currently experiencing extremely long wait times due to the dense concentration of play hours which far exceed our server capacity, especially during the peak times, and so we have decided to temporarily suspend the sale and delivery of Final Fantasy XIV Starter Edition and Complete Edition,” Naoki Yoshida, the game's producer and director, posted at the time.

In the following weeks, the company admitted that it couldn’t find enough servers on a short notice to deal with the overload – having long preferred a proprietary and custom approach to its hardware, unlike some of its competitors that went the pubic cloud route.

Last week, Square finally resumed sales of the game, but its infrastructure expansion plans will take months – and it remains to be seen if the same numbers of users will ever come back.

“The Oceanian data center has been added, and can now be selected from the Data Center menu via the title screen,” the company announced in the patch notes for Version 6.08, which was released on January 25.

This is not a brand new facility brought about by the Endwalker debacle; the company announced it was coming to the region in November, but previously said that new servers would be available by February. It beat that deadline by two weeks.

Tech trouble in Eorzea

The data centers in Final Fantasy XIV aren’t the kind we normally write about. Instead, they are logical data centers and are intended primarily to allow users to pick a “server,” which hosts a “World,” or a virtual neighborhood to play in. The physical data center in Oceania, with the logical data center name Materia, houses five servers: Bismarck, Ravana, Sephirot, Sophia, and Zurvan.

Where exactly its physical data centers are – you know, the kind we normally write about – Square Enix isn’t saying. “While we are aware that many of you are curious as to the actual physical location of the new data center, please understand that for security reasons specific to the operation of an online game, we are unable to divulge that information to the public,” Yoshida posted in 2017, when discussing the US operation. “What we can tell you is that the servers will remain on North American soil, in an area that features an advanced Internet infrastructure.”

The consensus in online forums is that players in North America are being served from Sacramento, California, with servers having been located in Montreal until 2017. Square Enix confirmed in the patch notes that the Oceania data center was indeed a new physical facility located in the region.

The physical location of the data centers is important because ping times can be a critical factor in the game.

In December, Square Enix revealed that it was having problems procuring the hardware it needed for expansion due to the global shortage in computer chips.

“We need tens of ‘server machines’ for every World that we add,” Yoshida posted. “However, due to the COVID-19 countermeasures currently in place, many factories across the globe which produce semiconductors have halted production, or have faced labor shortages. This has ultimately led to a decrease in the number of semiconductors being produced, and has resulted in a worldwide semiconductor shortage.”

“We have made considerable investment—even more so than usual—to secure the required hardware, but even so, a long lead time will be needed to prepare the server hardware.”

How the new data center will help

The new data center in Oceania will help with load balancing by moving users in Australia and New Zealand off the company’s facilities in in Japan, North America, and Europe, along with any other users who want to check out the new turf.

To encourage this transfer process, Square Enix is not only waiving the usual fee for Home World Transferring, but rewarded players who transferred with in-game goodies. Players creating new characters on machines in Oceania will also receive bonuses.

Square Enix didn’t say how long the incentives would last but indicated it would depend on the balancing process. “These incentives will remain in place until the active player populations of the Worlds in question have grown to a specified number,” the company said in the patch notes. “Please note that once a World has been removed from the ‘Preferred World’ category, the various bonuses will no longer be applied.”

Free Trial registrations are still suspended for the time being.

Further load balancing improvements

When Square Enix announced Oceania’s availability on January 14, it also said it planned to expand its physical data centers later in the year, but noted that the global chip supply crisis is likely to continue to affect its operations.

“Currently, the North American Data Center houses three logical data centers and 24 Worlds,” Yoshida posted. “Due to the global shortage of semiconductors, it has taken considerable time to procure server equipment, but the schedule calls for the first phase to be implemented around August 2022, with the second phase to follow in the spring or summer of 2023.

“In the first phase of the expansion, scheduled for August 2022, one new logical data center will be established and four new Worlds will be added under it. In the second phase of the expansion, we plan to add four more new Worlds under the newly established logical DC.”

The European data center is also slated for expansion, Yoshida said. “Currently, the European data center comprises two data centers and 12 Worlds,” he posted. “In order to accommodate the substantial increase in player population, we are planning a major server expansion in two phases, similar to that planned for the North American data center. Due to the ongoing global semiconductor shortage, it will take time for requisite server equipment to be delivered. As such, the first phase of this expansion is scheduled for July 2022, while the second is scheduled for the summer of 2023.”

In July, the company also intends to reconfigure its Japanese data centers: “We are looking to add one more logical data center and lower the volume of Worlds per logical data center, thereby significantly increasing the maximum number of concurrent logins possible by several thousand per World,” Yoshida reported. “We believe that—taking into account the floor space of the data center where the servers are physically housed and the current semiconductor shortage—this will be the fastest and most efficient method for large-scale expansion.”