Etsy Owes Data Center Efficiency to Facebook's Open Source VM

Etsy attributes recent accomplishments in data center efficiency in large part to using the Facebook-developed, open source HipHop Virtual Machine (HHVM), according to software engineer Dan Miller on Etsy’s blog. Etsy is an extremely popular marketplace for handmade, vintage and unique items. The company is expected to IPO soon.

Last year, Etsy’s infrastructure group took on a challenge to scale Etsy’s API traffic by 20 times. HHVM showed a promising increase in throughput, so the custom goods giant migrated over with fantastic results. It did so by fixing an API traffic problem and at the same time keeping both web and mobile platforms in-line in terms of feature releases.

Some other results include a 50 milliseconds drop in homepage median backend time for Etsy, and a 20 percent drop in overall CPU usage in the company’s API cluster with even more gains expected going forward.

HHVM has garnered a lot of interest because it is blazingly fast and efficient. The open-source virtual machine is designed for executing programs written in PHP and uses a just-in-time (JIT) compilation approach to achieve superior performance and flexibility in development.

The big issue in the past has been perceived stability, but the project has matured considerably and more use cases are emerging. Etsy and Box are two highly visible examples of using HHVM for API traffic.

"What we’re seeing, as businesses mature, is they very often build initial APIs on different pieces of hardware and software," said WP Engine's Tomas Puig. "Now, they're rebuilding with HHVM because of performance."

WP Engine provides a Wordpress-based content management platform and had HHVM out in production early. Puig discussed HHVM after it hit a big stability milestone of Facebook announcing availability of HHVM for PHP. Etsy noted that the company’s interest in HHVM really piqued at that time.

Through using HHVM, Etsy saw an increase in performance, which meant not needing more servers or data center space.

"HHVM is good at making smaller server builds perform and for taking huge data centers and making them incredibly efficient," said Puig. "For us, it's about performance for smaller buildouts. However, Facebook originally built it for tens of thousands of servers - imagine the infrastructure savings at that scale."

Etsy worked closely with Facebook, Baidu, Box (which also moved API traffic to HHVM) and Wikipedia throughout the process, according to an Etsy company release.

Before undertaking such a change in architecture, the engineering team kicked the tires on HHVM. After it revealed promise initially, Etsy tested HHVM with internal traffic. Through testing and making adjustments, the company’s confidence in HHVM grew.

However, the engineers faced two major problems regarding mobile apps, both revolving around APIs.

The first resulted in having to rewrite logic designed for web context to be executed in an API context, which borked the progression of web versus mobile platforms. The next involved API endpoints due to a big increase in API traffic.

“The second problem was how tempting it became for engineers to build lots of general API endpoints that could be called from many different mobile views,” wrote Dan Miller on Etsy’s Code As Craft blog. "If you use too many of these endpoints to generate a single view on mobile, you end up degrading that view’s performance.”

"Bespoke endpoints” fixed the problem because they aggregate small, reusable, cacheable REST endpoints. However, the fix created other issues. The way bespoke endpoints worked meant web pages had the potential to hit dozens of API endpoints. Etsy provides a lot of detail, but the end result was traffic on the API cluster grew more quickly than anticipated, as well as other problems. It was a case of "you fix something here, something breaks over there."

HHVM solved that problem by speeding up all requests. HHVM crushed it on performance in a variety of tests and proved to be "as rock-solid as a language runtime," according to Etsy in a statement.

“Following months of iterations, improvements and bug fixes, HHVM now serves all of the fan-out requests for our bespoke endpoints,” wrote Miller. “We used a variety of experiments to gain confidence in HHVM and to discover any bugs prior to deploying it in production.”

Etsy only encountered one language incompatibility and ran into a few problems with HHVM extensions.While WP Engine's Puig believes the project is mature, he said this was an example of "the high price of knowledge entry" in using HHVM.

Different applications will perform differently on HHVM. However, expect a wider range of uses coming out of the woodwork, with sizable ones revealing data center efficiency gains. For Etsy, HHVM adds a major production use case to its armory.

Facebook’s influence on the modern data center stretches from facility to hardware down to virtual machine. HHVM is another promising open source initiative born out Facebook that it has continued to support.

"I'm impressed at how Facebook has handled the open source ecosystem," said Puig. "They've really doubled down on projects. Paul [Tarjan] and the guys on the open source team are doing an amazing job of consistently releasing source."