How Etsy Optimized Data Center Infrastructure for Efficiency

Etsy, which recently held a successful IPO, is a growing marketplace of diverse, unique, and handmade goods. The company's operations philosophy is centered around what it calls "designated ops," which emphasizes deep collaboration and community not just across developers and operations, but across the entire company.

“It’s a mistake to think of these topics as domain expertise of one particular group," said Mike Rembetsy, Etsy’s vice president of technical operations. "These topics shouldn’t be -- though they are historically -- the realm of operations engineers.”

Rembetsy stressed that true efficiency is dependent on the human element, the communication between people. For the data center to run like a well-oiled machine, people need to work with the bigger interest in mind.

Etsy's designated ops approach is not to be confused with dedicated ops. Like DevOps, it stresses collaboration, but that collaboration extends out further than between only development and operations.

In designated ops, someone within the operations team will embed themselves into other teams. “The job of that person is to act like a member of that team,” said Rembetsy. “Any changes happening, their job is to help facilitate an operational mindset within the process of development. They’re not there to configure all the network checks etc., they’re there to help cultivate the knowledge back into the development process to help foster the collaboration."

This approach means if help is needed in one domain, there’s a bridge that’s facilitated to allow teams to collaborate, he said. “It’s pretty impressive to sit back and watch. I’m super proud of it”

Even the public-facing client side of Etsy needs to know a lot more about issues like networking resource constraints, and they need to know more than the average worker, said John Allspaw, senior vice president of technical operations at the company.

Etsy's Data Centers

As with any successful web property, it’s always interesting to take a peak inside the company's data centers, which Etsy operates with an emphasis on sustainability.

“It’s not just a data center value, it’s a company-wide value,” Allspaw said about the sustainability focus.

The company works on maximizing use of renewable energy with its data center providers Sabey and Equinix. “We want to provide renewable energy as well as onsite generation,” said Allspaw. “The difficulty with that is multi-tenant data centers are focused on reliability.”

Because reliability is goal number-one for the multi-tenant data center, often these interests aren’t fully aligned with renewable energy use.

The team has been talking with providers about onsite renewables, and it looks at Power Purchase Agreements if local sourcing isn’t possible. “It’s a challenge to find a provider that can provide a data center level of power consumption that meets requirements,” said Allspaw.

Equinix was Etsy's first data center provider (since 2011), and more recently it has added a Seattle location with Sabey.

The Equinix data center in San Jose, California, (SV5) was selected partially because it used a newer cooling technology that used less water. “Since then, when choosing another data center, we’ve continued on the idea of efficiencies, renewable energy, and design,” said Rembetsy. “The reason we do choose providers are values based on longer-term renewability.”

Efficiency at the box Level

Etsy began making their racks as dense as possible in 2011, going from 2kW per cabinet to 5-6kW per cab. The first step was to use Supermicro servers “They seemed to have the most efficient power supplies and we were running a relatively dense configuration,” said Rembetsy. Since then, it has diversified its vendors.

To get to its current densities, the company credits playing around with MapReduce and Hadoop several years ago. “Hadoop gave us a crash course in how to make a dense rack,” said Rembetsy.

The company performed big data jobs early on in the cloud, using Amazon's Elastic Map Reduce service.

“That [EMR] got us to a place where we think about what data we collect. At the time it was really inefficient from a cost perspective, but there weren’t any creative constraints,” said Rembetsy. “It was quite trivial to write Hadoop jobs that were inefficient.”

The team eventually decided to bring it in house. “Our Hadoop cluster wields a huge amount of flexibility," said Rembetsy. "We can get a lot done on bare metal, and we know very much what we’re looking for. Cloud services, that’s where efficiencies fall away. Spin up and spin down doesn’t have the same value proposition.”

Hadoop clusters tend to use a lot of power, so the company again looked for efficiency in servers.

Hadoop servers (Etsy uses Supermicro boxes to run Hadoop) were nice and cool in the front, but the back was pretty hot. Etsy employed cold-aisle containment to help airflow and reduce the number of fans.

Rembetsy and his team took a look at the CPUs itself and were able to switch from 145w top-of-the-line processors to 90w CPUs. “We looked at the wattages versus performance from a power efficiency perspective,” said Rembetsy.

“Efficiencies are going to come in different forms or shapes,” said Allspaw. “There are a couple of scenarios that are not entirely intuitive choices: the one that comes to mind is diversifying our CDN. Moving from one to three allowed us to drive further efficiency, but using three CDNs running simultaneously meant we had to make our code work for it. We pulled the cleverness out so we could treat CDN the same as any other.”