eBay Right-Sizing Infrastructure As Growth Surges

ebay-dean

Dean Nelson, of eBay, describing how he leverages density to support eBay's business of $62 billion in transactions of merchandise (Photo: Colleen Miller).

The infrastructure for eBay is enormous, and getting bigger by the moment. The giant Internet e-commerce company serves up 2 billion page views every day, requiring more than 75 billion daily requests to huge databases housing product information. That traffic is powered by high-density racks packed with up to to 96 servers and requiring 28 kilowatts of power. This Web-scale infrastructure enables eBay to process $62 billion in orders annually.

"We are our own little economy," said Dean Nelson, the Senior Director of Global Foundation Services at eBay.

For that economy to prosper, the company's data center infrastructure needs to become more efficient, even as it continues to scale up to handle more massive workloads - including a flood of new mobile traffic.

"It looks like we will be doubling our power costs in the next five years," Nelson said Tuesday at the Google Data Center Efficiency Summit in Zurich, Switzerland. "The way we have done it in the past cannot be sustained and does not keep us competitive."

Matching Redundancy and Applications

eBay's goal is to match its data center infrastructure to the reliability needs of its applications. By engineering the company's entire infrastructure to the highest levels of redundancy, eBay is among the many companies that have effectively overprovisioned their data centers. By building to a Tier IV standard - the highest level of reliability on the widely-used Uptime Institute tier system - eBay has relied on redundant power and cooling infrastructure that may not be needed.

Nelson said "right-sizing" eBay's infrastructure and matching hardware redundancy to applications could yield lower capital expenditures (CapEx) and operating expenses OpEx) going forward. That doesn't mean fewer data centers, but gradually refining the existing footprint and implementing cutting-edge designs on new space.

"We had a great big Tier IV budget," said Nelson. "But up to 80 percent of our equipment could live in Tier II. Search doesn't need to be in Tier IV. We could cut our CapEx and OpEx in half when we right-size those operations."

Mobile Transactions Accelerate
The payment processing service PayPal is growing 25 percent a year, Nelson said. But the fastest growth is coming from mobile transactions, which have quadrupled in the past two years and will hit $4 billion for 2011. Most of those transactions are coming via the eBay iPad app, Nelson said. "It's changing the way people shop," he said.

To support that growth, eBay has two basic configurations for its data center infrastructure:

"I love density," says Nelson, who said that 28 kilowatts is about as far as eBay can push the density of its racks and still use air cooling. "Liquid cooling is what's next," he said. "The manufacturers have got to solve this (density) problem."

Shift to Modular Design
That focus on density has led eBay to deploy a modular design in an 8,000 square foot data center that eBay is building in Phoenix, which will be able to accommodate up to 12 server-filled containers on its roof. The two-story building will also include raised-floor space on its first floor. The facility will use "hot water cooling" to support the requirement for year-round use of free cooling, despite the temperature in Phoenix, which routinely tops 100 degrees in the summer.

The new design was developed by the team of EDI, AHA Consulting Engineers and Winterstreet Architects, which prevailed in a design contest featuring public request for proposals (RFP) that was conducted through Data Center Pulse.

The RFP process is a key element of eBay's process of right-sizing its infrastructure, built around a "holistic TCO model." Nelson is looking to vendors for innovation to meet the requirements of comprehensive goal-based RFPs.

"When we put out a competitive RFP, that's when we see the creative thinking (from vendors)," Nelson said. "It changes the buying decisions."

eBay has some serious leverage with vendors. Nelson said the company will refresh 50 percent of its servers this year. That type of volume can prompt server makers to think outside the box - and outside previous warranty limits for operating ranges on temperature and humidity.

"The server manufacturers will warranty this equipment at higher temperatures," said Nelson. "Push them on this. You can, and it works."