High Density, Low Budget: Massive Bitcoin Mines Spring Up in Warehouses

This is the first feature in our three-part series on Bitcoin mining infrastructure.

The Bitcoin mining craze is driving the creation of a new breed of computing facilities featuring high-density hardware, low-reliability electrical infrastructure and off-the-shelf enclosures. These “hashing centers” often are built in old warehouses and house servers on shelving from hardware stores like the Home Depot.

It's a low-tech solution, yet these facilities are supporting compute density equivalent to that seen in the largest Facebook or Google server farms. Some Bitcoin mines cool their servers with liquid instead of air.

The sudden emergence of hashing centers reflects the rapid growth of the Bitcoin network, along with the intense focus on building high-powered infrastructure at the cheapest price point possible. As industrial mining operations scale up, they are improvising a new type of infrastructure, customized for rapid changes in hardware and economics.

“It’s all about finding the right cost of power and finding enough shelving,” said Bryan Ballard, CTO for Netsolus, which has built several dedicated Bitcoin facilities for customers. “In traditional data centers, you’re trying to find the right confluence of fiber, power and bandwidth. Our bandwidth is negligible. We’re looking at old steel mills and other sturdy facilities with good power.”

Ballard, whose company hosts more than 3 megawatts of Bitcoin customers and expects to build an additional 20 megawatts of capacity, said miners are building a different breed of facility than the traditional mission-critical enterprise data center.

"Sometimes you hesitate to call these buildings data centers," said Patrick McGinn, a product manager with CoolIT Systems. "They’re really powered shells."

Deploying many megawatts

One of the largest of these Bitcoin mines is rising in northern Sweden, where hardware vendor KnCMiner is packing custom hardware into rows and rows of steel shelving. The company, which makes mining rigs using specialized ASICs (Application Specific Integrated Circuits), has deployed an estimated 5 megawatts of gear and expects to add another 5 megawatts. KnC sees major growth ahead.

“Mining is a billion-dollar industry. Today,” writes KnC Chief Marketing Officer Nanok Bie.

In the U.S., MegaBigPower operates a large Bitcoin hashing center in a former warehouse in central Washington, where it uses Raspberry Pi micro-computers to manage tens of thousands of ASICs, all housed on shelves and cooled with air and household fans. The Raspberry Pi serves as a low-power controller, saving on energy.

Founder Dave Carlson has announced plans to create a franchise network, in which MegaBigPower will provide Bitcoin mining hardware for franchisees who can supply industrial facilities with 1 megawatt to 5 megawatts of power.

Carlson told CoinDesk that the first franchisee, California-based Aquifer, will provide up to 50 megawatts of capacity for MegaBigPower's custom mining rigs.

Gearing up for more growth

Bitcoin miners are scaling up to manage the next phase of expansion for a network that has experienced exponential growth. At the start of 2014, the Bitcoin network had a total compute power of 10 petahashes per second. This hash rate -- a measure of the number of bitcoin calculations that hardware can perform every second -- has since soared to more than 135 petahashes per second.

That equates to huge demand for power capacity to run the network. “We’re talking gigawatts, not megawatts,” said Ravi Iyengar, CEO of CoinTerra, which makes ASIC hardware and also operates a large mining facility. “There’s a whole ecosystem that will emerge.”

Silicon Valley veteran Jeremy Allaire says the Bitcoin network is in the midst of a major transition as it moves out of garages and basements and into dedicated facilities.

"Mining is undergoing a significant evolution from a hobbyist undertaking to an institutional model, with well-funded companies doing this as a business," said Allaire, who is now the CEO of Circle, a venture-backed digital currency startup. "We’re at this institutional scale today. We’ll see investments grow to billions of dollars in coming years. We’ll see the mining pools move from being run by hobbyists to being run by large companies."

Cheap infrastructure = more Bitcoin profit

Those large players have major resources but aren't keen on spending it on redundant infrastructure, such as the UPS units and diesel generators that provide backup power for enterprise data centers. The ongoing arms race in Bitcoin hardware has made mining a low-margin game, with the key criteria being the cost of power and the ability to cool high-density hardware.

"The Bitcoin miners are not interested in reliability, and are really margin-sensitive," said Wes Swenson, CEO of C7 Data Centers. "They're really sensitive to the cost of the colo."

To accommodate Bitcoin customers, C7 has rolled out a new data center design that Swenson describes as "N-0.5" that offers high-density space with lower reliability and no service-level agreement. The facility in Utah is cooled using ambient air, with evaporative cooling available for unusually warm days. The network gear and cooling equipment are on UPS backup, but the individual server power supplies are not. The facility can support high density, and Swenson says it operates at a Power Usage Effectiveness (PUE) of 1.06.

The new design is an addition to C7's traditional enterprise data center design, which features a redundant N+1 power infrastructure.
This tiered approach has helped C7 attract a group of Bitcoin customers using about 5 megawatts of power capacity, with another 5 megawatts slated to come online this fall. "We have more demand than we can build out," said Swenson.

This approach reflects the Bitcoin industry's focus on cost, rather than uptime, according to Josh Zerlan, chief operating officer for Butterfly Labs, which makes Bitcoin mining hardware.

“To build out one of these (warehouse) data centers isn’t as expensive,” he said. "You can put in $10 million instead of $100 million for this kind of facility."

Density is major focus

"Reliability and uptime are not that big a problem," said Zerlan. "You don’t need this huge infrastructure to support four-nine uptime. Density is the big issue. The scale and design of traditional data centers is not comparable to what Bitcoin brings to the market."

Bitcoin mining servers can pack as much as 1 megawatt of equipment into 1,000 square feet of space, according to Eric Doricko, a veteran of Exodus Communications who now helps Bitcoin businesses find data center space. That's a big change from the 8,000 to 12,000 square feet of space for 1 megawatt of traditional IT space.

Swenson noted that Bitcoin mining rigs occupying four rack units of space can use between 1 kilowatt and 2.5 kilowatts of electricity.

That's why many Bitcoin vendors and miners are turning to liquid cooling. CoinTerra is working with CoolIT Systems, which provides cold plates and heat pipe technology to bring liquid cooling directly to the chip.

Liquid cooling supports more density

Ballard says liquid cooling can support power densities of 48 kW to 60 kW per rack. "From a CapEx perspective, (water cooling) is cheaper than air cooling for the same workload," he said. "We’re seeing ROIs are much better with water cooling."

"In the Bitcoin market, (air cooling) is a broken model," said McGinn. CoolIT is seeing strong demand for its water cooling systems from Bitcoin mines seeking "many, many megawatts in very small places," he said.

Other large mining operations are turning to immersion cooling. Infrastructure startup Data Tank Mining announced that it has lined up pre-orders for five modular data center containers that can each support 1.2 megawatts of capacity. Each container will house tanks filled with Novec, a liquid cooling solution created by 3M. Inside each tank, densely-packed boards of ASICs run constantly as they crunch data for creating and tracking Bitcoins. As the chips generate heat, Novec boils off, removing the heat as it changes from liquid to gas. Data Tank Mining is a Hong Kong company which works closely with Allied Control, which has built immersion systems for large Bitcoin mines.

A Bitcoin solution? Or new design standard?

In a business in which uptime is the prime directive, the low-reliability designs of these hashing centers would appear to hold little interest for enterprise users. But some technologists believe these low-reliability designs for Bitcoin infrastructure have broader application. Cloud builders like Microsoft and Google with multiple data centers have shifted reliability to the network, engineering their systems so workloads can be easily moved in the event of a data center outage. Yahoo has toyed with the idea of building data centers with no UPS or generators.

Netsolus' Ballard believes these cheaper designs will see broader use as more companies begin to use their network to route around outages.

"I think this is the future of the data center business," said Ballard. "We’re really seeing where data is headed about 10 years from now. We’ve been approached by several cloud storage companies that are interested in what we’re doing. Bitcoin is able to demonstrate another deployment method that works."