Microsoft's Energy Practices in Quincy Under Fire

msft-quincy-aerial2-470

An aerial view of the two Microsoft data centers in Quincy, Washington.

Is Microsoft reducing its data center energy usage because it's the right thing to do for the environment and local communities, or only to save money? The company's commitment to sustainability was challenged by a report Monday that it was prepared to waste tens of megawatts of electricity at its Quincy, Washington data center in order to reduce a potential fine from the local utility. The incident was first reported by the New York Times on Monday.

Microsoft has invested millions of dollars in designing highly efficient servers and data centers, and has made a company-wide pledge to become carbon neutral. But the details of the December 2011 incident in Quincy suggest that when it came to the politics of power, in at least one instance, finances trumped philosophy.

Microsoft says the incident has prompted it to reassess its approach to working with local utilities, hiring additional staff with experience in utility relations. The company said it also has designed two of its newest data centers to operate without diesel generators, another source of contention in Quincy and a focus of sharp critiques from the Time.

Power Penalties at Issue

The dispute in Quincy is rooted in complex power agreements between Microsoft and the Grant County Power Utility District (PUD), the local electric utility. In late 2011, Microsoft faced a $210,000 fine from the PUD for using less power than it had requested for the year. Because data centers seek large up-front commitments of power capacity, many utility agreements include provisions to compensate the power company if the customer buys less than the amount it has requested. A Yahoo data center in Quincy also faced a penalty from the Grant County PUD, and paid its $94,608 fine.

Microsoft argued that the fine created an "illogical financial incentive" but said it was prepared to consume $70,000 in electricity in a “commercially unproductive” manner to avoid the fine, the Times reported. This apparently would be accomplished by running load banks, devices that generate heat and are usually used to test a data center’s ability to cool racks of densely-populated servers.

Microsoft's position led the PUD to revise its position and waive all but $60,000 of the penalty. But it also indicated that the company was willing to waste energy if the financial math made conservation unprofitable. The dispute conjured up images of Microsoft using load banks to burn off megawatts of electricity, and doing so alongside state-of-the-art servers and data halls that have been optimized to use as little energy as possible.

'Our Goal is to Reduce Consumption'

Christian Belady, the general manager of Data Center Services for Microsoft, says that the incident highlighted a disconnect between Microsoft's values and its negotiating tactics.

“Microsoft’s goal is to reduce energy consumption," said Belady. "The situation in Quincy was an isolated incident and the initial reaction was made without the context of a broader consideration of the implications. We quickly reversed course because it didn't seem like a good outcome to use power we didn't need. Since that time, we have hired additional energy and utility experts to ensure that we put the right incentives in place to continue to minimize energy use and costs and accelerate efficiency gains in our data centers.”

In late 2011 Microsoft hired Brian Janous as Utility Architect, a position that will focus on Microsoft's power sourcing, with a mandate to use less overall power and more renewable energy. Earlier this month Janous disclosed that Microsoft plans to reduce its use of diesel generators as part of its efforts to make Microsoft’s server farms less reliant on the utility grid.

Not all of Microsoft's dealings with Quincy utilities have been contentious. Last year the company teamed with local agencies to retool the local water treatment infrastructure, leasing a multi-million dollar water treatment plant built by Microsoft to the city of Quincy for just $10 a year. The plant will be retrofitted to allow waste water from the local farming community’s food processing plant to be used to cool the Microsoft data center. Local officials say the project will save millions of gallons of potable water for the local community.

There have been other municipal benefits. The presence of data centers from Microsoft, Yahoo, Sabey, Intuit, Dell and Vantage Data Centers has boosted Quincy's property tax revenue from $815,000 in 2005 to a projected $3.6 million this year, the Times reported.

"The article uses what we believe are a few isolated situations to paint a negative picture of the relationship between big data center operators, including Microsoft, and the local community," Janous wrote in a blog post responding to the Times' coverage. "We are proud of our relationship with Quincy.  Microsoft has created over 50 good-paying permanent jobs in Quincy, and hundreds more construction jobs.  Our property taxes have helped fund community services, including an expansion of the local library and a new fire station."

Generator Usage in Two States Draws Scrutiny

Microsoft's operations of its diesel backup generators was also a key focus of Monday's story in the New York Times, which reported that generator usage and diesel emissions had soared last year at data centers in Quincy and Santa Clara, Calif.

Diesel engine exhaust is a regulated pollutant, and can be toxic in high concentrations, so state regulators routinely review permits for generators. The growing number of diesel generators in Quincy generated debate in 2010 when Microsoft applied to add more generators for the second phase of its campus in Quincy.  The state ecology department conducted an evaluation of the health risks from diesel engine exhaust particulates, and found that the Microsoft expansion was not likely to impact public health.

The Tmes story examined how often Microsoft was using the generators in Quincy, and why.

"In 2010, during an expansion of the (Quincy) data center, Microsoft repeatedly ran the center on generator power," writes James Glanz of The Times. "The utility ... said its documents indicated that Microsoft asked to be disconnected from the grid. Several experts said in interviews that they found the episode confusing because data center designs typically anticipate expansions and allow construction to take place without disconnecting from the grid."

The Times noted that Microsoft’s generators operated for a combined total of 3,615 hours in Quincy in 2010, compared to 65 hours for a nearby Yahoo data center. Microsoft said the generator activity was necessary to provision new data center space featuring a modular design it had not previously used.

"In August 2010, in coordination with our local public utility, we had to run our entire data center on generator power for several days so that our new modular data center could be connected to the substation," said Microsoft spokesperson Andrea Platt. "We also saw higher-than-normal generator usage due to the installation, testing and commissioning of new generators for our new modular facility which went live in January 2011."

Platt said Microsoft's 3,615 hours of generator run time in 2010 remained below the 4,256 hours allowed by the company's permit from the state Department of Ecology. In 2011 Microsoft's generator usage in Quincy was reduced to 819 hours, or about 2 hours per generator per month, Platt said.

Infrastructure Upgrades in Santa Clara

The Times also reported that Microsoft's Santa Clara data center was listed as one of the largest diesel polluters in the Bay Area Air Quality Management District for 2008 and 2009. The company said its increased generator usage in that period was caused by a two-year project to upgrade electrical infrastructure at the facility.

"This work required us to disconnect from the power grid and run on generator power at certain times while the electrical components were updated, to ensure the safety of work crews and provide uninterrupted service to our customers," said Platt. "The remediation work was successfully completed in 2009. With the completion of the electrical remediation project, we are no longer in an emergency setting, and no longer using generator power extensively." Platt said that in 2011, the Microsoft Santa Clara data center ran its generators for just 152 hours.

Belady reiterated Microsoft's intention to reduce its reliance on generators because of their carbon impact, and said it has already eliminated diesel backup systems from two of its cloud computing facilities.

“Generators are expensive and costly to maintain and as such our goal is to not use them whenever possible, instead pushing resiliency into the software," said Belady. "We are already doing this in Chicago, where our container bays are not backed up with generators. In addition, our newest phase in our Boydton, Virginia facility is designed without any generators, a reflection of more fully resilient applications running there. We will continue trending in this direction where more and more of our data centers will not use generators."