AOL Launches New 'Lights Out' Data Center

AOL has begun operations at a new data center that will be completely unmanned, with all monitoring and management being handled remotely. The new "lights out" facility is part of a broader updating of AOL's infrastructure that leverages virtualization and modular design to rapidly deploy and manage server capacity.

The new approach to infrastructure is driven by AOL's focus on content, according to Mike Manos, AOL Senior Vice President of Technology Operations. As the company builds its portfolio of blogs and web sites - which includes The Huffington Post as well as blogs like Engadget - it is adopting a flexible infrastructure that can deploy server capacity quickly in locations with strong user populations.

The new data center, known as ATC, is a big step in this direction. Manos describes the facility in a blog post, but does not offer details on ATC's size or location.

AOL is using building its data center using standard blocks of infrastructure that can be deployed in increments of 500 kilowatts to 1 megawatt of power. AOL is not currently using containers, as Microsoft did while Manos was heading the company's data center effort, but could do so in the future.

Moving Content Closer to Users

"The beauty of the approach is we can get capacity up any where quickly," said Manos. "Keep in mind we are moving the company to be a content player – you don’t need big facilities, but you do need coverage where the readers are."

Deployment requirements shift when you don't have to factor staffing into the equation, which is the approach AOL has taken. "ATC is a 100 percent lights out facility," Manos writes. "There are absolutely no employees stationed at the facility full time, contract, or otherwise." AOL isn't the first data center operator to deploy unmanned facilities, but is one of the largest providers to do so.

"Failures and issues on-site go into a planned maintenance mode," said Manos. "We simply move the instances (or create new ones) to other data center facilities and the failed equipment is addressed in a scheduled way using outsourced or vendor partners. This is hugely cost beneficial for us as we can now build SLAs around the service without having to worry about the cost of staffing for ‘something to go wrong’."

Management Software A Key Enabler

Manos said AOL has developed configuration management software that has accelerated the process for deploying new server capacity. "We went from provisioning servers in days, to getting base virtual machines up and running in under 8 seconds," he writes. "We used Open Source products and added our own development glue into our own systems to make all this happen.

"We have been running this technology and approach in our existing larger facilities for a few months to ensure the model works under failure scenarios. We did trial the approach in staffed facilities just in case. After a quarter of proof we took the leap."

Manos also said that AOL has been underappreciated as a leader in Internet infrastructure.

"While it may be fun and 'new' to look at the tens of thousands of machines at Facebook, Google, or Microsoft – it is often overlooked that AOL had tens of thousands of machines (and still does!) and solved many of the same problems years ago," Manos writes. "To be honest it was a personal revelation for me when I joined. There are few companies who have had to grow and operate at this kind of scale and every approach is a bit unique and different. It was an interesting lesson, even for one who had a ton of experience doing something similar in 'Internet Darling' infrastructures."