The World's Highest Data Center

alma-wideanlge

Some of the racks of the ALMA Correlator, one of the most powerful supercomputers in the world, which operates at a remote, high altitude site in the Andes of northern Chile. (Photo Credit: ESO)

High atop the Chajnantor Plateau in the Chilean Andes, at more than 16,500 feet above sea level, a unique supercomputer brings the heavens into focus. Here, at the world's highest data center, sits the ALMA Correlator, a powerful system that allows a system of high-altitude antennas separated by up to 16 kilometers to work together as a single giant telescope.

The ALMA Correlator is a critical component in a radio telescope system that astronomers are already using to make new discoveries about how planets, stars, and galaxies form. Unlike optical telescopes, which observe visible light emitted by stars, ALMA explores a region of the spectrum of invisible light, the millimeter and sub-millimeter wavelength realm.

How do you operate a supercomputer at 16,500 feet above sea level? It takes some extraordinary engineering and some operational adjustments. The air is so thin that twice the normal airflow is necessary to cool the machine, which draws some 140 kilowatts of power. Computer disk drives won't work reliably in thin air, so the correlator and its associated computers must be diskless. Seismic activity is common, so the correlator had to be designed to withstand the vibrations associated with earthquakes.

The thin air high in the Andes also affects human performance. While the ALMA Array Operations Site (AOS) Technical Building is expected to be unmanned during routine operations, it took about 20 weeks of human effort there to unpack and install the correlator. "There are thousands upon thousands of cable connections that we had to make, and every one of our cables is the same color blue, so I’m just glad we devised a good labeling system while at sea-level," said Rich Lacasse of the National Radio Astronomy Observatory (NRAO), leader of the ALMA Correlator Team.

ALMA: $1.3 Billion Project

ALMA is short for Atacama Large Millimeter/sub-millimeter Array, and is operated through an international partnership between North America, Europe, and East Asia in cooperation with the Republic of Chile. The $1.3 billion project was inaugurated last month after several years of construction.

Why build a complex telescope in the mountains? The extreme elevation and low humidity at the site help minimize the interference caused by water vapor in Earth’s atmosphere. The telescope has already provided unprecedented views of the cosmos with only a portion of its full array.

The ALMA correlator’s 134 million processors will continually combine and compare faint celestial “signals” received by as many as 50 dish-shaped antennas in the main ALMA array, enabling the antennas to work together as a single, enormous astronomical telescope. In radio telescope arrays, sensitivity and image quality increase with the number of antennas.

When observing, ALMA’s antennas point at the same celestial object in the sky, gathering faint radio waves. Before astronomers can make detailed images or do other analyses, the information collected by dishes must be extensively processed. The ALMA correlator performs the first critical steps in this data processing. Installation of the correlator was completed in December.

"The completion and installation of the correlator is a huge milestone toward the fulfillment of North America’s share of the international ALMA construction project,"said Mark McKinnon, North American ALMA Project Director at NRAO. "The technical challenges were enormous, and our team pulled it off," he added.

Cutting Cost With Custom Configuration

One of the challenges facing the correlator design team was cost. "As we confronted this project, we realized that it would have taken, at the time, a billion dollars’ worth of off-the-shelf personal computers to perform the needed calculations," said John Webber, former head of the NRAO Central Development Laboratory in Charlottesville, Virginia. "We built our own custom machine for about 11 million dollars."

ALMA’s European partner, led by the European Southern Observatory (ESO), provided 550 “next-generation” circuit boards that increase the system’s ability to distinguish fine shades of color, or wavelength. With these filters, the light which ALMA sees can be split up into 32 times more wavelength ranges than in the initial design, and each of these ranges can be finely tuned.

"This vastly improved flexibility is fantastic" - said Alain Baudry, from the University of Bordeaux, the European ALMA correlator team leader- "it lets us ‘slice and dice’ the spectrum of light that ALMA sees, so we can concentrate on the precise wavelengths needed for a given observation, whether it’s mapping the gas molecules in a star-forming cloud, or searching for some of the most distant galaxies in the Universe."

For a photo gallery of the worlds highest data center, continue to the next page.

alma-blinkinglights

This view shows lights glowing on some of the racks of the correlator in the ALMA Array Operations Site Technical Building. This photograph shows one of four quadrants of the correlator. The full system has four identical quadrants. (Photo Credit: ALMA, S. Argandoña)

alma-circuit-boards

ESO (THe European Southern Observatory) provided a key part of the correlator: a set of 550 state-of-the-art digital filter circuit boards was designed and built for ESO by the University of Bordeaux in France. With these Tunable Filter Banks, the light which ALMA sees can be split up into 32 times more wavelength ranges than in the initial design, and each of these ranges can be finely tuned. (Photo Credit: ESO)

alma-cabling

This photograph shows just some of the many thousands of cables needed to connect the electronics racks of the correlator together. The full system requires 32,768 rack-to-rack digital interfaces and 16,384 cables to transport the signals between the racks. (Photo credit: ESO)

alma-exterior

The ALMA Array Operations Site (AOS) Technical Building, the highest altitude high-tech building in the world, at an altitude of 5,000 metres on the Chajnantor Plateau in the Chilean Andes. This building houses the ALMA correlator supercomputer, enabling its many antennas, which are separated by up to 16 kilometres on the plateau, to work together as a single, giant telescope. (Photo Credit: ALMA, A. Caproni of ESO)

alma-oxygen

Enrique Garcia, a correlator technician, examines the system while breathing oxygen from a backpack. ALMA’s Array Operations Site is so high that only half as much oxygen is available as at sea level. (Photo Credit: ESO/Max Alexander)

Want more? For a video overview, continue to the next page.

This video from the ESO (European Southern Observatory) provides an overview of the ALMA Correlator and its work.

For more coverage of supercomputing at DCK, see our HPC Channel.