IBM and DESY Develop Big Data Architecture For Science

IBM and Deutsches Elektronen-Synchrotron (DESY) are joining forces to speed up management and storage of massive volumes of X-ray data.

Together they are building a Big Data and Analytics architecture using IBM software-defined technology. It will handle more than 20 gigabytes per second of data at peak performance and help scientists worldwide gain faster insights into the atomic structure of novel semiconductors, catalysts, biological cells and other samples.

DESY develops, builds and operates large particle accelerators used to investigate the structure of matter. It operates a 1.7 mile-long super microscope called the PETRA III accelerator. PETRA III speeds up electrically charged particles nearly to the speed of light, sending them through a tight magnetic slalom course to generate brilliant X-ray radiation.

This synchrotron radiation is used by more than 2,000 scientists each year to examine the internal structure of a variety of materials with atomic resolution. The problem and challenge is storing and handling huge volumes of X-ray data.

DESY is tackling the Big Data challenge with help of IBM Research and IBM software-defined storage technology Elastic Storage. Elastic Storage will provide scientists with high-speed access to increasing volumes of research data, and can easily scale to store and handle more than 20 gigabytes of data flowing every second from PETRA III.

The architecture will also allow DESY to develop an open ecosystem for research and offer analysis as a service, as well as cloud solutions to users worldwide.

“IBM’s software-defined storage technologies can provide DESY the scalability, speed and agility it requires to morph into a real-time analytics service provider,” said Jamie Thomas, General Manager Storage and Software Defined Systems, IBM. “IBM will take the experience gained at DESY and transfer it to other fields of data intensive science such as astronomy, climate research and geophysics and design storage architectures for the analysis of data generated by distributed detectors and sensors.”

The scalability of the system developed can support DESY and a number of international partners that are currently building the X-ray laser European XFEL, a research light source that will generate even more data.

"We expect about 100 petabyte per year from the European XFEL," said Dr. Volker Gülzow, head of DESY IT. That is comparable to the yearly data volume produced at the world's largest particle accelerator, the Large Hadron Collider (LHC) at the research center CERN in Geneva.

“A typical detector generates a data stream of about 5 gigabit per second, which is about the data volume of one complete CD-ROM per second," Gülzow said. "And at PETRA III we do not have just one detector, but 14 beamlines equipped with many detectors, and they are currently being extended to 24. All this Big Data must be stored and handled reliably."

DESY is housed in Hamburg and Zeuthen in Germany and is home to 3,000 scientists from over 40 countries a year.