DOE Plans New Data Streaming Pipeline for Energy Research

By Amber Ankerholz

Researchers could send data to a remote computing facility for analysis in real time.

Experts from SLAC National Accelerator Laboratory and other national labs have teamed up to build a new data streaming pipeline that will allow energy researchers to send data to supercomputing centers for analysis in real time.

The five-year project is called Intelligent Learning for Light Source and Neutron Source User Measurements Including Navigation and Experiment Steering, or ILLUMINE.

ILLUMINE will “facilitate rapid data analysis and autonomous experiment steering capabilities to support cutting-edge research driven by unprecedented data production rates, tightly coupling high-throughput experiments, advanced computing architectures, and novel AI/ML algorithms,” according to the website.

“Our ultimate goal is to stream data straight from the experiment directly to a remote computing facility without ever having to save anything to disk,” said project lead Jana Thayer, technical research manager at SLAC. “The ability to analyze the data while the experiment is still going on will allow us to make educated decisions such as changing the temperature and pressure to optimize the research results. That means faster times to solutions and more accurate science.”