DOE Plans New Data Streaming Pipeline for Energy Research
Experts from SLAC National Accelerator Laboratory and other national labs have teamed up to build a new data streaming pipeline that will allow energy researchers to send data to supercomputing centers for analysis in real time.
The five-year project is called Intelligent Learning for Light Source and Neutron Source User Measurements Including Navigation and Experiment Steering, or ILLUMINE.
ILLUMINE will “facilitate rapid data analysis and autonomous experiment steering capabilities to support cutting-edge research driven by unprecedented data production rates, tightly coupling high-throughput experiments, advanced computing architectures, and novel AI/ML algorithms,” according to the website.
“Our ultimate goal is to stream data straight from the experiment directly to a remote computing facility without ever having to save anything to disk,” said project lead Jana Thayer, technical research manager at SLAC. “The ability to analyze the data while the experiment is still going on will allow us to make educated decisions such as changing the temperature and pressure to optimize the research results. That means faster times to solutions and more accurate science.”
Related content
- US Department of Energy’s INCITE Seeks Proposals for 2019
-
HPC Data Analytics
As data analytics workloads become more common, HPC administrators need to assess their hardware, software, and processes.
-
Tool your HPC systems for data analytics
As data analytics workloads become more common, administrators need to assess their hardware, software, and processes. -
Detecting security threats with Apache Spot
Security vulnerabilities often remain unknown when the data they reveal is buried in the depths of logfiles. Apache Spot uses big data and machine learning technologies to sniff out known and unknown IT security threats. - US Department of Energy Announces $10.5 Million Grant for Advanced Energy Research