DOE Plans New Data Streaming Pipeline for Energy Research
Experts from SLAC National Accelerator Laboratory and other national labs have teamed up to build a new data streaming pipeline that will allow energy researchers to send data to supercomputing centers for analysis in real time.
The five-year project is called Intelligent Learning for Light Source and Neutron Source User Measurements Including Navigation and Experiment Steering, or ILLUMINE.
ILLUMINE will “facilitate rapid data analysis and autonomous experiment steering capabilities to support cutting-edge research driven by unprecedented data production rates, tightly coupling high-throughput experiments, advanced computing architectures, and novel AI/ML algorithms,” according to the website.
“Our ultimate goal is to stream data straight from the experiment directly to a remote computing facility without ever having to save anything to disk,” said project lead Jana Thayer, technical research manager at SLAC. “The ability to analyze the data while the experiment is still going on will allow us to make educated decisions such as changing the temperature and pressure to optimize the research results. That means faster times to solutions and more accurate science.”
Related content
- US Department of Energy’s INCITE Seeks Proposals for 2019
-
News for Admins
In the news: Palo Alto Networks Introduces Cortex Cloud; Canonical to Provide 12 Years of Kubernetes Support; Mirantis Releases Open Source k0rdent; D-Wave Now Offers On-Premises Quantum Computing Systems; IP Fabric 7.0 Released; UK Releases Code of Practice for Securing AI; Red Hat Releases Kubernetes-Native Connectivity Link; OpenVox Automation Framework Announced; Sysdig Launches Open Source Stratoshark for Cloud Observability; Data Center Electricity Demand Projected to Double or Triple by 2028; Red Hat Releases OpenShift Virtualization Tools; 16GB Raspberry Pi 5 Announced; Open Storage Network Adds More Sites.
-
HPC Data Analytics
As data analytics workloads become more common, HPC administrators need to assess their hardware, software, and processes.
-
Tool your HPC systems for data analytics
As data analytics workloads become more common, administrators need to assess their hardware, software, and processes. -
Detecting security threats with Apache Spot
Security vulnerabilities often remain unknown when the data they reveal is buried in the depths of logfiles. Apache Spot uses big data and machine learning technologies to sniff out known and unknown IT security threats.