Genomic research is pivotal for advancing bioenergy applications, a key interest of the Department of Energy (DOE). Massive genomic datasets are essential for developing sustainable biofuels and understanding environmental impacts. However, a critical bottleneck exists in "read mapping"—the process of aligning short DNA sequences to a reference genome—where the speed of DNA sequencing far exceeds the capacity of data analysis. This mismatch delays bioenergy research advancements and hampers the potential of genomics to drive innovations in sustainable energy solutions. Building on proven, published research - the first in-storage processing system designed for genomic sequence analysis - this Phase I project aims to develop a massively scalable, parallel implementation using standard commercial server infrastructure. By leveraging advanced parallel processing capabilities in our existing analytic data storage platform, we intend to significantly accelerate the read mapping process. This innovative approach will enhance the efficiency of genomic data analysis, directly benefiting bioenergy research by enabling faster discovery and development of sustainable biofuels. The project will focus on three key areas: (1) Enhance the storage of large genomic datasets within the innovative analytic data storage platform to improve efficiency and speed; (2) Accelerate Read Mapping by developing solutions to perform genetic sequence matching while data is stored within a distributed storage environment; and (3) Validate how this solution can be utilized for bioenergy research. This work will be supported by the researcher who developed the in-storage genomic sequencing technology as well as a university supporting the bioenergy validation. If successful, this project could revolutionize genomic data processing, reducing the time and cost of sequencing, with broad applications in healthcare, agriculture, and bioenergy. These improvements would advance scientific research, potentially leading to significant economic and environmental benefits.

High-performance computing (HPC) systems at the Department of Energy (DOE) and other research institutions are critical for large-scale scientific discovery. These inherently multi-tenant facilities, with millions of processes active simultaneously, necessitate robust mechanisms to ensure high performance and stringent security across multiple users. However, existing resource allocation methods often struggle to dynamically and fairly distribute system resources under heavy load, leading to performance bottlenecks and potential security vulnerabilities. To address these challenges, we propose integrating Hierarchical Weighted Fair Queueing (H-WFQ) into our existing analytic data storage system. By implementing H-WFQ, we will enhance user isolation by controlling and limiting resource access at both tenant and process levels, thereby significantly improving security and performance. During Phase I, we will develop methods to integrate H-WFQ support into our analytic storage system to strengthen user isolation. Specifically, we will design multi-level resource management mechanisms for dynamic, fair resource allocation among tenants and users. We will validate the enhanced system's effectiveness through performance and security testing in simulated HPC environments and ensure compatibility with the Globus data management services. This integration will significantly enhance the efficiency, security, and performance of HPC systems by providing robust isolation aligned with administrative policies. In Phase II, we aim to commercialize this solution for large multi-tenant HPC facilities. Strengthening user isolation protects sensitive data and ensures compliance with regulatory standards across industries, representing a breakthrough in resource management for HPC systems critical to scientific and technological advancements.

High brightness X-ray/synchrotron light sources are employed in many experiments performed at Department of Energy (DOE) research facilities and in laboratories around the globe. Detectors and other telemetry produce Petabyte-scale datasets from these experiments stored as high-definition (HD) image files in Hierarchical Data Format Version 5 (HDF5). The complexity and production of experiment data is rapidly increasing at DOE's modern X-ray facilities, presenting enormous challenges in data acquisition, processing, analysis, storage, and management. These challenges, if left unresolved, will ultimately impact scientific productivity. AirMettle Inc. is developing a real-time smart data lake solution that simplifies big data analytics and accelerates processing by an order of magnitude, or more. AirMettle's key innovation involves massively parallel distributed data processing within an easily deployable software-defined storage framework ideal for modern cloud environments. The solution performs basic analytics tasks at the storage layer that reduce network traffic, improve data freshness, and enable real-time operation. This project will enable the storage service to efficiently ingest and then partition HDF5-formatted light source (HD image) data so it can be processed in a massively parallel manner in the storage layer. The teams will also invent the APIs necessary to enable AI-based analytics functions and models to be applied during the in-storage processing of this data. The resulting commercial software/service from this project will be a data storage solution that enables orders of magnitude faster ingestion and analysis of petabyte scale light source data sets, as well as image and video processing for public and private organizations. This should accelerate the pace of scientific research while enhancing industrial processes such as semiconductor manufacturing, leading to lower costs and higher quality consumer goods.

Modern X-ray and synchrotron light sources at Department of Energy (DOE) facilities generate vast datasets, often reaching petabyte scale. The growth in data volume and complexity presents significant challenges in processing, analyzing, storing, and managing these datasets efficiently. Addressing these challenges requires advanced tools that extract insights rapidly and securely while minimizing networking and storage bottlenecks. This project addresses these challenges by enhancing an analytical data platform to enable the rapid, in-place analytics on large image-rich datasets which are distributed across a large storage cluster on which the platform runs. In Phase I, we demonstrated the ability to partition large Hierarchical Data Format Version 5 (HDF5) datasets and to process them in parallel within this platform using User Defined Functions (UDFs), including AI inference models. We also validated the platform’s compatibility with tools commonly used by DOE researchers for accessing remote storage and computational resources. Phase II will expand the platform's capabilities to analyze a broader range of image-rich scientific and commercial datasets, improve its stability, security, and performance, and prepare the solution for commercial trials. The enhanced platform is expected to reduce analysis time for large image-rich datasets by up to 100x by leveraging massively parallel processing. This capability will empower researchers to derive actionable insights more quickly and efficiently, accelerating scientific discovery. Potential commercial applications include the analysis of drone surveillance data, with future opportunities in advanced manufacturing, medical imaging, and environmental monitoring. By addressing the growing demand for scalable analytics, the platform will foster innovation, enhance research efficiency, and support the development of data-intensive technologies.

The massive volumes of multi-dimensional array-oriented data generated by NOAA programs and the scientific community at large are predominantly stored in industry standard Network Common Data Form (NetCDF). Key challenges exist in making use of data stored in netCDF: data sets are often too large to be copied and transferred across networks for every user, and each time data is accessed by an analytics tool it must be retrieved, subsets extracted, and subsequently formatted, among other requirements, which can account for 80-90% of the total time needed to insight. To unlock the enormous potential of petabyte scale netCDF-formatted data stored at different locations, in this SBIR Phase I project, AirMettle, Inc. with its research partners from the University of Wisconsin-Madison proposes to explore the feasibility of integrating in-situ analysis capabilities for multi-dimensional data (netCDF) into our highly innovative real-time smart data lake solution. Dramatically accelerated data analytics performed at the storage layer addresses key challenges noted within the Climate Adaption and Mitigation Topic. Reducing data traffic between sites, shrinking required compute resources, and lowering costs – all while accelerating climate analyses by an order of magnitude – would bring great benefit to NOAA and the broader scientific community.

AirMettle Inc. is transforming big data analytics for NOAA and the broader scientific community with a real-time smart data lake solution. Our innovative method utilizes massively parallel in-storage data processing within a versatile software defined storage framework, deployable on-premises or as a cloudbased service. Building upon our successful NOAA SBIR Phase I project, we strive to improve the handling of large, multi-dimensional NetCDF4 datasets vital to climate and weather forecasting. This project incorporates on-demand rescaling, allowing users to directly load only the required data at their desired resolution from the storage service. We will validate the benefits for climatologists and enhance the solution's commercial robustness. Our goal is to accelerate basic operations by 100x and reduce the data retrieved from storage by over 10x for typical requests. The potential commercial applications span meteorology and climatology across various sectors, making it easier and faster for experts to access and analyze crucial climate and weather data. AirMettle's cutting-edge data lake solution is poised to revolutionize big data analytics and garner widespread interest from both public and private stakeholders.

The broader impact of this Small Business Technology Transfer (STTR) Phase I project will be to fundamentally accelerate the pace of data science. New solutions that alleviate or bypass the bottlenecks inherent in existing data analytics technologies are required to unlock the value contained in “big data” and bring greater benefits to research, business, and society at large. The potential benefits include improved quality control for manufactured goods, reduced fraud in financial transactions, and enhanced customization of consumer services. Many common software applications, such as search engines, online shopping, e-commerce, medical applications, and social networks, are backed by data analytical processing services. This project will enable massive data sets to be pre-processed directly by a shared storage service, then allow this capability to be efficiently utilized by client analytic applications. This project will accelerate transformation of analytical data to useful insights while reducing network congestion, simplifying complex analytics systems, and lowering information technology costs. This Small Business Technology Transfer (STTR) Phase I project examines the challenge of how to dramatically accelerate data-intensive computing problems by enabling large data objects to be processed directly in the storage layer then efficiently utilized by client applications. The technology will be built on a software defined data lake used for big data applications. Key technology will be added to enable JSON, one of the most widely used data interchange formats, to be processed in a distributed manner in-place within this storage solution with the objective of enabling client analytic applications to retrieve not the complete object but only the desired subset of content they require. The storage solution will be augmented to transform the data into a format that can be directly consumed by the clients. This will substantially increase efficiency within the storage itself and between analytics clients and the storage solution – while acceelrating data processing by bypassing bottlenecks. The result will be a smart data lake that can reduce network traffic, improve data freshness, and enable real-time operation – while accelerating big data analytics by an order of magnitude or more. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.