Final Project Summary

The NEARDATA project, a research project funded by the European Union’s Horizon Europe programme, has successfully completed its mission to build a world-class platform for extreme near-data processing. By bringing computation directly to where data is stored, the project has removed critical bottlenecks in managing massive datasets across the "compute continuum", from local hospital devices to massive cloud infrastructures.

The project has developed innovative technologies for Extreme Data Processing like:

A fully-fledged Extreme Data Hub (pyrun.cloud) including a full-fledged DataCockpit including many data connectors for scientific formats, and an optimized data processing platform (Lithops, Dask) including the different pipelines developed in the project.
A Confidential Data Broker adapted to confidential federated learning deployments that has been implemented over SCONE software, and validated in Surgery use cases.
A Stream Processing platform ensuring low-latency and elastic edge data processing that was built on top of Dell’s Pravega system and validated in Surgery and Genomics use cases.
An international Extreme Data Hub in Metabolomics (Metaspace) with optimized extreme data pipelines providing processed information to AI pipelines.

Breakthrough Achievements in Public Health

The project’s most tangible impacts are found in its five specialized medical and scientific use cases:

Surgery: Developed StreamSense, a semantic search engine that allows surgeons to find specific moments in surgical video archives in less than 30 milliseconds. This drastically accelerates AI training for real-time surgical assistance.
Disease Prevention: Created predictive models to identify genetic predispositions for Type 2 Diabetes, helping move healthcare from reactive treatment to proactive prevention.
Pathogen Tracking: Developed the KPop algorithm for the rapid tracking and classification of pathogens, a vital tool for managing future pandemics and food security.
Genetics Research: Optimized the Transcriptomics Atlas pipeline, reducing the computing costs and time required to process RNA sequences by two orders of magnitude.
Metabolomics: Optimized metabolomics pipelines (included in the popular datahub METASPACE) that allow researchers to securely access and analyze molecular data across heterogeneous repositories.

Europe’s Global Leadership

NEARDATA has firmly positioned Europe as a leader in "extreme data" and digital sovereignty:

Scientific Excellence: The consortium produced 42 scientific publications, with 38% appearing in elite global venues like Nature Communications.
Data Sovereignty: By pioneering Confidential Computing and Trusted Execution Environments (TEEs), the project ensures that sensitive European health data remains secure and under European control.
The DataNexus Cluster: NEARDATA led the formation of a cluster of seven EU projects collaborating to solve the world's most complex data mining and analytics challenges.

Future Development and Economic Potential

The technologies developed are designed for wide-scale industrial and commercial adoption:

Industrial Integration: Global partners like Dell Technologies are exploring the integration of NEARDATA’s search and analytics engines into commercial products like NativeEdge and Data Lakehouse.
Empowering SMEs: Platforms like PyRun (from URV) eliminate the "complexity tax" of cloud computing for Python developers. PyRun was also integrated in the open source OpenNebula platform. Meanwhile, Scontain provides "sconified" services that allow small and medium enterprises to use advanced AI while meeting the strictest security regulations.

NEARDATA Key Achievements

Focus Area	Key Achievement	Future Development Potential and Positioning
Public Health and Medicine	Development of predictive models for Type 2 Diabetes. The StreamSense engine for surgical video search in < 30ms. The KPop algorithm for accelerated pathogen tracking. Integration of metabolomics repositories (METASPACE) and the Transcriptomics Atlas for precise diagnostics.	Creation of a pan-European surgical AI and precision medicine infrastructure. All these advances enable a shift from reactive medicine to proactive prevention, improving health security and the response to future pandemics.
Scientific Leadership	Production of 42 scientific publications, with 38% in elite venues such as Nature Communications. Process optimization that reduced the cost of RNA sequence analysis by two orders of magnitude.	Well positioned in Open Science through the publication of massive datasets: Appendix300 (surgery), Transcriptomics Atlas (genetics), and METASPACE. Influence on future global standards for genomic data compression (MPEG-G).
Data Privacy and Sovereignty	Implementation of Confidential Computing and Trusted Execution Environments (TEEs) to protect ultra-sensitive genomic and clinical data without slowing down computation.	Guarantee of digital sovereignty, allowing European hospitals to collaborate on AI training by sharing knowledge (models) without the need to move or expose private patient data.
Economic and Industrial Growth	Launch of commercial services such as PyRun and services from Scontain/SCONE for a secure cloud (Zero Trust). Integration of these technologies into the global portfolios of partners like Dell Technologies.	Democratization of access to complex data analysis for SMEs through a bioinformatics "app-store" model. Scontain's architecture allows companies to adopt advanced AI while complying with the strictest cybersecurity regulations.

Summary of the KPIs achieved by each use case

KPI	Use Case	Summary of results
KPI-1	Epistasis(GWD)	Lithops-HPC connector improves data ingestion in GWD pipeline by 36x.
KPI-1	Epistasis(GWD)	Hyperparameter selection improves performance by 5x.
KPI-5	Epistasis(GWD/MDR)	Cyclomatic-Complexity reveals 1.5x fewer execution paths.
KPI-5	Epistasis(GWD/MDR)	Yaqin’s metrics reveals 1.6x fewer branches, loops and nesting depth.
KPI-3	Epistasis(GWD/MDR)	The auto-scaler improves execution time of MDR use-case by 1.5x.
KPI-2	Surgery	StreamSense achieve low video frame indexing latency between 63ms and 360ms.
KPI-2	Surgery	Semantic video search latency is around 30ms for large collections of surgical video.
KPI-1	Surgery	Data transfer savings in AI loading. Integrating with PyTorch data transfers are reduced between 83.79% and 99.83%.
KPI-4	Surgery	Enhanced encryption of files with TEE, access control mechanisms. The security of the system was rigorously validated through adversarial testing and TEE attestation, confirming that both the confidentiality and integrity of model updates and training data were consistently enforced.
KPI-1	Transcriptomics	Early stopping technique have increased alignment throughput by 19.5%.
KPI-1	Transcriptomics	Use of a newer release of human genome index has resulted in execution times improvements of up to 12x and smaller STAR index file (from 85GB to 30GB).
KPI-1	Transcriptomics	The usage of spot instances reduced, on average, 50% of the execution cost.
KPI-1	Genomics	The Nexus FASTQGzip streamlet provides a good compression vs speed trade-off. It achieves 3.8x better compression ratio than plain FASTQ and 13.2x faster data processing than using FASTQ Gzip files.
KPI-1	Genomics	FaaStream is up to 65.14% cheaper than Flink running the genomics Kpop job.
KPI-1	Metabolomics	Depending on the size, we get a speed-up on processing time ranging from 1.13x to 1.22x faster.
KPI-4	Metabolomics	Achieved full confidential computing support (data at rest, in transit and in use) on cloud storage service (MinIO) and FaaS Lithops Singularity, aided by SCONE mechanisms and using the TEE; achieved partial confidential computing support (data at rest and in transit) due to limitations of the ported system (Metaspace is not fully compliant with Lithops Singularity + SCONE).