Building Reproducible, Performance-Engineering Workflows for Earth Observation

From Scientific Notebooks to Reusable Workflows

Earth Observation (EO) data provides valuable insights into environmental change, natural disasters and land use. However, developing machine learning (ML) workflows that are reproducible, modular and capable of running efficiently on high-performance computing (HPC) systems presents significant technical challenges.

To address these challenges, ICHEC selected SEODA (Scalable Earth Observation Data Analytics) as an internal Digital Innovation Flagship (DIF) programme within the EuroCC2 project. The project aimed to transform scientific machine learning workflows into reusable, distributed pipelines that support reproducibility, benchmarking and performance optimisation.

The work began with scientific use cases provided by domain experts, initially implemented as Jupyter notebooks designed to run on single-GPU systems. The team worked to standardise key elements of these workflows, including dataset handling, model definitions, evaluation metrics, output management and distributed execution.

A Modular Pipeline for Earth Observation Machine Learning

A central outcome of SEODA was the development of a modular software pipeline designed to make machine learning workflows easier to configure, execute, monitor and analyse.

The pipeline brings together four key components:

• ai4eo_pipeline provides PyTorch data loaders for different dataset types.
• iclearn contains shared machine learning utility functions.
• icflow supports workflow execution, benchmarking and hyperparameter sweeps.
• icsystemutils provides tools for CPU/GPU monitoring and profiling.

Figure 1: SEODA workflow components

Together, these components support an experiment-driven workflow in which configurations define the execution of machine learning tasks. Outputs, metrics and metadata can then be collected in report archives, helping researchers track experiments, compare results and improve reproducibility.

FloodNet: A Practical Earth Observation Use Case

To explore the pipeline in a real-world scientific context, the project used FloodNet, a dataset developed for post-disaster scene understanding and flood management. FloodNet contains high-resolution aerial imagery collected after Hurricane Harvey in Texas in August 2017, alongside pixel-level annotations. These annotations support semantic segmentation, a computer vision task in which each pixel in an image is assigned to a category.

Working through this use case helped connect the scientific requirements of Earth Observation with the practical considerations of implementing, executing and analysing machine learning workloads on an HPC system.

A Cross-Disciplinary Team

SEODA brought together expertise from across ICHEC, combining Earth Observation, performance engineering, software development, HPC platform support, machine learning and training. The original idea came from Venkatesh Kannan, Associate Director (Solutions R&D), and the project developed through collaboration between colleagues with complementary skills.

The project contributors included:

• Project leadership and performance engineering: Buket Benek Gursoy, Computational Scientist, and Helene de Foras, intern student, formed the core performance engineering team, leading the project.
• Earth Observation domain expertise: Prakash Pilinja Subrahmanya, Postdoctoral Fellow, contributed domain expertise to help shape the scientific use cases.
• HPC application and platform support: Oisin Robinson, Computational Scientist from the National Service Support team, supported project applications for the MeluXina supercomputer.
• Software and platform development: James Grogan, Computational Scientist, and Benjamin Collingwood, HPC Parallel Programmer, contributed software and platform expertise.
• Machine learning, training and educational resources: Jake Mac Uilliam (Williams), Education, Training, and Skills Development Lead; Katie O’Connor, Computational Scientist; and Diarmuid Everard Coleman, Research Assistant, contributed machine learning expertise and work on teaching resources and repositories.

Software Repositories and Resources

SEODA’s software and related performance engineering resources are available through the following ICHEC repositories:

• ai4eo_pipeline: pipeline components for Earth Observation machine learning workflows.
• Performance Tool Shed: performance engineering tools and supporting utilities.
• Performance Recipes: reusable recipes and practical resources for performance engineering.

These resources provide a starting point for developing, running and analysing machine learning workflows, while supporting the reuse of software and practical knowledge across projects.

Building Foundations for Future HPC Workflows

SEODA delivered a distributed Earth Observation machine learning pipeline, prepared to run on MeluXina as a foundation for future infrastructure such as CASPIr. The lessons and capabilities developed through SEODA will inform future workflow development and contribute to ICHEC’s ongoing performance engineering activities

SEODA’s work was also shared with the wider geospatial community through an invited speaking engagement at the Women in Geospatial Ireland Chapter on 1 April 2026, presenting “Performance Engineering for Earth Observation Data Analytics.”