LEAPP#

Launcher for ESGF Automated Preparation and Publication

LEAPP is a command-line interface (CLI) designed to streamline the management of large-scale ESGF data operations on HPC systems. This tool acts as a front-end for launching specific SLURM jobs on an HPC system, ensuring consistent and reproducible workflows.

Overview#

Publishing climate model output to the Earth System Grid Federation (ESGF) is a multi-step process: data must be validated against controlled vocabularies, described with mapfiles and checksums, transferred to durable storage, and finally registered with an ESGF index node via esgpublish. On an HPC system, each of these steps can involve scanning or moving terabytes of data — work that belongs on compute nodes and under SLURM’s scheduler, not on a login node.

LEAPP wraps this entire pipeline behind a small set of consistent commands. Instead of hand-crafting sbatch invocations, esg.yaml configuration files, and mapfile lists for every dataset, you run:

./leapp prepare --data-path /path/to/data --project cmip6plus --run-id myrun1
./leapp publish --data-path /path/to/data --project cmip6plus --run-id myrun1

and LEAPP takes care of vocabulary checks, mapfile generation, data staging, SLURM job submission, parallel publishing across multiple jobs, and cleanup — the same way, every time, regardless of dataset or project.

The publication pipeline#

At a high level, a dataset moves through three stages before it’s discoverable on ESGF:

  1. Prepare — mapgen generates ESGF mapfiles and checksums (as a SLURM job), and stage transfers the data to S3-compatible storage (via SSH+rclone or local rclone, run directly rather than through SLURM).
  2. Publish — publish registers the generated mapfiles with the ESGF index node using esgpublish, splitting the work across multiple parallel SLURM jobs for large datasets.
  3. Manage — unpublish reverts a publication, cleanup removes stale working files, and freeze/unfreeze control group ownership and permissions on published data.

Design goals#

LEAPP was built around a few explicit goals:

  • Nothing hardcoded. Every HPC path, SLURM account/partition/QOS, and ESGF project setting lives in a single config.yaml — no institution- or project-specific values baked into the binary.
  • Multi-project by default. Adding a new ESGF project (CMIP6Plus, CMIP6, CORDEX, or a custom one) is a config file entry, not a code change.
  • Portable across HPC systems. The same binary and job scripts work on any SLURM cluster; only config.yaml changes between sites.
  • Safe to retry. Staging is idempotent, and jobs verify their inputs (config files, mapfile lists) with retries before doing anything destructive.

Who it’s for#

LEAPP is aimed at teams operating ESGF data publication pipelines on shared HPC systems — typically climate modelling groups who need a repeatable, auditable way to take raw model output from disk to a published ESGF dataset without every publication being a bespoke, manual effort.

Features#

  • Automated ESGF workflows - Streamlined data preparation and publication
  • SLURM integration - Efficient job scheduling on HPC systems
  • Multi-project support - Works with CMIP6Plus, CMIP6, CORDEX, and more
  • Parallel publishing - Distribute workload across multiple compute nodes
  • Flexible configuration - YAML-based configuration for different HPC environments

Quick Start#

  1. Prepare data (generate mapfiles, stage to S3)#

    ./leapp prepare --data-path /path/to/data --project cmip6plus --run-id myrun1

  2. Publish to ESGF (after prepare jobs complete)#

    ./leapp publish --data-path /path/to/data --project cmip6plus --run-id myrun1

Documentation Sections#

  • Getting Started - Installation, requirements, and initial configuration
  • Commands - Complete reference for all CLI commands
  • Configuration - Detailed configuration options and examples

Source#

This tool was originally forked from optimesgf and refactored to be flexible across different HPC systems and ESGF projects.