Getting Started#

Requirements#

To use LEAPP, you must have:

  • Access to an HPC system with SLURM: You need a valid user account on an HPC system that uses SLURM for job scheduling.
  • ESGF tools: You need access to ESGF preparation and publication tools (esgmapfile, esgpublish) installed in conda environments.
  • Configuration file: You must create a config.yaml file with your HPC and project-specific settings.

Installation#

Pre-built releases (binary + jobs/*.slurm scripts + config.yaml.example) are published to the project’s Generic Package Registry on every tagged release.

mkdir -p leapp && cd leapp
curl --header "PRIVATE-TOKEN: <your_access_token>" -L -o leapp.tar.gz \
  "https://gitlab.hpc.cineca.it/api/v4/projects/<project_id>/packages/generic/leapp/latest/leapp-linux-amd64.tar.gz"
tar xzf leapp.tar.gz && rm leapp.tar.gz
chmod +x leapp
./leapp --version

Replace <project_id> with the project’s ID and <your_access_token> with a personal or project access token (read_api/read_package_registry scope) — required since this GitLab instance is not publicly accessible. To pin a specific version instead of always tracking the newest release, replace latest in the URL with a tag name (e.g. v0.1.0).

Keep the binary and jobs/ together The extracted directory contains everything needed to run the tool. Keep leapp and jobs/ in the same directory — the binary expects jobs/*.slurm to be reachable as a relative path from your current working directory.

Option 2: Compile from source#

git clone https://gitlab.hpc.cineca.it/esgf/leapp.git
cd leapp

Standard build#

go build -o leapp ./cmd/leapp
CGO_ENABLED=0 go build -o leapp ./cmd/leapp

HPC Tip
Use the static build (CGO_ENABLED=0) to avoid GLIBC version compatibility issues on HPC systems.

Initial Setup#

1. Create Configuration File#

Copy the example configuration and customize it:

cp config.yaml.example config.yaml

Edit config.yaml with your HPC and project-specific settings. See the Configuration section for details.

2. Set Up Password File#

The publish command requires a password for ESGF PID credentials:

Create the password file in your work directory#

echo 'your_password_here' > /path/to/work_dir/.leapp_secret

Setup correct permissions#

chmod 600 /path/to/work_dir/.leapp_secret

Basic Workflow#

Simplified (Two Commands)#

  1. Prepare data (generate mapfiles, stage to S3)#

    ./leapp prepare --data-path /path/to/data --project cmip6plus --run-id myrun1

    This launches mapgen (a SLURM job) and stage (direct SSH/local rclone transfer, not a SLURM job).

  2. Publish to ESGF (after prepare jobs complete)#

    ./leapp publish --data-path /path/to/data --project cmip6plus --run-id myrun1

Manual (Step by Step)#

  1. Generate mapfiles (check CV included)#

    ./leapp mapgen --data-path /path/to/data --project cmip6plus --run-id myrun1

  2. Stage data to S3#

    ./leapp stage --data-path /path/to/data --project cmip6plus --run-id myrun1

    Add --dry-run to preview the transfer without copying anything.

  3. Publish (after jobs complete)#

    ./leapp publish --data-path /path/to/data --project cmip6plus --run-id myrun1

Important Always verify that the mapgen SLURM job has finished successfully and that stage has completed its transfer before running publish.

Unpublishing (Optional)#

If you need to remove published datasets from the ESGF index (e.g., for corrections or test cleanup):

./leapp unpublish --data-path /path/to/data --project cmip6plus --run-id myrun1

The unpublish command runs immediately (not as a SLURM job) and will use the most recent run if --run-id is not specified.

Commands Forward