Configuration#

LEAPP uses a YAML configuration file with four main sections.

Configuration Structure#

hpc:
  work_dir: /path/to/work/directory
  conda_env: /path/to/conda/env
  pub_env: /path/to/publish/env
  rclone_bin: /path/to/rclone
  config_dir: /path/to/config/dir
  rclone_config: /path/to/.rclone.conf   # optional, defaults to ${work_dir}/.rclone.conf
  stage_host: data.example.org           # optional, SSH host for staging; local rclone if unset

output:
  base_dir: /path/to/output

slurm:
  account: your_account
  partition: your_partition
  qos: your_qos

  # Per-job overrides (fall back to account/partition/qos above if unset)
  mapgen_partition: your_mapgen_partition
  mapgen_account: your_account
  mapgen_qos: your_qos
  mapgen_cpus: 4
  mapgen_memory: 30G
  mapgen_time: "04:00:00"

  publish_partition: your_publish_partition
  publish_account: your_account
  publish_qos: your_qos
  publish_cpus: 1
  publish_memory: 8G
  publish_time: "04:00:00"

  max_parallel_jobs: 8

projects:
  cmip6plus:
    name: CMIP6Plus
    cmor_path: /path/to/cmor/tables/
    data_node: esgf-data.example.org
    index_node: esgf-node.example.org
    pid_prefix: "21.14100"
    pid_host: handle-esgf-trusted.dkrz.de
    pid_port: "5671"
    pid_user: esgf-publisher
    pid_vhost: esgf-pid
    s3_remote: s3remote:bucket
    s3_bucket: my-bucket-name

HPC Settings#

Field	Description
`work_dir`	Working directory for temporary files and password storage
`conda_env`	Path to conda environment with ESGF preparation tools (esgmapfile)
`pub_env`	Path to conda environment for publishing (esgpublish)
`rclone_bin`	Path to rclone binary
`config_dir`	Directory containing ESGF configuration files
`rclone_config`	(Optional) Path to rclone’s own config file. Defaults to `${work_dir}/.rclone.conf` if unset
`stage_host`	(Optional) SSH host used by `stage` to run rclone remotely. If unset, `stage` runs rclone locally instead

Output Settings#

Field	Description
`base_dir`	Default output directory for logs, mapfiles, and job outputs. Can be overridden per-command with `--output-dir`

SLURM Settings#

Global account/partition/qos apply to any job type that doesn’t define its own override. mapgen and publish each support their own partition, account, QOS, and resource overrides — useful when, for example, mapfile generation should run on a different queue than publishing.

Field	Description
`account`	Default SLURM account for job submission
`partition`	Default SLURM partition
`qos`	Default Quality of Service
`mapgen_partition`, `mapgen_account`, `mapgen_qos`	(Optional) Overrides for `mapgen` jobs only
`mapgen_cpus`, `mapgen_memory`, `mapgen_time`	Resource requests for `mapgen` jobs (passed as `--cpus-per-task`, `--mem`, `--time`)
`publish_partition`, `publish_account`, `publish_qos`	(Optional) Overrides for `publish` jobs only
`publish_cpus`, `publish_memory`, `publish_time`	Resource requests for `publish` jobs
`max_parallel_jobs`	Default number of parallel publish jobs (overridable with `--max-parallel`)

Why per-job overrides? Without a mapgen_partition/publish_partition override, both job types run on the same global partition. Set the per-job fields when mapfile generation and publishing need different queues, accounts, or QOS on your cluster.

stage does not appear here — it runs directly via SSH+rclone or local rclone (see stage_host above), not as a SLURM job, so it has no partition/account/QOS/resource settings.

Project Settings#

Each project requires the following fields:

Field	Description
`name`	Display name for the project
`cmor_path`	Path to CMOR tables
`data_node`	ESGF data node hostname
`index_node`	ESGF index node hostname
`pid_prefix`	PID prefix for handles
`pid_host`	PID service hostname
`pid_port`	PID service port
`pid_user`	PID service username
`pid_vhost`	PID virtual host
`s3_remote`	rclone remote for S3 staging
`s3_bucket`	(Optional) S3 bucket name, if your remote requires it separately
`clone_from`	(Optional) Clone configuration from another project

Adding a New Project#

Add a new entry under projects:

projects:
  mynewproject:
    name: MyNewProject
    cmor_path: /path/to/cmor/tables/
    data_node: esgf-data.example.org
    index_node: esgf-node.example.org
    pid_prefix: "21.14100"
    pid_host: handle-esgf-trusted.dkrz.de
    pid_port: "5671"
    pid_user: esgf-publisher
    pid_vhost: esgf-pid
    s3_remote: s3remote:bucket

Then use it with --project mynewproject.

Password Management#

The password for ESGF PID credentials is stored separately for security:

Create .leapp_secret in your work_dir:

echo 'your_password_here' > /path/to/work_dir/.leapp_secret
chmod 600 /path/to/work_dir/.leapp_secret

The file must contain only the password (no spaces or newlines)
Set permissions so only authorized users can read it