Build a list on every target from continuous integration#

Build one list on every target of a site file from continuous integration (CI): one run per target, each with a run id of its own, and one target at a time unless the targets build in places of their own. The caller names no partition, path, licence group or quota command; the site file holds them, and the caller keeps the exit status and the result file of each run.

Before you start#

  • A site file that passes check, in a repository with the build lists, as in Write a site file for your cluster.

  • A CI runner that runs as the build account, on a host that can start jobs on the cluster, with eb-pipeline on its PATH, as in Install eb-pipeline.

  • For GitHub Actions, a self-hosted runner with a label of its own; the workflow below calls it eb-build.

The site file on this page builds for two kinds of node, zen2 and zen4, each with its own software tree:

# Example University: the zen2 and zen4 nodes of the cpu partition, each
# with a software tree of its own, and EasyBuild from a module.
schema_version = 1
kind = "site"

[site]
name = "example"
build_account = "alice"
default_group = "alice"

[scheduler]
kind = "slurm"
launch = "srun"
job_name_template = "{project}-{build}-{phase}-{target}"

[phase_defaults]
plan = "-p cpu -N 1 --ntasks-per-node=1 -c 1 -t 15:00"
short = "-p cpu -N 1 --ntasks-per-node=1 -c 1 -t 15:00"
build = "-p cpu -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"
cpu = "-p cpu -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"
cache = "-p cpu -N 1 --ntasks-per-node=1 -c 1 -t 15:00"

[[targets]]
name = "zen2"
build_lists = ["zen2.list"]
site_prefix = "/opt/site/zen2"

[targets.phases]
build = "-p cpu -C zen2 -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"

[[targets]]
name = "zen4"
build_lists = ["zen4.list"]
site_prefix = "/opt/site/zen4"
default = true

[targets.phases]
build = "-p cpu -C zen4 -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"

[filesystems]
scratch = "/scratch/alice"
tmp = "/scratch/alice/tmp"
# A build directory and a download directory per target, so that the
# targets can build at once.
build_path = "/scratch/alice/build/{target}"
source_paths = ["/scratch/alice/sources/{target}", "/opt/site/sources"]
install_root = "/opt/site"
log_root = "/opt/site/logs"

[modules]
init = ["EasyBuild"]

Steps#

  1. List the targets that the runs go through, in the order of the site file, as lines and as the array in JavaScript Object Notation (JSON) that a CI matrix reads:

    eb-pipeline targets --site site.toml
    eb-pipeline targets --site site.toml --format json
    
    zen2
    zen4
    ["zen2","zen4"]
    

    A site file that does not pass check makes targets exit 2 with nothing on stdout, so no run starts.

  2. For GitHub Actions, copy examples/ci/github-actions.yml of the eb-pipeline repository into .github/workflows/ of your repository, and change eb-build to the label of your runner:

name: build
# Build one list on every target of the site file, one job per target.
# Copy this file into .github/workflows/ of the repository that holds
# site.toml and the build lists. `eb-build` is the label of a self-hosted
# runner on the build account, with eb-pipeline on its PATH.
on:
  workflow_dispatch:
    inputs:
      list:
        description: Build list, relative to the repository root
        required: true
      build_type:
        description: Build type, as the site file names it
        required: true

permissions:
  contents: read

env:
  EB_PIPELINE_SITE: site.toml

jobs:
  targets:
    runs-on: [self-hosted, eb-build]
    outputs:
      names: ${{ steps.targets.outputs.names }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      # A JSON array on one line, as GITHUB_OUTPUT takes it. An invalid
      # site file exits 2 here, and no build job starts.
      - id: targets
        run: |
          names=$(eb-pipeline targets --format json)
          echo "names=$names" >> "$GITHUB_OUTPUT"

  build:
    needs: targets
    runs-on: [self-hosted, eb-build]
    # One dispatch at a time builds a target: while one builds it, another
    # dispatch's job for that target waits, and a running build is never
    # cancelled. GitHub keeps only one pending job per group, so a third
    # dispatch replaces the waiting one, which is cancelled. The group
    # holds across dispatches and workflows of this repository, not across
    # targets: two dispatches can still build two targets at once. The
    # eb-pipeline guide "Build a list on every target from continuous
    # integration" has the details.
    concurrency:
      group: eb-pipeline-${{ matrix.target }}
      cancel-in-progress: false
    strategy:
      # One target at a time. Without {target} in [filesystems] build_path
      # every target gets the same EASYBUILD_BUILDPATH, and EasyBuild removes
      # the build directory of an easyconfig when a build of it starts, so
      # two targets building one easyconfig at once break each other's build.
      # Raise max-parallel only when build_path has {target}, or sits on
      # storage that no two targets' build nodes share, and each target also
      # has its own first source path and, with staged rebuilds, its own
      # state root. The guide named above has the details.
      max-parallel: 1
      # A failed target leaves the others running.
      fail-fast: false
      matrix:
        target: ${{ fromJSON(needs.targets.outputs.names) }}
    env:
      EB_PIPELINE_LIST: ${{ inputs.list }}
      EB_PIPELINE_BUILD_TYPE: ${{ inputs.build_type }}
      EB_PIPELINE_TARGET: ${{ matrix.target }}
      # stack shard writes into <shard_dir>/<run-id>, and eb-staged-rebuild
      # refuses a run id it already holds, so each target and each attempt
      # has its own.
      EB_PIPELINE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.target }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - run: eb-pipeline run --result "result-$EB_PIPELINE_TARGET.json"
      - if: always()
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
        with:
          name: result-${{ matrix.target }}
          path: result-${{ matrix.target }}.json
  1. Keep the run id of each target apart. The workflow sets EB_PIPELINE_RUN_ID to ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.target }}: stack shard writes its plan into SHARD_DIR/RUN_ID, which has no target in it, and eb-staged-rebuild refuses a run id it already holds. Without --run-id, the run id is the log stamp, which is the same for two runs that start in the same second.

  2. Keep max-parallel: 1 unless no two targets that build at once share a build directory, a download directory or, with staged rebuilds, a state root. {target} in filesystems.build_path and in the first entry of filesystems.source_paths gives each target its own, as in the site file above; check what each target gets:

    for t in $(eb-pipeline targets --site site.toml); do
        eb-pipeline explain --site site.toml --target "$t" | grep -E "BUILDPATH|SOURCEPATH"
    done
    
    EASYBUILD_BUILDPATH = "/scratch/alice/build/zen2"
    EASYBUILD_SOURCEPATH = "/scratch/alice/sources/zen2:/opt/site/sources"
    EASYBUILD_BUILDPATH = "/scratch/alice/build/zen4"
    EASYBUILD_SOURCEPATH = "/scratch/alice/sources/zen4:/opt/site/sources"
    

    With staged rebuilds, {target} in rebuild.state_root, or no state_root, gives each target a state root of its own. Logs need nothing: each run writes under LOG_ROOT/LOG_STAMP/TARGET, and with [logs.rotate] a run rotates only when no other run under the log root is live, as The log tree and its locks says.

  3. Keep the concurrency of the build job. max-parallel holds within one run of the workflow, and each dispatch is a run of its own. The build job’s concurrency group, eb-pipeline-<target>, lets one dispatch at a time build a target: another dispatch’s job for that target waits until the running one has finished, and cancel-in-progress: false leaves the running build alone. GitHub keeps only one pending job per group, so a third dispatch replaces the waiting one, which is cancelled. The group belongs to the repository and has no workflow name in it, so another workflow of the repository that uses the same group waits as well. It keeps two dispatches off one target, not off one path: one dispatch can build a target while another builds a different one. Targets that share a build directory, a download directory or a state root therefore need one group for the whole build job, eb-pipeline with no target in it, so that one build job of the repository runs at a time.

  4. For another CI system, run the targets one after the other from a script, and keep its exit status:

#!/bin/bash
# Build one list on every target of site.toml, one target after the other.
# Usage: build-all.sh LIST BUILD_TYPE RUN_ID
# Exits as eb-pipeline does: 1 when a target failed, else 78 when one was
# refused, else 75 when one is pending, else 0; a usage error or a signal
# stops the loop with its own status.
set -u
export EB_PIPELINE_SITE=site.toml EB_PIPELINE_LIST=$1 EB_PIPELINE_BUILD_TYPE=$2
run_id=$3
names=$(eb-pipeline targets) || exit
worst=0
for t in $names; do
    EB_PIPELINE_TARGET=$t EB_PIPELINE_RUN_ID=$run_id-$t \
        eb-pipeline run --result "result-$t.json"
    rc=$?
    echo "$t: exit $rc"
    case $rc in
        0) ;;
        1) worst=1 ;;
        78) [ "$worst" -eq 1 ] || worst=78 ;;
        75) [ "$worst" -ne 0 ] || worst=75 ;;
        *) exit "$rc" ;;
    esac
done
exit "$worst"

The script exits as one run of eb-pipeline would, from what the runs of its targets came to; Exit status gives the order.

Check that it worked#

Each run prints its lines as it goes, and the script the exit status of each target. Here the script ran against a copy of the site file above with the local scheduler, no modules and paths under a work directory, and with a stand-in for eb that fails the second line of the list:

bash build-all.sh build.list production 1234; echo "exit $?"
INFO [licences] zen2: not configured
INFO [shard] zen2: not configured
INFO [sources] zen2: not configured
INFO [admission] zen2: not configured
INFO [caches] zen2: not configured
INFO [logs] zen2: not configured
INFO [build] zen2: Example-1.0-GCCcore-15.2.0: start
INFO [build] zen2: Example-1.0-GCCcore-15.2.0: ok
INFO [build] zen2: Broken-1.0-GCCcore-15.2.0: start
ERROR [build] zen2: Broken-1.0-GCCcore-15.2.0: failed (exit 1)
zen2: exit 1
INFO [licences] zen4: not configured
...
ERROR [build] zen4: Broken-1.0-GCCcore-15.2.0: failed (exit 1)
zen4: exit 1
exit 1

Each target leaves a result file, and the exit status of the run is its top-level exit_code:

grep -H '^  "exit_code"' result-*.json
result-zen2.json:  "exit_code": 1,
result-zen4.json:  "exit_code": 1,

In GitHub Actions, each build job uploads its result file as the artifact result-TARGET.

If it fails#

Message

Cause

What to do

ERROR [check] example: targets: duplicate name `zen2`, from targets, and no build job

The site file does not pass check; targets exits 2 and prints no names.

Correct the site file, and run eb-pipeline check on it.

ERROR [main] -: cannot read list missing.list: No such file or directory (os error 2)

The list is not where the run looks for it, relative to the directory the job runs in.

Give the list relative to the top of the repository, which the workflow checks out; the script stops with exit 2.

No result file to upload

The run stopped on an input error before it wrote one, and exited 2 or, for the licence gate alone, 78.

Read the run’s ERROR lines in the job log.

A line refused with another eb-staged-rebuild holds the lock at PATH

Two targets that share a state root rebuilt at once, and the second waited longer than rebuild.lock_timeout.

Give each target a state root of its own, or keep max-parallel: 1 and one concurrency group for the build job.

The commands and their output on this page come from docs/examples/ci.sh.