Build a list on every target from continuous integration#
Build one list on every target of a site file from continuous integration (CI): one run per target, each with a run id of its own, and one target at a time unless the targets build in places of their own. The caller names no partition, path, licence group or quota command; the site file holds them, and the caller keeps the exit status and the result file of each run.
Before you start#
A site file that passes
check, in a repository with the build lists, as in Write a site file for your cluster.A CI runner that runs as the build account, on a host that can start jobs on the cluster, with eb-pipeline on its
PATH, as in Install eb-pipeline.For GitHub Actions, a self-hosted runner with a label of its own; the workflow below calls it
eb-build.
The site file on this page builds for two kinds of node, zen2 and zen4, each with its own software tree:
# Example University: the zen2 and zen4 nodes of the cpu partition, each
# with a software tree of its own, and EasyBuild from a module.
schema_version = 1
kind = "site"
[site]
name = "example"
build_account = "alice"
default_group = "alice"
[scheduler]
kind = "slurm"
launch = "srun"
job_name_template = "{project}-{build}-{phase}-{target}"
[phase_defaults]
plan = "-p cpu -N 1 --ntasks-per-node=1 -c 1 -t 15:00"
short = "-p cpu -N 1 --ntasks-per-node=1 -c 1 -t 15:00"
build = "-p cpu -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"
cpu = "-p cpu -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"
cache = "-p cpu -N 1 --ntasks-per-node=1 -c 1 -t 15:00"
[[targets]]
name = "zen2"
build_lists = ["zen2.list"]
site_prefix = "/opt/site/zen2"
[targets.phases]
build = "-p cpu -C zen2 -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"
[[targets]]
name = "zen4"
build_lists = ["zen4.list"]
site_prefix = "/opt/site/zen4"
default = true
[targets.phases]
build = "-p cpu -C zen4 -N 1 --ntasks-per-node=1 -c 16 -t 1-00:00:00"
[filesystems]
scratch = "/scratch/alice"
tmp = "/scratch/alice/tmp"
# A build directory and a download directory per target, so that the
# targets can build at once.
build_path = "/scratch/alice/build/{target}"
source_paths = ["/scratch/alice/sources/{target}", "/opt/site/sources"]
install_root = "/opt/site"
log_root = "/opt/site/logs"
[modules]
init = ["EasyBuild"]
Steps#
List the targets that the runs go through, in the order of the site file, as lines and as the array in JavaScript Object Notation (JSON) that a CI matrix reads:
eb-pipeline targets --site site.toml eb-pipeline targets --site site.toml --format json
zen2 zen4 ["zen2","zen4"]
A site file that does not pass
checkmakestargetsexit 2 with nothing on stdout, so no run starts.For GitHub Actions, copy
examples/ci/github-actions.ymlof the eb-pipeline repository into.github/workflows/of your repository, and changeeb-buildto the label of your runner:
name: build
# Build one list on every target of the site file, one job per target.
# Copy this file into .github/workflows/ of the repository that holds
# site.toml and the build lists. `eb-build` is the label of a self-hosted
# runner on the build account, with eb-pipeline on its PATH.
on:
workflow_dispatch:
inputs:
list:
description: Build list, relative to the repository root
required: true
build_type:
description: Build type, as the site file names it
required: true
permissions:
contents: read
env:
EB_PIPELINE_SITE: site.toml
jobs:
targets:
runs-on: [self-hosted, eb-build]
outputs:
names: ${{ steps.targets.outputs.names }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# A JSON array on one line, as GITHUB_OUTPUT takes it. An invalid
# site file exits 2 here, and no build job starts.
- id: targets
run: |
names=$(eb-pipeline targets --format json)
echo "names=$names" >> "$GITHUB_OUTPUT"
build:
needs: targets
runs-on: [self-hosted, eb-build]
# One dispatch at a time builds a target: while one builds it, another
# dispatch's job for that target waits, and a running build is never
# cancelled. GitHub keeps only one pending job per group, so a third
# dispatch replaces the waiting one, which is cancelled. The group
# holds across dispatches and workflows of this repository, not across
# targets: two dispatches can still build two targets at once. The
# eb-pipeline guide "Build a list on every target from continuous
# integration" has the details.
concurrency:
group: eb-pipeline-${{ matrix.target }}
cancel-in-progress: false
strategy:
# One target at a time. Without {target} in [filesystems] build_path
# every target gets the same EASYBUILD_BUILDPATH, and EasyBuild removes
# the build directory of an easyconfig when a build of it starts, so
# two targets building one easyconfig at once break each other's build.
# Raise max-parallel only when build_path has {target}, or sits on
# storage that no two targets' build nodes share, and each target also
# has its own first source path and, with staged rebuilds, its own
# state root. The guide named above has the details.
max-parallel: 1
# A failed target leaves the others running.
fail-fast: false
matrix:
target: ${{ fromJSON(needs.targets.outputs.names) }}
env:
EB_PIPELINE_LIST: ${{ inputs.list }}
EB_PIPELINE_BUILD_TYPE: ${{ inputs.build_type }}
EB_PIPELINE_TARGET: ${{ matrix.target }}
# stack shard writes into <shard_dir>/<run-id>, and eb-staged-rebuild
# refuses a run id it already holds, so each target and each attempt
# has its own.
EB_PIPELINE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.target }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- run: eb-pipeline run --result "result-$EB_PIPELINE_TARGET.json"
- if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: result-${{ matrix.target }}
path: result-${{ matrix.target }}.json
Keep the run id of each target apart. The workflow sets
EB_PIPELINE_RUN_IDto${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.target }}:stack shardwrites its plan intoSHARD_DIR/RUN_ID, which has no target in it, and eb-staged-rebuild refuses a run id it already holds. Without--run-id, the run id is the log stamp, which is the same for two runs that start in the same second.Keep
max-parallel: 1unless no two targets that build at once share a build directory, a download directory or, with staged rebuilds, a state root.{target}infilesystems.build_pathand in the first entry offilesystems.source_pathsgives each target its own, as in the site file above; check what each target gets:for t in $(eb-pipeline targets --site site.toml); do eb-pipeline explain --site site.toml --target "$t" | grep -E "BUILDPATH|SOURCEPATH" done
EASYBUILD_BUILDPATH = "/scratch/alice/build/zen2" EASYBUILD_SOURCEPATH = "/scratch/alice/sources/zen2:/opt/site/sources" EASYBUILD_BUILDPATH = "/scratch/alice/build/zen4" EASYBUILD_SOURCEPATH = "/scratch/alice/sources/zen4:/opt/site/sources"
With staged rebuilds,
{target}inrebuild.state_root, or nostate_root, gives each target a state root of its own. Logs need nothing: each run writes underLOG_ROOT/LOG_STAMP/TARGET, and with[logs.rotate]a run rotates only when no other run under the log root is live, as The log tree and its locks says.Keep the
concurrencyof the build job.max-parallelholds within one run of the workflow, and each dispatch is a run of its own. The build job’sconcurrencygroup,eb-pipeline-<target>, lets one dispatch at a time build a target: another dispatch’s job for that target waits until the running one has finished, andcancel-in-progress: falseleaves the running build alone. GitHub keeps only one pending job per group, so a third dispatch replaces the waiting one, which is cancelled. The group belongs to the repository and has no workflow name in it, so another workflow of the repository that uses the same group waits as well. It keeps two dispatches off one target, not off one path: one dispatch can build a target while another builds a different one. Targets that share a build directory, a download directory or a state root therefore need one group for the whole build job,eb-pipelinewith no target in it, so that one build job of the repository runs at a time.For another CI system, run the targets one after the other from a script, and keep its exit status:
#!/bin/bash
# Build one list on every target of site.toml, one target after the other.
# Usage: build-all.sh LIST BUILD_TYPE RUN_ID
# Exits as eb-pipeline does: 1 when a target failed, else 78 when one was
# refused, else 75 when one is pending, else 0; a usage error or a signal
# stops the loop with its own status.
set -u
export EB_PIPELINE_SITE=site.toml EB_PIPELINE_LIST=$1 EB_PIPELINE_BUILD_TYPE=$2
run_id=$3
names=$(eb-pipeline targets) || exit
worst=0
for t in $names; do
EB_PIPELINE_TARGET=$t EB_PIPELINE_RUN_ID=$run_id-$t \
eb-pipeline run --result "result-$t.json"
rc=$?
echo "$t: exit $rc"
case $rc in
0) ;;
1) worst=1 ;;
78) [ "$worst" -eq 1 ] || worst=78 ;;
75) [ "$worst" -ne 0 ] || worst=75 ;;
*) exit "$rc" ;;
esac
done
exit "$worst"
The script exits as one run of eb-pipeline would, from what the runs of its targets came to; Exit status gives the order.
Check that it worked#
Each run prints its lines as it goes, and the script the exit status of each target.
Here the script ran against a copy of the site file above with the local scheduler, no modules and paths under a work directory, and with a stand-in for eb that fails the second line of the list:
bash build-all.sh build.list production 1234; echo "exit $?"
INFO [licences] zen2: not configured
INFO [shard] zen2: not configured
INFO [sources] zen2: not configured
INFO [admission] zen2: not configured
INFO [caches] zen2: not configured
INFO [logs] zen2: not configured
INFO [build] zen2: Example-1.0-GCCcore-15.2.0: start
INFO [build] zen2: Example-1.0-GCCcore-15.2.0: ok
INFO [build] zen2: Broken-1.0-GCCcore-15.2.0: start
ERROR [build] zen2: Broken-1.0-GCCcore-15.2.0: failed (exit 1)
zen2: exit 1
INFO [licences] zen4: not configured
...
ERROR [build] zen4: Broken-1.0-GCCcore-15.2.0: failed (exit 1)
zen4: exit 1
exit 1
Each target leaves a result file, and the exit status of the run is its top-level exit_code:
grep -H '^ "exit_code"' result-*.json
result-zen2.json: "exit_code": 1,
result-zen4.json: "exit_code": 1,
In GitHub Actions, each build job uploads its result file as the artifact result-TARGET.
If it fails#
Message |
Cause |
What to do |
|---|---|---|
|
The site file does not pass |
Correct the site file, and run |
|
The list is not where the run looks for it, relative to the directory the job runs in. |
Give the list relative to the top of the repository, which the workflow checks out; the script stops with exit 2. |
No result file to upload |
The run stopped on an input error before it wrote one, and exited 2 or, for the licence gate alone, 78. |
Read the run’s |
A line |
Two targets that share a state root rebuilt at once, and the second waited longer than |
Give each target a state root of its own, or keep |
The commands and their output on this page come from docs/examples/ci.sh.