The site file#
The site file holds everything eb-pipeline knows about a site: its scheduler, its node types, its paths, its modules and the tools it calls.
It is one file in Tom’s Obvious Minimal Language (TOML), named by --site.
eb-pipeline schema prints the same keys with their types and defaults; tests/docs.rs checks this page against them.
Format#
The file starts with
schema_version = 1andkind = "site".An unknown key, or a value of the wrong type, is an error.
checklists every other problem of the file at once, oneERROR [check] NAME: PROBLEMline each, thenERROR [main] -: N problem(s) in the site file, and exits 2.An optional table that the file leaves out turns its phase off, and the plan and the result record the phase as
not configured. So doesenabled = falsein the tables that haveenabled.site.name,site.projectand the name of each target hold letters, digits,.,_and-only. A target name is not.or.., since it names a directory.A value that holds the arguments of a command, such as
scheduler.launchor a key of[phase_defaults], is split into words as a shell splits them, with single and double quotes.The variables
EB_PIPELINE__SECTION__KEYset values over the file before it is checked; Environment variables describes them.
Placeholders#
eb-pipeline replaces a placeholder in braces in these keys, and check refuses any other placeholder in them.
Keys |
Placeholders |
|---|---|
|
|
|
|
|
|
the entries of |
|
Placeholder |
Value |
|---|---|
|
|
|
the build type |
|
the phase that starts the command, such as |
|
the name of the target |
|
the run’s log directory, |
|
|
|
the target’s |
|
|
|
the entries of |
|
the same for |
{target} in paths#
{target} in filesystems.build_path, in an entry of filesystems.source_paths and in rebuild.state_root gives each target a path of its own.
plan and explain show the paths with the name in place.
build_path = "/scratch/alice/build/{target}"gives each target anEASYBUILD_BUILDPATHof its own, and{work_dir}follows it.EasyBuild looks for a source in every entry of
source_pathsand downloads a missing one into the first, sosource_paths = ["/scratch/alice/sources/{target}", "/opt/site/sources"]gives each target a download directory of its own and still finds the sources in/opt/site/sources.state_root = "/opt/site/{target}/.eb-staged-rebuild"gives each target an eb-staged-rebuild state root, and so a lock, of its own.
check refuses {target} in the other paths of [filesystems], and a state_root with {target} when two targets that take staged rebuilds share a software_root: eb-staged-rebuild locks a live tree through its state root, so one tree takes one state root.
Top level#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
integer |
required |
The version of the format; |
|
string |
required |
The kind of file; |
|
table |
required |
The site and its tools. |
|
table |
required |
How a run starts its commands. |
|
table |
none |
The launch arguments of each phase role. |
|
array of table |
required |
The node types to build for; at least one. |
|
table |
required |
The paths. |
|
table |
none |
Admission of each build unit under an inode quota, with |
|
table |
none |
Splitting a list into shards, with |
|
table |
none |
Fetching the sources of every line before the first build. |
|
table |
required |
The environment modules each command loads. |
|
table |
none |
The command that writes module caches after a clean build. |
|
table |
none |
Keeping the logs of a failed line, and rotating the log tree. |
|
table |
none |
Rebuilding installed software in a staged tree, with eb-staged-rebuild. |
|
table |
none |
The licence gate. |
[site]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
required |
The name of the site. A run exports it as |
|
string |
|
The value of |
|
string |
none |
The account that owns the installs; exported as |
|
string |
none |
The default Unix group of the installs; exported as |
|
string |
none |
A label for the generation of the software stack, such as |
|
table |
all tools looked up on |
The programs of the sibling tools and the versions they need. |
[site.tools]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
table |
|
eb-stack, which runs |
|
table |
|
eb-keep-logs, which runs |
|
table |
|
eb-staged-rebuild, which runs |
[site.tools.TOOL]#
TOOL is eb_stack, eb_keep_logs or eb_staged_rebuild.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
the tool’s own name, looked up on |
The program: a path, or a name to look up on |
|
string |
none |
The lowest version accepted, as dotted numbers such as |
[scheduler]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
required |
|
|
string |
|
The launch command of |
|
array of string |
|
Arguments of every launch line, after |
|
table of string |
|
Variables set for every command; a variable that |
|
string |
|
The job name each launch line passes as |
With kind = "slurm", a command starts as LAUNCH COMMON_ARGS --job-name=NAME ROLE_ARGS COMMAND, where ROLE_ARGS are the launch arguments of the command’s role and COMMAND starts with the module wrapper when there is one.
A role without launch arguments launches a bare srun --job-name=NAME, and inside an allocation srun then starts one task for each task of the allocation, so give each role a task count, such as -N 1 --ntasks-per-node=1.
[phase_defaults]#
Each key holds the launch arguments of one role, for every target that does not set its own in [targets.phases].
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
none |
|
|
string |
none |
The builds of the whole list, of the prelude and of each shard. |
|
string |
none |
Admission, source prefetch, |
|
string |
none |
The builds of the CPU-only prerequisites that |
|
string |
none |
|
[[targets]]#
One table per node type.
A run builds for one target, named by --target.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
required |
The name of the target, unique in the file. It names the target’s log directory, and |
|
array of string |
|
The file names of the build lists of the target; |
|
string |
none |
|
|
integer |
none |
|
|
string |
none |
|
|
string |
none |
The CPU subdirectory of the European Environment for Scientific Software Installations (EESSI), such as |
|
string |
none |
The accelerator subdirectory of EESSI, such as |
|
string |
none |
The install prefix of the target, exported as |
|
string |
|
The live software tree of the target: |
|
string |
|
The live module tree of the target: |
|
boolean |
|
Whether the target takes staged rebuilds. |
|
string |
|
The ledger configuration of the target’s staged rebuilds. The file names one target, so |
|
boolean |
|
Marks the default target of the site; |
|
table |
|
The launch arguments of the target, per role; a role it leaves out takes the one in |
|
table |
none |
A command that wraps the CPU-role commands of the target. |
|
table |
none |
Which lines of a list the target builds; without it, all of them. |
|
table |
none |
The CPU of the target’s build nodes; the commands get it only when |
|
table |
none |
The CPU of the nodes that build the CPU-role units. When it is set, the source prefetch and build commands export a CPU class as |
|
table |
none |
A table with any keys, passed to the source prefetch commands as |
|
table of tables |
|
Variables per package name, |
|
table of tables |
|
Values per build type, |
With [modules.eessi] on and site_install = true, EESSI-extend replaces EASYBUILD_INSTALLPATH when it loads, with $EESSI_SITE_SOFTWARE_PREFIX/versions/RELEASE/software/linux/CPU_TARGET, followed by /ACCEL_TARGET for an NVIDIA target when EESSI_ACCELERATOR_INSTALL is set.
explain names that path as the target’s install_note.
A target that then takes staged rebuilds, with filesystems.export_install_vars = "single", sets software_root and modules_root to the software and modules directories under it, and check refuses the file otherwise.
[targets.phases]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
none |
|
|
string |
none |
|
|
string |
none |
|
|
string |
none |
|
|
string |
none |
|
[targets.route]#
A route wraps the commands of the CPU-role units, the CPU-only prerequisites that stack shard lifts out, in a command of the site’s: the command, its arguments, -- when there are arguments, then the launch line and the command it wraps.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
none |
|
|
string |
none |
The wrapper, before the launch line; required with |
|
array of string |
|
Arguments after |
|
table of string |
|
Variables of the wrapped commands; the values take placeholders. |
|
string |
none |
The root of the route’s runtime: |
|
array of string |
|
Paths the route shares read-only, with placeholders; joined with |
|
array of string |
|
Paths the route shares read-write, with placeholders; joined with |
|
string |
none |
The name of a variable, in letters, digits and |
|
array of string |
|
The phases whose CPU-role commands the route wraps: |
A route with only kind and command puts command alone before the launch line.
[targets.select]#
A build list can mark a line for some targets only, with an option of the line that names them.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
required |
The long option that carries the selection, with its leading dashes, such as |
|
string |
required |
The token of this target in the option’s list; not empty, and without |
|
string |
|
What separates the tokens; not empty. |
|
string |
|
|
|
boolean |
|
|
plan and run apply the selection to the whole list, or to each shard unit, and drop a unit left without lines.
With a selection, run writes the lines it keeps, in list order and with strip applied, to LOG_DIR/selected-TARGET.list, and passes that file to stack shard --build-list.
The plan and the result list the lines left out under deselected, and no status count includes them.
[targets.hardware]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
none |
The CPU vendor, as |
|
integer |
none |
The CPU family, as |
|
integer |
none |
The CPU model, as |
[targets.cpu_hardware]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
none |
The CPU vendor of the nodes that build CPU-role units. |
|
integer |
none |
Their CPU family. |
|
integer |
none |
Their CPU model. |
[targets.build_types.NAME]#
--build-type NAME applies this table; a build type without one takes the values of the target.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
target value |
|
|
integer |
target value |
|
[filesystems]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
required |
The scratch root; not empty. |
|
string |
required |
|
|
string |
required |
|
|
array of string |
|
The entries of |
|
string |
required |
The install prefix of a target that sets no |
|
string |
|
|
|
string |
required |
The root of the log tree, which holds a directory |
|
string |
|
Where |
|
boolean |
|
|
|
string |
|
The value of |
|
string |
none |
|
[admission]#
Before each build unit, eb-stack stack admit checks that the unit fits in the free inodes of the scratch quota, and reserves them for the job.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
boolean |
|
|
|
boolean |
|
|
|
string |
none |
A shell command that prints the quota as JSON: |
|
string |
none |
A shell command that prints the ids of the running jobs of the build account, such as |
|
string |
none |
The lock file that every admission takes: |
|
string |
|
|
|
string |
none |
The directory of reservations: |
|
integer |
tool default |
The inodes to keep free: |
|
integer |
|
The inodes to count for each live job without a reservation: |
|
integer |
none |
The inodes of a unit for which the shard plan gives none; required when the table is enabled. |
|
string |
|
The variable that holds the job id at run time, in letters, digits and |
|
integer |
none |
The inode quota to plan with: |
|
string |
none |
A file of inode footprints: |
|
integer |
tool default |
The inodes of a list line beyond its build tree: |
|
integer |
|
How many times |
|
integer |
|
The wait before the second attempt, in seconds; it doubles before each later one. |
|
integer |
|
The longest wait between two attempts, in seconds. |
A unit’s footprint is the inode_footprint of the shard plan, else default_footprint.
When an admitted unit’s builds end, or the run ends first, run removes RESERVATIONS/JOB.json while it holds lock, taken as lock_kind.
A record that another writer has replaced since, such as a forecast or waiting one, stays, with a WARN line.
[sources]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
boolean |
|
|
|
boolean |
|
Run |
|
array of string |
|
Arguments of each prefetch command, after the line’s own options. |
A prefetch command gets the options of its line, except those that start with --rebuild.
A line whose prefetch fails is failed and is not built.
--fetch takes no account of installed modules, so --robot fetches the sources of every dependency, those EESSI provides included.
With --disable-robot in extra_args, a line fetches its own sources only: the prefetch command has no --robot, the line’s --robot is dropped, and the paths of the line’s --robot=PATH go first in --robot-paths, so that its easyconfig is still found.
[modules]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
array of string |
required |
The modules to load, in order, after |
|
boolean |
|
|
|
string |
|
The shell that runs the module wrapper, a script for a POSIX shell, by name or by path: |
|
string |
none |
A file to source before |
|
table |
none |
The site layer of EESSI. |
|
table of tables |
|
Variables per package name, |
The module wrapper starts when init is not empty, unuse_path is true or shell_init is set; Environment variables describes it.
[modules.eessi]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
boolean |
|
|
|
boolean |
|
|
|
string |
none |
Exported as |
|
string |
|
The variable that carries the target’s |
|
string |
|
The variable that carries the target’s |
With the table on, every command also gets EESSI_SITE_SOFTWARE_PREFIX, the target’s install prefix, and check checks the form of each target’s cpu_target and accel_target.
[caches]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
boolean |
|
|
|
string |
none |
The command that writes the module caches, run once after a clean build of a target that has an entry in |
|
string |
none |
Passed to |
|
string |
none |
Passed as |
|
array of table |
|
One |
A build is clean when no line failed, was refused or is pending; otherwise the caches phase is skipped.
[[caches.targets]]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
string |
required |
The name of a target; each target has one entry at most. |
|
string |
required |
The cache configuration file, passed as |
|
string |
|
The launch arguments of the cache command for this target. |
|
string |
none |
Passed as |
[logs]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
boolean |
|
|
|
boolean |
|
Keep the EasyBuild log of a failed line with |
|
table |
none |
Rotate the log tree with |
With keep_failed, each eb gets --tmp-logdir=LOG_DIR/easybuild-logs/STEM, and after a line fails run starts eb-keep-logs failed STATUS STEM LOG_DIR LOG_DIR/easybuild-logs/STEM, with EASYBUILD_OUTPUT_STYLE=basic whatever [scheduler.env] sets, so that eb --last-log prints the log path on one line.
[logs.rotate]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
integer |
required |
|
|
integer |
none |
|
The logs phase starts eb-keep-logs rotate LOG_ROOT --compress-after DAYS --delete-after DAYS, with TZ=UTC, since the log directories are named in Coordinated Universal Time (UTC).
A run rotates only when no other run under the log root is live: each run holds LOG_ROOT/.eb-pipeline-rotate.lock shared from its start, and rotates when it can take it alone; otherwise the logs phase is skipped.
Where eb-keep-logs --help lists --lock-timeout, the rotate also gets --lock-timeout 0, and exits 75, recorded as skipped, while another rotate holds LOG_ROOT/.eb-keep-logs-rotate.lock.
The log tree and its locks lists the files of a run and the locks, and gives their order.
[rebuild]#
A line goes through eb-staged-rebuild run instead of eb when all three hold:
the target takes staged rebuilds: its
staged, elserebuild.staged, istrue;the line carries
--rebuild,-for--forceas a whole option, which--force-downloadand--rebuild-from-scratchare not;the line’s module is installed under the target’s
modules_root.
Otherwise the line builds with eb; on a target that takes staged rebuilds, a line that has the option but no installed module gets a note in the plan and a WARN line in the run.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
boolean |
|
|
|
boolean |
|
Whether a target that sets no |
|
string |
tool default |
|
|
integer |
|
|
|
string |
|
|
|
integer |
tool default |
|
|
string |
none |
A program, passed as |
|
string |
none |
A file in JSON, passed as |
A staged line starts eb-staged-rebuild run --software SOFTWARE_ROOT --modules MODULES_ROOT --backend BACKEND --run-id RUN_ID-STEM --lock-timeout SECONDS, with the other keys of the table that are set, then --, the easyconfig and the options of the line.
STEM is the easyconfig file name without .eb, with .2, .3 and so on added when the list repeats a line, since eb-staged-rebuild refuses a run id it has seen.
A line whose wait for a lock runs out is refused; its note and its ERROR line name the lock that another eb-staged-rebuild holds and, when it is the state root’s, say that lock_timeout ran out.
[licences]#
The licence gate checks the site file only: every package that needs a licence has a group, a version threshold or an exemption. It does not check file permissions or group membership on the host.
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
boolean |
|
|
|
boolean |
|
Run the gate: every name of |
|
table of string |
|
Package name to Unix group, exported as |
|
table of string |
|
Package name to a version in dotted numbers; the package’s group applies below that version only, and to a version that does not parse. The package needs a group. |
|
table |
none |
A second list of the packages that need a licence, kept apart from |
|
array of string |
|
Names of |
|
table of tables |
|
Variables per package name, |
The gate is on when enabled and enforce are both true, and only then do the lines get EB_PIPELINE_LICENCE_GROUP and the variables of [licences.env].
When the gate’s problems are the only ones of the file, run exits 78; check and plan exit 2.
[licences.required]#
Key |
Type |
Default |
Meaning |
|---|---|---|---|
|
array of string |
|
The package names that each need a group, a threshold or an exemption. |
See also#
The site file templates, two complete site files to start from
The fields of the site file in eb_pipeline::schema, whose doc comments are the text
eb-pipeline schemaprints.