The eb-pipeline command#
eb-pipeline reads a site file and a build list, and plans or runs the phases that build the list on one target of the site.
Each command has its own options; docs/examples/cli.sh prints every help text on this page.
Synopsis#
eb-pipeline <COMMAND>
eb-pipeline check [OPTIONS] --site <SITE>
eb-pipeline schema
eb-pipeline init [OPTIONS]
eb-pipeline explain [OPTIONS] --site <SITE>
eb-pipeline targets [OPTIONS] --site <SITE>
eb-pipeline plan [OPTIONS] --site <SITE> --list <LIST> --target <TARGET> --build-type <BUILD_TYPE>
eb-pipeline run [OPTIONS] --site <SITE> --list <LIST> --target <TARGET> --build-type <BUILD_TYPE>
eb-pipeline --help prints:
Site-neutral EasyBuild pipeline driver configured by a per-site TOML file
Usage: eb-pipeline <COMMAND>
Commands:
check Parse and validate a site file
schema Print the site file reference: every key, type, default and meaning
init Write a starter site file from a built-in template
explain Print the resolved configuration of one or all targets
targets Print the target names in file order, for a CI matrix or a shell loop
plan Compute phases, launch lines, environments and tool calls without executing
run Execute the plan's phases as subprocesses
help Print this message or the help of the given subcommand(s)
Options:
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
Examples:
eb-pipeline init --template small-cluster --out site.toml
eb-pipeline check --site site.toml
eb-pipeline targets --site site.toml --format json
eb-pipeline plan --site site.toml --list list.txt --target cpu --build-type production --out plan.json
eb-pipeline run --site site.toml --list list.txt --target cpu --build-type production --result result.json
Configuration: every flag above falls back to an EB_PIPELINE_<FLAG> variable
(EB_PIPELINE_SITE, EB_PIPELINE_LIST, EB_PIPELINE_TARGET, EB_PIPELINE_BUILD_TYPE,
EB_PIPELINE_RUN_ID, EB_PIPELINE_LOG_STAMP, EB_PIPELINE_LISTS_DIR); a flag on the
command line wins. EB_PIPELINE__SECTION__KEY=value sets one site file value, for
example EB_PIPELINE__FILESYSTEMS__SCRATCH=/scratch/eb; the result is validated
like the file.
Output streams: stdout carries plan, explain, schema, init, targets and run --dry-run
output only. Events go to stderr as `LEVEL [phase] target: message`. The result JSON
goes to --result.
Exit codes:
0 all requested work completed
1 work attempted and failed (a build line or a phase); see the result file
2 usage error, invalid site file, or a tool below its minimum version
75 a resource is temporarily unavailable or a step is pending; rerun
78 refused by a precondition, by admission or by eb-staged-rebuild; lines
built before the refusal stay built, see the result file
130 interrupted by SIGINT, after the running child ended
143 interrupted by SIGTERM, after the running child ended
Sibling tool codes are mapped explicitly:
eb-stack admit 0 admitted, 78 refused, 1 failed, 2 usage
eb-stack shard the code in shards.planner_exit_unsharded (default 3) means
unsharded; the run continues
eb-staged-rebuild 2 is a refusal there; the driver records it as refused
(exit 78), 75 as pending (exit 75), any other non-zero as failed
eb-keep-logs rotate 75 means another rotate holds the log root's lock;
the logs phase is recorded as skipped, any other non-zero
as failed
A signal (130, 143) or a usage error (2) ends the run with that code. Otherwise,
when several apply: 1, then 78, then 75, then 0. Codes are never summed.
eb-pipeline --version prints the version of the crate, the same for a release and for a build from source:
eb-pipeline 0.3.0
Arguments#
The first argument names the command. No command takes an argument other than its options.
Command |
What it does |
|---|---|
|
Read the site file, check it against every rule of The site file, and print warnings about settings that are valid but probably wrong. |
|
Print every key of the site file with its type, its default and what it does, section by section. |
|
Print a built-in template site file, or write it to a new file. |
|
Print what each target resolves to: paths, launch lines and the environment of its commands. |
|
Print the names of the targets in the order of the site file. |
|
Print the plan of a run as JavaScript Object Notation (JSON): its phases and every command with its launch line and environment. It starts nothing. |
|
Run the phases of the plan, one command at a time, and write the result file. |
|
Print the help of the program, or of the command it names. |
Options#
Every command takes -h, which prints a summary of its options, and --help, which prints them in full; both exit 0.
eb-pipeline -V and eb-pipeline --version print the version.
An option takes its value as the next argument or after an equals sign, as --site site.toml or --site=site.toml.
An option with a variable in the table takes its value from that variable when the command line does not give it.
check#
Option |
Variable |
Meaning |
|---|---|---|
|
|
The site file; required. |
|
|
A directory that must hold every file that the |
|
Start |
check writes INFO [check] NAME: site file is valid to stderr when the file is valid, where NAME is site.name.
Before that line it writes a WARN line for each of these settings:
modules.unuse_pathwith a firstmodules.initentry that is not an absolute path;admission.requiredwithoutadmission.quota_cmdandadmission.reservations, so thatrunrefuses with exit 78;admission.quota_cmdandadmission.reservationswithoutadmission.live_jobs_cmd.
schema#
schema takes no options.
It prints the reference of the site file on stdout, one block per section, with one entry per key:
the key, its type, whether it is required or its default, and what it does.
init#
Option |
Meaning |
|---|---|
|
The template to print; required unless |
|
Print the name and summary of each template, one per line, and nothing else. |
|
Write the template to |
The site file templates shows each template.
explain#
Option |
Variable |
Meaning |
|---|---|---|
|
|
The site file; required. |
|
|
The target to explain; every target when absent. |
|
|
Apply the |
|
|
For each target, explain prints the resolved paths, the launch line and arguments of each phase role, and the environment of its commands.
It shows <run-id> and <log-stamp> where a run puts its own.
targets#
Option |
Variable |
Meaning |
|---|---|---|
|
|
The site file; required. |
|
|
plan#
Option |
Variable |
Meaning |
|---|---|---|
|
|
The site file; required. |
|
|
The build list; required. |
|
|
The target to build on; required. |
|
|
The build type; required. A name without a |
|
|
The run label passed to the tools; |
|
|
The name of the run’s directory under the log root, a Coordinated Universal Time (UTC) as |
|
Write the plan to |
|
|
Read |
run#
run takes the first six options of plan, and these:
Option |
Meaning |
|---|---|
|
Write the result file to |
|
Print each command of the plan on stdout and start nothing. No result file is written, even with |
--run-id defaults to the log stamp, and --log-stamp to the UTC time at which the run starts, taken once.
A line that eb-staged-rebuild builds passes RUN_ID-STEM as its run id, where STEM is the easyconfig file name without .eb, with .2, .3 and so on added when the list repeats the line.
With --dry-run, a planned command prints as [PHASE] COMMAND and a phase that starts nothing as # [PHASE] STATUS.
Environment#
Each option in the tables above that names a variable reads it when the option is not on the command line.
EB_PIPELINE__SECTION__KEY=VALUE sets one value of the site file, and a run sets variables of its own for the commands it starts.
Environment variables lists all of them.
Exit status#
Status |
Meaning |
|---|---|
0 |
All requested work completed. |
1 |
|
2 |
A usage error, an invalid site file, an unknown target, a list that cannot be read, is empty or has a quote that is not closed, a tool below its |
75 |
|
78 |
|
130 |
|
143 |
|
The commands other than run exit 0, 1 or 2.
Exit status has the cases behind each status, how run picks one when several apply, and how it reads the exit statuses of the tools it starts.
Files#
File |
Use |
|---|---|
the site file, |
Everything about the site; The site file lists its keys. |
the build list, |
One easyconfig per line, followed by the options |
the plan, |
What |
the result file, |
What the run did, line by line and phase by phase; The plan and result files lists its keys. |
|
The log directory of a run, under |
A list line is split into words as a shell splits them, with single and double quotes.
See also#
The API of the
eb_pipelinelibrary, which the command is built on.