Python API

Generated from the installed package. Functions and classes without a leading underscore are listed. The command-line behaviour is Command line.

Package

Stage an EasyBuild rebuild, then exchange it with the stable live path.

Command line

Command line for eb-staged-rebuild.

eb_staged_rebuild.cli.main(argv=None)

Engine

Build complete staged installs and transactionally swap them into place.

class eb_staged_rebuild.engine.Spec(software: str, modules: str, state_root: str, eb_args: list, backend: str = 'auto', grace_hours: float = 144, dry_run: bool = False, module_rels: list = <factory>, eb_executable: str = 'eb', builder: object = None, faults: object = None, run_id: str = None, skip_live_check: bool = False, lock_timeout: float = None, clock: object = None, probes: dict = None, prepare_executable: str = None, ledger_config: str = None, ledger_policy: dict = None)

Bases: object

Parameters:

• software (str)

• modules (str)

• state_root (str)

• eb_args (list)

• backend (str)

• grace_hours (float)

• dry_run (bool)

• module_rels (list)

• eb_executable (str)

• builder (object)

• faults (object)

• run_id (str)

• skip_live_check (bool)

• lock_timeout (float)

• clock (object)

• probes (dict)

• prepare_executable (str)

• ledger_config (str)

• ledger_policy (dict)

software: str

modules: str

state_root: str

eb_args: list

backend: str = 'auto'

grace_hours: float = 144

dry_run: bool = False

module_rels: list

eb_executable: str = 'eb'

builder: object = None

faults: object = None

run_id: str = None

skip_live_check: bool = False

lock_timeout: float = None

clock: object = None

probes: dict = None

prepare_executable: str = None

ledger_config: str = None

ledger_policy: dict = None

class eb_staged_rebuild.engine.Result(returncode: int, run_id: str = None, state: str = '', message: str = '', command: str = '')

Bases: object

Parameters:

• returncode (int)

• run_id (str)

• state (str)

• message (str)

• command (str)

returncode: int

run_id: str = None

state: str = ''

message: str = ''

command: str = ''

eb_staged_rebuild.engine.software_rels_from_dry_run(text)

eb_staged_rebuild.engine.run(spec)

Build into a new generation, then swap it with the live directory.

eb_staged_rebuild.engine.resume(state_root, run_id, builder=None, faults=None, clock=None, lock_timeout=None)

Continue a run. A finished run is a no-op. A rolled-back run is refused.

eb_staged_rebuild.engine.rollback(state_root, run_id, faults=None, clock=None, lock_timeout=None)

Exchange every retired tree back into place as one recoverable operation.

eb_staged_rebuild.engine.describe_module(path)

Builders

Run EasyBuild so it writes a generation that is not the live software directory.

eb_staged_rebuild.builders.filter_eb_args(args)

Drop install-path and bubblewrap options. Keep the easyconfig and the rest.

eb_staged_rebuild.builders.ensure_rebuild(args)

eb_staged_rebuild.builders.child_env(base, software, modules, live_modules)

eb_staged_rebuild.builders.installpath_dirs(generation_root)

eb_staged_rebuild.builders.installpath_command(ctx)

eb_staged_rebuild.builders.bwrap_command(ctx)

eb_staged_rebuild.builders.userns_command(ctx)

Run an install-path build with both live trees mounted read-only.

eb_staged_rebuild.builders.dry_run_command(ctx)

List module names without installing. Uses the live prefix.

eb_staged_rebuild.builders.run_command(cmd, env, faults, step)

Keep the caller’s lock held until the command process group has stopped.

Catchable termination signals request group shutdown. The group leader remains unreaped until the last group signal, so its identifier cannot be reassigned to an unrelated process. Commands must keep their workers in their process group and join them before returning.

eb_staged_rebuild.builders.format_command(cmd)

Probe

Capability probes. Results describe this machine and this directory, nothing else.

eb_staged_rebuild.probe.probe_bwrap()

eb_staged_rebuild.probe.probe_user_namespaces()

eb_staged_rebuild.probe.probe_userns_sandbox()

Test a user and mount namespace with a verified read-only bind mount.

eb_staged_rebuild.probe.filesystem(path)

Longest mount in /proc/mounts covering path. This host only.

eb_staged_rebuild.probe.probe_rename_exchange(directory)

Try renameat2(RENAME_EXCHANGE) on two temporary directories.

The result applies to the filesystem that holds directory. It is not a measurement of any other filesystem.

eb_staged_rebuild.probe.parse_framework_version(text)

eb_staged_rebuild.probe.probe_easybuild(eb='eb')

eb_staged_rebuild.probe.probe_all(directory, eb='eb')

eb_staged_rebuild.probe.stage_paths_from_bwrap_info(info, framework_version, bwrap_installpath)

Return (software, modules, how) or raise Refused.

bwrap_installpath_software is used when EasyBuild wrote it. EasyBuild 5.3.0 and 5.3.1 do not write it; their prepare_bwrap joins <bwrap_installpath>/software and <bwrap_installpath>/modules. That layout is used only for those two versions. Any other version without the key is refused, and the stage is left unpublished.

Swap

Crash-aware directory exchange with a restoring two-rename fallback.

The preferred operation is Linux renameat2(RENAME_EXCHANGE). Some shared filesystems reject rename flags, so the fallback uses three ordinary renames. Every caught failure in that sequence restores the two original names before it is re-raised. A journaled caller can use recover_interrupted_swap() after an uncatchable process or machine failure.

class eb_staged_rebuild.swap.SwapResult(method: str)

Bases: object

Parameters:

method (str)

method: str

eb_staged_rebuild.swap.recover_interrupted_swap(live, staged, rename=<built-in function rename>)

Restore an interrupted ordinary-rename fallback, if one is present.

eb_staged_rebuild.swap.swap_directories(live, staged, *, allow_exchange=True, faults=None, rename=<built-in function rename>, exchange_call=None, step_prefix='swap')

Exchange sibling directories and restore them after every caught failure.

staged contains the candidate tree before the call and the retired live tree after a successful call. Unsupported rename flags select the ordinary rename fallback. Any other exchange error leaves both names unchanged.

Journal

Crash-safe run journal. Readers only ever see a fully written JSON file.

eb_staged_rebuild.journal.journal_path(state_root, run_id)

eb_staged_rebuild.journal.run_dir(state_root, run_id)

eb_staged_rebuild.journal.generation_root(state_root, run_id)

eb_staged_rebuild.journal.save_journal(path, data, faults, step)

eb_staged_rebuild.journal.load_journal(path)

eb_staged_rebuild.journal.list_journals(state_root)

Module files

Read and rewrite the install-directory assignment in an EasyBuild module file.

EasyBuild 5.4 writes Lua as local root = "<installdir>" and Tcl as set root <installdir> (no quotes). Both expressions are matched the way easybuild/tools/module_generator.py matches them.

eb_staged_rebuild.modulefile.parse_root(text)

Return ("lua"|"tcl", path) or (None, None).

eb_staged_rebuild.modulefile.loads(text)

Dependency names from Lua load/depends_on and Tcl module load.

eb_staged_rebuild.modulefile.replace_path(text, old, new)

Replace old as a path token with new.

Existing occurrences of new are left in place, so a new path that contains the old path as a prefix is not rewritten twice.

eb_staged_rebuild.modulefile.rewrite_root(text, new_root)

Point the module file at new_root and return the new text.

eb_staged_rebuild.modulefile.module_identity(modules_root, module_path)

name/version from the last two path components of a module file.

eb_staged_rebuild.modulefile.generation_complete(path)

A generation is complete when it is a real directory with install metadata.

Accepted markers, any one of them:

• PAYLOAD (used by the fault-injection tests)

• easybuild/*.eb (EasyBuild copies the easyconfig in at the end of a build)

• .eb-staged-rebuild-complete (written by this tool before it publishes)

eb_staged_rebuild.modulefile.install_complete(path)

A staged install is complete when it is a real directory with build evidence.

Accepted, any one of them: PAYLOAD (the fault-injection builders) or a readable easybuild/*.eb (EasyBuild copies the easyconfig in at the end of a build). The tool’s own completion stamp is not evidence: a publication-preparation step that empties the tree leaves the stamp.

Discovery

Find module files EasyBuild wrote and decide which install directory they name.

eb_staged_rebuild.discover.walk_module_tree(modules_root)

Return (files, links).

files is a list of regular .lua and .tcl files. links is a list of (relative posix path, link text). Symlinked directories are not descended into.

eb_staged_rebuild.discover.generation_for(text, stage_software, live_software)

Return (generation_dir, module_text).

Install-path builds already name the stage directory. Bubblewrap builds name the live directory, because that is the path inside the bind mount; the text is rewritten to the staged directory, which is where the files are.

eb_staged_rebuild.discover.swap_install_for(text, stage_software, live_software)

Return the staged tree, stable live tree, and stable-path module text.

eb_staged_rebuild.discover.order_module_files(files, modules_root)

Publish dependencies first. A cycle falls back to path order for the rest.

eb_staged_rebuild.discover.map_link_target(target, stage_modules, live_modules)

Point a staged class symlink at the same relative path in the live tree.

Garbage collection

Remove hidden trees retired by successful swaps or explicit rollbacks.

The clock starts at publish or rollback. The default grace period is the unmeasured value carried by ns-051; operators must replace it with the site’s longest job plus a safety margin. Cluster-wide lsof is not used.

A module file moved aside by retire starts that same clock for the stable software directory the module named. gc reports that directory. It does not delete it: the directory is the path jobs already have open, and gc only removes a hidden sibling recorded by a directory swap.

eb_staged_rebuild.gc.collect(state_root, grace_hours, now, dry_run=False, include_failed=False, lock_timeout=None)

Collect retired trees while holding the same lock as publish/rollback.

Retire

Retire a module file whose package is published elsewhere.

retire moves one module file out of the module tree and into <state-root>/retired/<timestamp>/<absolute path without the leading /> with one same-filesystem os.replace. The software directory the module named is not touched. A job that already loaded the module keeps its paths. A job that loads the module afterwards no longer finds it in this tree and falls through to the next entry on MODULEPATH.

The record is a journal under <state-root>/runs/retire-<timestamp>/ in the same format as a run journal (kind is retire and modules_plan is empty). rollback RUN_ID puts the file back. gc reads the record to decide when the software directory is ready to remove.

eb_staged_rebuild.retire.is_retire(journal)

eb_staged_rebuild.retire.retired_dir(state_root)

eb_staged_rebuild.retire.retire(state_root, module_file, reason=None, dry_run=False, faults=None, clock=None, lock_timeout=0)

Move module_file into the state root and journal it. Returns the journal.

eb_staged_rebuild.retire.rollback_retire(state_root, journal, faults, clock)

Put a retired module file back and journal it. Caller holds the lock.

Lock

Serialize a state root with a local gate and a POSIX record lock.

The flock gate excludes threads and processes sharing a kernel. The distinct POSIX sidecar excludes nodes when the filesystem provides coherent record locks. Open the sidecar only while holding the gate: closing any descriptor for its inode drops this process’s record locks. Other code in this process must not open that sidecar. Release and close it before releasing the gate.

Never unlink either lock file while the state root is in use. Every node must use this protocol; flock-only holders on other nodes do not participate. The filesystem controls recovery after node failure. Fork children must close inherited descriptors without unlocking the parent’s open file description.

class eb_staged_rebuild.lockfile.FileLock(state_root)

Bases: object

One instance per acquirer; do not share an instance between threads.

acquire(timeout=None)

release()

Errors

exception eb_staged_rebuild.errors.StagedRebuildError

Bases: Exception

A failed run that left the live software directory alone.

exception eb_staged_rebuild.errors.Refused

Bases: StagedRebuildError

The operation was refused before it could empty or hide a live path.

exception eb_staged_rebuild.errors.NeedsIntervention

Bases: StagedRebuildError

The journal and the live module file disagree. Nothing further was changed.

Atomic writes

Same-directory writes. The destination name is replaced only after the bytes are durable.

class eb_staged_rebuild.atomicio.Metadata(info, acls)

Bases: tuple

acls

Alias for field number 1

info

Alias for field number 0

eb_staged_rebuild.atomicio.fsync_dir(path)

eb_staged_rebuild.atomicio.file_mode(path)

Mode for a new file: the existing file’s mode, otherwise 0644 masked by umask.

eb_staged_rebuild.atomicio.read_acls(fd)

Read listed and virtual ACLs through one regular-file descriptor.

GPFS can expose system.nfs4_acl through getxattr without listing it. Every known ACL name is probed; denied inspection and metadata changes refuse replacement. Two complete passes protect the snapshot within one ctime tick.

eb_staged_rebuild.atomicio.read_metadata(path)

eb_staged_rebuild.atomicio.atomic_write(path, data, faults, step, mode=None, metadata_from=None)

Replace a destination only after its bytes and access metadata are durable.

Existing files retain their ownership, mode and listed ACL attributes (POSIX access, generic NFSv4 and GPFS native NFSv4), including the absence of an ACL. metadata_from selects a prepared permission policy instead. New files without a metadata source inherit the directory’s default ACL. Metadata that cannot be inspected, restored byte for byte, or that changes before the rename leaves the destination unchanged.

Fault points <step>.write, <step>.fsync and <step>.rename precede those operations. The destination is never unlinked first.

eb_staged_rebuild.atomicio.read_bytes(path)

eb_staged_rebuild.atomicio.sha256_bytes(data)

Paths

Path checks that do not follow symlinks.

eb_staged_rebuild.paths.abspath(path)

eb_staged_rebuild.paths.is_within(child, parent)

eb_staged_rebuild.paths.lstat_kind(path)

missing, symlink, dir, file, or other.

eb_staged_rebuild.paths.is_real_dir(path)

eb_staged_rebuild.paths.assert_real_components(path, stop_at=None)

Return a refusal reason if any existing component of path is a symlink.

stop_at is an ancestor that is not itself checked (used when the caller already accepted the software root). None checks every existing component.

Build events

Opt-in ledger events around one software-build command.

A caller names one operation with explicit metadata and a ledger configuration, then hands over the command. The started record must be acknowledged by eb-stack ledger append before the command runs; with required false the command runs anyway and the record stays pending. After the command a finished record (exit status 0 and every post-build check passed) or an aborted record (anything else) follows.

Records are queued durably in <outbox>/<operation>/outbox.json before delivery and are delivered in order through the bounded adapter in ledger_adapter. Each record keeps its bytes and its build.event_id until the ledger returns an exact receipt, so a retry after a lost acknowledgement is acknowledged once by eb-stack. A retry never repeats a command that already ran: its recorded result is returned instead.

Only the allowlisted metadata below is serialized. The environment, the command and its arguments are never written to the outbox or the ledger.

Use run_build and deliver from the main thread; the command runs in its own process group, which is stopped before either returns.

Command line:

python3 -m eb_staged_rebuild.build_events --config C.json --metadata M.json -- CMD ARG...
python3 -m eb_staged_rebuild.build_events --config C.json --metadata M.json --deliver

class eb_staged_rebuild.build_events.Outcome(returncode, phase, command_status, pending, message)

Bases: tuple

command_status

Alias for field number 2

message

Alias for field number 4

pending

Alias for field number 3

phase

Alias for field number 1

returncode

Alias for field number 0

eb_staged_rebuild.build_events.check_config(config)

Normalize a configuration object; refuse unsafe or unknown settings.

eb_staged_rebuild.build_events.load_config(path)

eb_staged_rebuild.build_events.check_metadata(metadata)

Normalize caller metadata. Fields outside the schema are refused.

eb_staged_rebuild.build_events.load_metadata(path)

eb_staged_rebuild.build_events.event_id(operation, stage)

eb_staged_rebuild.build_events.target_of(meta)

eb_staged_rebuild.build_events.started_record(meta, now, previous_modulefile=None)

eb_staged_rebuild.build_events.terminal_record(meta, now, status, started_id, previous_modulefile=None)

finished for a zero exit whose outputs check, aborted otherwise.

eb_staged_rebuild.build_events.run_build(config, metadata, argv, faults=None, clock=<built-in function time>, env=None)

Record started, run argv, then record its result. Returns an Outcome.

A repeated call for an operation whose command already ran delivers what is pending and returns the recorded result without running argv.

eb_staged_rebuild.build_events.deliver(config, metadata, faults=None, clock=<built-in function time>)

Deliver pending events of a recorded operation without running anything.

eb_staged_rebuild.build_events.main(argv=None)

Ledger adapter

Bounded adapter for eb-stack ledger append.

The only coupling to eb-stack is its command line and the first line it prints, the receipt; the tile log and checkpoint lines after it are not read. A receipt is returned only when the command exits 0, its summary parses, and the head file it wrote names the reported size and root. With public_key the receipt also needs eb-stack ledger verify to prove the record is in that signed head. Any other outcome is ambiguous: the record may or may not be in the log, so the caller retries the same record and build.event_id makes the retry idempotent.

eb_staged_rebuild.ledger_adapter.publication_records(config, journal, stage, now, corrects=None)

One ledger record per published module file.

module and location name the stable software/<name>/<version> directory. That is the path a directory swap publishes and the path an RPATH built for the live prefix contains. The stage and the hidden retired sibling are not the published location.

corrects maps a module file to the ledger id of its finished record; the records are then the correction stages of a rollback.

eb_staged_rebuild.ledger_adapter.load_config(path)

class eb_staged_rebuild.ledger_adapter.EbStackLedger(config)

Bases: object

append(record_path)

Ledger maintenance

Readers and maintenance tools for an eb-stack ledger.

Three local file tools over the log that eb-stack ledger append writes:

• backfill appends finding records for software observed at an explicit time, from an operator-written manifest pinned by its SHA-256.

• anchor signs the day’s head with eb-stack ledger head and adds it to a directory of immutable daily anchors after eb-stack ledger verify proves it extends the latest anchor.

• project writes per-package JSON and Markdown history from a log whose head eb-stack ledger verify accepted.

Every signature comes from eb-stack ledger head or ledger append and every proof from eb-stack ledger verify, run through the adapter’s bounded runner. The log and head files are read here only to refuse bad input early and to bind what is projected to the bytes the CLI verified: the bytes are read once, copied into a private directory, verified there, and parsed from the same buffer.

exception eb_staged_rebuild.ledger_maintenance.VerificationError

Bases: Refused

eb-stack ledger verify did not accept the log, a head or a proof.

exception eb_staged_rebuild.ledger_maintenance.StaleHead

Bases: VerificationError

The signed head covers fewer records than the log holds.

exception eb_staged_rebuild.ledger_maintenance.Equivocation

Bases: Refused

Two signed heads disagree about the same history.

eb_staged_rebuild.ledger_maintenance.parse_head(data, what='head')

A signed head file exactly as eb-stack writes it.

eb_staged_rebuild.ledger_maintenance.parse_log(data, what='ledger')

Records of a ledger log, each line checked against the ledger format.

eb-stack ledger verify remains the authority; this refuses early, and a log ending in a partial line is refused rather than repaired.

class eb_staged_rebuild.ledger_maintenance.VerifiedLog(head, records)

Bases: object

Records and the signed head eb-stack ledger verify accepted for them.

eb_staged_rebuild.ledger_maintenance.read_verified(eb_stack, log, head, public_key, timeout=120)

Read log and head once and have eb-stack verify those bytes.

The head must be signed by public_key and cover every record: a head that covers fewer is stale and refused, so nothing unverified is read.

eb_staged_rebuild.ledger_maintenance.load_manifest(path, expected_sha256, now)

An observed-history manifest, refused unless its bytes hash to expected_sha256.

eb_staged_rebuild.ledger_maintenance.backfill_records(manifest, digest)

One finding record per observation, stamped with the observation time.

Nothing claims an install or a build happened: the record says the software was seen at observed_at and names where the evidence is.

eb_staged_rebuild.ledger_maintenance.backfill(eb_stack, log, key, head, public_key, manifest_path, manifest_sha256, timeout=120, faults=None, clock=<built-in function time>)

Append the manifest’s observations, each at most once.

Every record is built from the manifest alone, so a retry sends the same bytes and eb-stack acknowledges it as already recorded. The log is checked first: a manifest id already backfilled from other bytes, or an event id recorded with other content, is refused before anything is appended. Each receipt needs eb-stack ledger verify to prove the record is in the signed head written by the append.

eb_staged_rebuild.ledger_maintenance.list_anchors(anchors)

Anchors by day, each a single-link regular file whose head is signed that day.

eb_staged_rebuild.ledger_maintenance.anchor(eb_stack, log, key, public_key, anchors, timeout=120, faults=None, clock=<built-in function time>, lock_timeout=60)

Publish today’s signed head as <anchors>/<YYYY-MM-DD>.head.json, once.

The head is signed now, for the day it is signed on: no anchor is made for another day. It must bind to the log and extend the latest anchor, both proved by eb-stack ledger verify, before it is linked into place. An existing file is never replaced: when today’s anchor exists the call proves it still extends into the log and reports it.

eb_staged_rebuild.ledger_maintenance.verify_anchors(eb_stack, log, public_key, anchors, timeout=120)

Prove every anchor extends the one before it and the latest binds to the log.

eb_staged_rebuild.ledger_maintenance.package_of(module)

eb_staged_rebuild.ledger_maintenance.build_history(verified)

The deterministic history model of a verified log, grouped by package.

eb_staged_rebuild.ledger_maintenance.file_stem(name)

A file name for a package that cannot leave the output directory.

eb_staged_rebuild.ledger_maintenance.md(value)

Text safe in a Markdown table cell: no markup, HTML, line breaks or bidi controls.

eb_staged_rebuild.ledger_maintenance.render_package(model, package)

eb_staged_rebuild.ledger_maintenance.render(model)

Every output file, by path relative to the output directory.

eb_staged_rebuild.ledger_maintenance.project(eb_stack, log, head, public_key, out, timeout=120, faults=None)

Write JSON and Markdown history of every package in a verified log.

The same verified log and head always give the same bytes. Package files are replaced one at a time and the index last, so an interrupted run leaves the previous index; files of packages no longer listed are removed.

eb_staged_rebuild.ledger_maintenance.main(argv=None)

Outbox

Local outbox of publication events, one file per run.

Records are written durably before any delivery and are delivered one at a time in the order they were queued. An event is acknowledged only with the publisher’s receipt. A failed or ambiguous delivery leaves the event pending with its record unchanged, so the retry carries the same build.event_id and the same bytes, which the publisher acknowledges at most once.

exception eb_staged_rebuild.outbox.PublisherError

Bases: Exception

The publisher gave no receipt. The record may or may not be published.

eb_staged_rebuild.outbox.outbox_path(run_dir)

eb_staged_rebuild.outbox.load(run_dir)

eb_staged_rebuild.outbox.pending(data)

eb_staged_rebuild.outbox.enqueue(run_dir, records, faults, step)

Queue records whose build.event_id is new. Durable when this returns.

eb_staged_rebuild.outbox.deliver(run_dir, publisher, faults)

Deliver pending events in order and stop at the first without a receipt.

Returns the events still pending. Fault points: ledger.append:<id> before the publisher runs and ledger.acked:<id> after its receipt, before the receipt is saved.

User-namespace helper

Enter a user/mount-namespace sandbox and make the live trees read-only.

eb_staged_rebuild.userns_sandbox.main(argv=None)

Cleanup, live check, faults, constants

Remove an owned generation tree without following links or changing live data.

eb_staged_rebuild.cleanup.remove_generation_tree(path, boundary)

A before/after content record of a live install directory.

eb_staged_rebuild.livecheck.snapshot(path)

eb_staged_rebuild.livecheck.same_snapshot(before, after)

eb_staged_rebuild.livecheck.empty_real_dir(path)

exception eb_staged_rebuild.faults.InjectedCrash(step)

Bases: BaseException

Test-only crash. A BaseException so except Exception does not swallow it.

class eb_staged_rebuild.faults.FaultInjector(fail_at=None)

Bases: object

Record each named step and, when asked, raise before that step runs.

reach(step)

Named policy values and their sources.