Recover and roll back¶

status reads journals. resume continues a build or an interrupted publish. rollback puts a published generation back, or puts a retired module file back. gc deletes a hidden sibling after the grace period. The commands below were run against /tmp/ebsr-docs-prefix.

Read the journal¶

eb-staged-rebuild status --state-root "$STATE"
eb-staged-rebuild status --state-root "$STATE" demo-run

After the failed build, before resume:

demo-run  failed  backend=installpath  modules=0/0
  eb exited 3; live modules were not changed

The live PAYLOAD was still payload-v1. A failed build that exits before planning has an empty module plan, so the count is 0/0.

status does not take the state-root lock and does not accept --lock-timeout. One unreadable journal stops the listing.

Resume a failed build¶

eb-staged-rebuild resume --state-root "$STATE" demo-run

eb-staged-rebuild: resume demo-run from failed: rebuilding the unpublished generation
eb-staged-rebuild: swapped all/demo/1.lua (renameat2-exchange)
eb-staged-rebuild: published module all/demo/1.lua
eb-staged-rebuild: run demo-run published 1 module(s)

resume from created, building, or failed rebuilds the generation when nothing has been published. resume from built or publishing publishes the generation that is already recorded. A complete run is a no-op. A rolled-back run is refused: start a new run id. A retire id is refused:

eb-staged-rebuild: run retire-20261110T130126Z is a retire record; rollback restores the module file

That command exited 2.

Interrupted ordinary-rename fallback¶

When renameat2(RENAME_EXCHANGE) returns EINVAL, ENOSYS, or EOPNOTSUPP, publication uses three renames. The hold name is the staged path plus .fallback-old. Any other errno from the exchange leaves both names as they were and does not select the fallback.

A caught error during the three renames restores the original two names before it is raised. A SIGKILL or a machine loss can stop between those renames. The next resume calls recover_interrupted_swap, which recognises the two interrupted shapes (live missing and the hold present, or live present and the staged name missing) and restores the old live name before another attempt. Leave a path ending in .fallback-old for resume. That recovery repairs the journaled names. The window where one name is missing is closed only by RENAME_EXCHANGE.

Roll a published generation back¶

eb-staged-rebuild rollback --state-root "$STATE" demo-run

eb-staged-rebuild: restored all/demo/1.lua
eb-staged-rebuild: run demo-run rolled back; replacement kept at its retired path

The live PAYLOAD was payload-v1 again. Status:

demo-run  rolled_back  backend=installpath  modules=0/1
  rolled back
  restored   all/demo/1.lua  live=/tmp/ebsr-docs-prefix/software/demo/1 retired=/tmp/ebsr-docs-prefix/software/demo/.1.demo-run

Rollback walks the module plan in reverse. It exchanges each retired directory back onto the stable path, restores each module file from the backup taken at publish, and removes symlinks this run created. A second rollback of the same id prints:

eb-staged-rebuild: run demo-run is already rolled back

If rollback fails partway, the tool exchanges back to the published generation and rewrites the published module files, then re-raises the error. The published tree is what a retry of rollback starts from.

Only a journal in state complete can roll back. The replacement tree stays at the hidden sibling so a later gc can remove it.

Collect hidden siblings¶

The default grace is 144 hours: the notes’ 120-hour walltime plus 24 hours. That walltime was not measured for a deployment site. Set --grace-hours from the site’s longest job, or set EB_STAGED_REBUILD_GRACE_HOURS.

eb-staged-rebuild gc --state-root "$STATE" --grace-hours 144 --dry-run

keep: /tmp/ebsr-docs-prefix/software/demo/.1.demo-run (within grace (rolled-back candidate))

--include-failed on the same tree printed the same keep line. The clock for a rolled-back run starts at rolled_back_at. The clock for a complete run starts at published_at. A failed run is eligible only with --include-failed, and only when a module is still pending and the hidden sibling exists.

After the grace period, --dry-run lists remove. On the same prefix, once a second rolled-back run (demo-prepare) was also present:

eb-staged-rebuild gc \
  --state-root /tmp/ebsr-docs-prefix/.eb-staged-rebuild \
  --grace-hours 0 \
  --dry-run

remove: /tmp/ebsr-docs-prefix/software/demo/.1.demo-prepare (rolled-back candidate)
remove: /tmp/ebsr-docs-prefix/software/demo/.1.demo-run (rolled-back candidate)

Drop --dry-run only after those paths are the ones to delete. Collection deletes the exact hidden sibling recorded beside its stable directory. It uses owned-directory removal on that sibling’s filesystem. It does not call lsof and cannot see a file that is open on another node. A stable directory named by retire is reported as keep or manual and is left in place. See Retire a module published elsewhere.

Lock timeout¶

run, resume, rollback, gc, and retire take <state-root>/.lock and then <state-root>/.lock.posix. Omit --lock-timeout and the wait has no deadline, except retire, whose default is 0 (refuse at once).

While another process held the lock, this command exited 2:

eb-staged-rebuild gc \
  --state-root /tmp/ebsr-docs-prefix/.eb-staged-rebuild \
  --grace-hours 144 \
  --dry-run \
  --lock-timeout 0.2

eb-staged-rebuild: another eb-staged-rebuild holds the lock at /tmp/ebsr-docs-prefix/.eb-staged-rebuild/.lock