Rotate the log tree#
Compress and delete the old run directories under the log root with eb-keep-logs rotate, at the end of each run, from cron, or both, without touching the logs of a run that is still writing them.
Before you start#
eb-keep-logs on the
PATHof the build account, or named in[site.tools.eb_keep_logs]bin. A version whoseeb-keep-logs --helplists--lock-timeouttakes a lock of its own; eb-keep-logs 0.2.1 takes none.A site file with a
[logs]table, as in Write a site file for your cluster.For a rotation from cron, the crontab of the build account, and
flockfrom util-linux on the host that runs it.
The output on this page comes from docs/examples/rotation.sh, which runs a site file with the local scheduler and a stand-in for eb-keep-logs that prints its arguments.
Steps#
Add a
[logs.rotate]table to the site file, with the age in days at which eb-keep-logs compresses a run directory and, if you want, the age at which it deletes one:[logs.rotate] compress_after_days = 30 delete_after_days = 365
Check the rotation that a run starts at its end:
eb-pipeline run --site site.toml --list build.list --target cpu --build-type production --dry-run | grep '^\[logs\]'INFO [run] cpu: dry run: nothing executed [logs] TZ=UTC eb-keep-logs rotate $WORK/logs --compress-after 30 --delete-after 365 --lock-timeout 0
TZ=UTCmakes eb-keep-logs count the age of a run from the midnight of its log stamp in Coordinated Universal Time (UTC), as the stamp is. A run leaves out--lock-timeout 0wheneb-keep-logs --helpdoes not list it.Run as before. A run takes
LOG_ROOT/.eb-pipeline-rotate.lockshared while it writes logs, and rotates at its end only when it can take that lock alone, so a rotation never compresses the logs of a live run; while other runs are live, the last of them to end rotates.To rotate also when no run ends, as on a quiet day, have cron start
eb-keep-logs rotateunder the same lock, withflock:15 3 * * * TZ=UTC flock -w 3600 -E 75 /opt/site/logs/.eb-pipeline-rotate.lock eb-keep-logs rotate /opt/site/logs --compress-after 30 --delete-after 365flockwaits up to an hour for the live runs and a rotation that holds the lock to end, then holds it alone whileeb-keep-logs rotateruns, and a run that starts meanwhile waits for it. When the hour runs out, it exits 75 without startingeb-keep-logs. Take the lock first and eb-keep-logs’ own lock second, asflockaroundeb-keep-logs rotatedoes: The log tree and its locks gives the order.
Check that it worked#
After a run, the logs phase of its result file is ok, and rotate.log in its log directory holds what eb-keep-logs printed, here the stand-in’s arguments:
grep -B1 -A1 '"name": "logs"' result.json
cat logs/20261117_090000/cpu/rotate.log
"exit_code": 0,
"name": "logs",
"status": "ok"
eb-keep-logs rotate $WORK/logs --compress-after 30 --delete-after 365 --lock-timeout 0
Run the cron line once by hand, here from the parent of the log root logs; exit 0, after the line that the stand-in printed, means that eb-keep-logs rotated:
TZ=UTC flock -w 3600 -E 75 logs/.eb-pipeline-rotate.lock eb-keep-logs rotate logs --compress-after 30 --delete-after 365; echo "exit $?"
eb-keep-logs rotate logs --compress-after 30 --delete-after 365
exit 0
While a run under the log root is live, the same line with a wait of one second gives up, and starts nothing:
TZ=UTC flock -w 1 -E 75 logs/.eb-pipeline-rotate.lock eb-keep-logs rotate logs --compress-after 30 --delete-after 365; echo "exit $?"
exit 75
If it fails#
A run that does not rotate leaves its logs phase skipped, with the reason in its note, and keeps its exit status; only a rotation that fails changes it.
Message |
Cause |
What to do |
|---|---|---|
|
Another run under the log root was live when this one ended. |
Nothing: the run that ends alone rotates. On a log root that always has a live run, rotate from cron. |
|
The run started while another run, or the cron line, rotated. |
Nothing: the run goes on when the rotation ends. |
|
Another |
Nothing: that rotate does the work. |
|
eb-keep-logs 0.2.1, which refuses |
Rotate from cron only with the |
|
|
Read |
|
The run cannot take its rotation lock, here because the path is a directory, and rotates without it. |
Make the path a file that the build account can write, on a filesystem that supports |
Exit 75 from the cron line |
Runs under the log root kept the lock for the hour, or another rotate held eb-keep-logs’ lock. |
Nothing: the tree was busy and nothing changed; the next start tries again. |