Module pack 

One file for every store path: a packed TileStore, so a log costs a constant number of inodes however long its history.

The pack is an append-only file. Each record carries a store path and either its bytes, a deletion, or a rename, and ends in a SHA-256 over the record, so a torn tail is found and ignored. Bytes that shrink under DEFLATE are stored compressed, record by record; hash tiles do not shrink and are stored as they are. A path’s current bytes are the last record that names it, so PackStore::compact rewrites the pack with one record per live path and renames it over the old one.

A compaction also writes an index as the pack’s first record: every live path with where its body sits, and the length of the pack the index covers. Opening such a pack reads the index and then only the records appended after the region it covers, rather than every record; a record the index named is checked against its digest when it is read. Packs written by format v1 (no index) open by reading every record, and become v2 at their next compaction.

Writers that share a pack (PackStore::open_shared) take <pack>.lock for each change rather than for their lifetime: under the lock a writer reads what the others appended, drops a torn tail a crashed writer left, applies its change and releases. A shared writer and an exclusive one (PackStore::open) still exclude each other through the same lock file.

One writer at a time: the writer holds <pack>.lock, created exclusively and removed when the store is dropped, the way the tile log’s own locks work; a crash leaves it, and PackStore::break_lock removes it once the operator knows no writer is alive. Readers open the pack without a lock and see every record whose digest holds; a reader re-reads the tail before each operation, so it sees what a writer has appended since, and reopens the pack after a compaction.

Structs

Compacted

What a compaction did.

PackStore

A TileStore over one append-only file.

Constants

PACK_MAGIC

The first bytes of a pack file as this version writes it (format v2, whose first record may be an index).

PACK_MAGIC_V1

The first bytes of a pack file in format v1, which still opens.

SHARED_LOCK_WAIT

How long a shared writer waits for <pack>.lock before giving up.