rfc9162-tiles
rfc9162-tiles stores an RFC 9162 log as tlog tiles. Writer appends. Reader returns entries and proofs that rfc9162 checks. PosixStore is a directory. PackStore is one file. MemoryStore keeps the files in the process.
Writer::open and Reader::open take a key function. no_keys adds nothing beyond the identity hash SHA-256(entry), which every entry has. The writer and the reader of one log use the same function.
Append to a directory and read it back
//! A directory log: append, read, prove, look up by the identity hash.
use rfc9162_tiles::index::{identity, no_keys};
use rfc9162_tiles::{PosixStore, Reader, TileStore, Writer};
fn main() {
let dir = tempfile::tempdir().expect("temp dir");
let store = PosixStore::open(dir.path()).expect("store");
let mut writer = Writer::open(store.clone(), no_keys()).expect("writer");
let first = writer.append(b"first record").expect("append");
let second = writer.append(b"second record").expect("append");
println!("appended {first} and {second}");
println!(
"size {} root {}",
writer.size(),
rfc9162::hex(&writer.root())
);
drop(writer);
let reader = Reader::open(store.clone(), no_keys()).expect("reader");
let record = reader.get(first).expect("get").expect("present");
println!(
"get {first} = {}",
String::from_utf8(record).expect("utf-8")
);
let proof = reader
.inclusion(second)
.expect("inclusion")
.expect("in tree");
println!(
"inclusion {second} {}",
rfc9162::verify_inclusion(
&rfc9162::leaf_hash(b"second record"),
usize::try_from(second).unwrap(),
usize::try_from(reader.size()).unwrap(),
&proof,
&rfc9162::hex(&reader.root()),
)
);
let hits = reader.find(&identity(b"first record")).expect("find");
println!("find identity of first record -> {hits:?}");
let missing = reader.find(&identity(b"absent")).expect("find");
println!("find identity of absent -> {missing:?}");
let mut files = store.list("").expect("list");
files.retain(|path| !path.ends_with(".lock"));
println!("files");
for path in files {
println!(" {path}");
}
}//! A directory log: append, read, prove, look up by the identity hash.
use rfc9162_tiles::index::{identity, no_keys};
use rfc9162_tiles::{PosixStore, Reader, TileStore, Writer};
fn main() {
let dir = tempfile::tempdir().expect("temp dir");
let store = PosixStore::open(dir.path()).expect("store");
let mut writer = Writer::open(store.clone(), no_keys()).expect("writer");
let first = writer.append(b"first record").expect("append");
let second = writer.append(b"second record").expect("append");
println!("appended {first} and {second}");
println!(
"size {} root {}",
writer.size(),
rfc9162::hex(&writer.root())
);
drop(writer);
let reader = Reader::open(store.clone(), no_keys()).expect("reader");
let record = reader.get(first).expect("get").expect("present");
println!(
"get {first} = {}",
String::from_utf8(record).expect("utf-8")
);
let proof = reader
.inclusion(second)
.expect("inclusion")
.expect("in tree");
println!(
"inclusion {second} {}",
rfc9162::verify_inclusion(
&rfc9162::leaf_hash(b"second record"),
usize::try_from(second).unwrap(),
usize::try_from(reader.size()).unwrap(),
&proof,
&rfc9162::hex(&reader.root()),
)
);
let hits = reader.find(&identity(b"first record")).expect("find");
println!("find identity of first record -> {hits:?}");
let missing = reader.find(&identity(b"absent")).expect("find");
println!("find identity of absent -> {missing:?}");
let mut files = store.list("").expect("list");
files.retain(|path| !path.ends_with(".lock"));
println!("files");
for path in files {
println!(" {path}");
}
}Drop the writer before opening another one. The writer holds .state/treeState.lock until drop. A second Writer::open while that file exists returns Error::Locked.
find takes a key. The identity hash finds the entry that hashed to it. An identity that no entry has returns an empty list.
//! A directory log: append, read, prove, look up by the identity hash.
use rfc9162_tiles::index::{identity, no_keys};
use rfc9162_tiles::{PosixStore, Reader, TileStore, Writer};
fn main() {
let dir = tempfile::tempdir().expect("temp dir");
let store = PosixStore::open(dir.path()).expect("store");
let mut writer = Writer::open(store.clone(), no_keys()).expect("writer");
let first = writer.append(b"first record").expect("append");
let second = writer.append(b"second record").expect("append");
println!("appended {first} and {second}");
println!(
"size {} root {}",
writer.size(),
rfc9162::hex(&writer.root())
);
drop(writer);
let reader = Reader::open(store.clone(), no_keys()).expect("reader");
let record = reader.get(first).expect("get").expect("present");
println!(
"get {first} = {}",
String::from_utf8(record).expect("utf-8")
);
let proof = reader
.inclusion(second)
.expect("inclusion")
.expect("in tree");
println!(
"inclusion {second} {}",
rfc9162::verify_inclusion(
&rfc9162::leaf_hash(b"second record"),
usize::try_from(second).unwrap(),
usize::try_from(reader.size()).unwrap(),
&proof,
&rfc9162::hex(&reader.root()),
)
);
let hits = reader.find(&identity(b"first record")).expect("find");
println!("find identity of first record -> {hits:?}");
let missing = reader.find(&identity(b"absent")).expect("find");
println!("find identity of absent -> {missing:?}");
let mut files = store.list("").expect("list");
files.retain(|path| !path.ends_with(".lock"));
println!("files");
for path in files {
println!(" {path}");
}
}Two entries produce a partial level-0 tile and a partial bundle, both of width 2:
//! A directory log: append, read, prove, look up by the identity hash.
use rfc9162_tiles::index::{identity, no_keys};
use rfc9162_tiles::{PosixStore, Reader, TileStore, Writer};
fn main() {
let dir = tempfile::tempdir().expect("temp dir");
let store = PosixStore::open(dir.path()).expect("store");
let mut writer = Writer::open(store.clone(), no_keys()).expect("writer");
let first = writer.append(b"first record").expect("append");
let second = writer.append(b"second record").expect("append");
println!("appended {first} and {second}");
println!(
"size {} root {}",
writer.size(),
rfc9162::hex(&writer.root())
);
drop(writer);
let reader = Reader::open(store.clone(), no_keys()).expect("reader");
let record = reader.get(first).expect("get").expect("present");
println!(
"get {first} = {}",
String::from_utf8(record).expect("utf-8")
);
let proof = reader
.inclusion(second)
.expect("inclusion")
.expect("in tree");
println!(
"inclusion {second} {}",
rfc9162::verify_inclusion(
&rfc9162::leaf_hash(b"second record"),
usize::try_from(second).unwrap(),
usize::try_from(reader.size()).unwrap(),
&proof,
&rfc9162::hex(&reader.root()),
)
);
let hits = reader.find(&identity(b"first record")).expect("find");
println!("find identity of first record -> {hits:?}");
let missing = reader.find(&identity(b"absent")).expect("find");
println!("find identity of absent -> {missing:?}");
let mut files = store.list("").expect("list");
files.retain(|path| !path.ends_with(".lock"));
println!("files");
for path in files {
println!(" {path}");
}
}appended 0 and 1
size 2 root 7456f1d72b3956155ce6655bfcd08decca38ec102bc2dea97d4c1070e8dd625f
get 0 = first record
inclusion 1 true
find identity of first record -> [0]
find identity of absent -> []
files
.state/treeState
tile/0/000.p/2
tile/entries/000.p/2
Keep the log in one pack file
PackStore::open creates an append-only file and holds <pack>.lock until the store drops. Writer and Reader can borrow that store. compact rewrites the file so each live path is one record. For two entries the live paths are the tree state, the partial hash tile and the partial bundle.
open_shared takes the pack lock for each change, so more than one process can append. The snippet opens a second pack that way and appends one record.
//! One pack file for the same tile log, then a compaction of that file.
use rfc9162_tiles::index::no_keys;
use rfc9162_tiles::{PackStore, Reader, Writer};
fn main() {
let dir = tempfile::tempdir().expect("temp dir");
let path = dir.path().join("log.pack");
let store = PackStore::open(&path).expect("pack");
{
let mut writer = Writer::open(&store, no_keys()).expect("writer");
writer.append(b"one").expect("append");
writer.append(b"two").expect("append");
println!(
"size {} root {}",
writer.size(),
rfc9162::hex(&writer.root())
);
}
let reader = Reader::open(&store, no_keys()).expect("reader");
let second = reader.get(1).expect("get").expect("present");
println!("get 1 = {}", String::from_utf8(second).expect("utf-8"));
println!("live bytes {}", store.live_bytes().expect("live"));
let compacted = store.compact().expect("compact");
println!(
"compact after {} paths {}",
compacted.after, compacted.paths
);
let shared_path = dir.path().join("shared.pack");
let shared = PackStore::open_shared(&shared_path).expect("shared");
let mut writer = Writer::open(&shared, no_keys()).expect("writer");
let index = writer.append(b"shared").expect("append");
println!("shared append {index}");
}//! One pack file for the same tile log, then a compaction of that file.
use rfc9162_tiles::index::no_keys;
use rfc9162_tiles::{PackStore, Reader, Writer};
fn main() {
let dir = tempfile::tempdir().expect("temp dir");
let path = dir.path().join("log.pack");
let store = PackStore::open(&path).expect("pack");
{
let mut writer = Writer::open(&store, no_keys()).expect("writer");
writer.append(b"one").expect("append");
writer.append(b"two").expect("append");
println!(
"size {} root {}",
writer.size(),
rfc9162::hex(&writer.root())
);
}
let reader = Reader::open(&store, no_keys()).expect("reader");
let second = reader.get(1).expect("get").expect("present");
println!("get 1 = {}", String::from_utf8(second).expect("utf-8"));
println!("live bytes {}", store.live_bytes().expect("live"));
let compacted = store.compact().expect("compact");
println!(
"compact after {} paths {}",
compacted.after, compacted.paths
);
let shared_path = dir.path().join("shared.pack");
let shared = PackStore::open_shared(&shared_path).expect("shared");
let mut writer = Writer::open(&shared, no_keys()).expect("writer");
let index = writer.append(b"shared").expect("append");
println!("shared append {index}");
}size 2 root 4f55f619d9215235778b2b9f17d6f4915b16171214d152381293669764de722e
get 1 = two
live bytes 532
compact after 532 paths 3
shared append 0
compact on this pack keeps 3 paths and leaves the file at 532 bytes. The length before compact includes a deleted writer lock whose body is the process id, so that length changes from run to run. Readers open with PackStore::open_read, which leaves <pack>.lock free.
Sign a checkpoint
Signer::new takes the origin and an Ed25519 key. The origin is the key name: non-empty, no spaces, no +. Writer::checkpoint signs the current size and root and writes the note to checkpoint. A published checkpoint for a larger size is left in place.
checkpoint::verify checks the note against trusted log keys. Cosigner appends a cosignature/v1 line. verify_cosignatures checks those lines. The timestamp below is the Unix time 1700000000.
//! Sign a checkpoint, check it, and add a witness cosignature.
use ed25519_dalek::SigningKey;
use rfc9162_tiles::checkpoint::{verify, verify_cosignatures};
use rfc9162_tiles::index::no_keys;
use rfc9162_tiles::{Cosigner, MemoryStore, Signer, Writer};
fn main() {
let signer =
Signer::new("example.org/log", SigningKey::from_bytes(&[7u8; 32])).expect("signer");
let mut writer = Writer::open(MemoryStore::new(), no_keys()).expect("writer");
writer.append(b"alpha").expect("append");
writer.append(b"beta").expect("append");
let note = writer
.checkpoint(&signer, "example.org/log")
.expect("checkpoint");
print!("{}", note.to_text());
let trusted = [signer.verifier()];
let verified = verify(¬e, &trusted).expect("verify");
println!("verified {}", verified[0].name());
let witness =
Cosigner::new("example.org/witness", SigningKey::from_bytes(&[9u8; 32])).expect("witness");
let mut note = note;
witness.cosign(&mut note, 1_700_000_000);
let lines = verify_cosignatures(¬e, &[witness.verifier()]).expect("cosignatures");
println!(
"cosignatures {} timestamp {}",
lines.len(),
lines[0].timestamp
);
println!("verifier {}", signer.verifier().to_text());
}//! Sign a checkpoint, check it, and add a witness cosignature.
use ed25519_dalek::SigningKey;
use rfc9162_tiles::checkpoint::{verify, verify_cosignatures};
use rfc9162_tiles::index::no_keys;
use rfc9162_tiles::{Cosigner, MemoryStore, Signer, Writer};
fn main() {
let signer =
Signer::new("example.org/log", SigningKey::from_bytes(&[7u8; 32])).expect("signer");
let mut writer = Writer::open(MemoryStore::new(), no_keys()).expect("writer");
writer.append(b"alpha").expect("append");
writer.append(b"beta").expect("append");
let note = writer
.checkpoint(&signer, "example.org/log")
.expect("checkpoint");
print!("{}", note.to_text());
let trusted = [signer.verifier()];
let verified = verify(¬e, &trusted).expect("verify");
println!("verified {}", verified[0].name());
let witness =
Cosigner::new("example.org/witness", SigningKey::from_bytes(&[9u8; 32])).expect("witness");
let mut note = note;
witness.cosign(&mut note, 1_700_000_000);
let lines = verify_cosignatures(¬e, &[witness.verifier()]).expect("cosignatures");
println!(
"cosignatures {} timestamp {}",
lines.len(),
lines[0].timestamp
);
println!("verifier {}", signer.verifier().to_text());
}The verifier line is what Reader::open_checkpoint and the sidecar’s --trusted flag parse: name+hex-key-id+base64(type || public key).
//! Sign a checkpoint, check it, and add a witness cosignature.
use ed25519_dalek::SigningKey;
use rfc9162_tiles::checkpoint::{verify, verify_cosignatures};
use rfc9162_tiles::index::no_keys;
use rfc9162_tiles::{Cosigner, MemoryStore, Signer, Writer};
fn main() {
let signer =
Signer::new("example.org/log", SigningKey::from_bytes(&[7u8; 32])).expect("signer");
let mut writer = Writer::open(MemoryStore::new(), no_keys()).expect("writer");
writer.append(b"alpha").expect("append");
writer.append(b"beta").expect("append");
let note = writer
.checkpoint(&signer, "example.org/log")
.expect("checkpoint");
print!("{}", note.to_text());
let trusted = [signer.verifier()];
let verified = verify(¬e, &trusted).expect("verify");
println!("verified {}", verified[0].name());
let witness =
Cosigner::new("example.org/witness", SigningKey::from_bytes(&[9u8; 32])).expect("witness");
let mut note = note;
witness.cosign(&mut note, 1_700_000_000);
let lines = verify_cosignatures(¬e, &[witness.verifier()]).expect("cosignatures");
println!(
"cosignatures {} timestamp {}",
lines.len(),
lines[0].timestamp
);
println!("verifier {}", signer.verifier().to_text());
}example.org/log
2
mDy1fATN3VJjTtqzinvvhXCKl08RS72aqexdTOZla0s=
— example.org/log XizvALF5Epy36TLnS5yEwMRpxemNpzlbTZZc1scO6QbldBRA0HpqXVW46JhXROKwPoUDF91BIsSUWNnT17jheIyR3Ac=
verified example.org/log
cosignatures 1 timestamp 1700000000
verifier example.org/log+5e2cef00+AepKbGPinFIKvvVQexMuxfmVR3auvr57kkIe6mkURtIs
The same signing key under another name has another key id. The checkpoint topic describes the note. The fixed seed 7 repeated is a documentation key.
Signer::from_pem reads a PKCS#8 PEM file (-----BEGIN PRIVATE KEY-----). The log how-to passes that file to rfc9162-sidecar snapshot --key.