Introduction

open-wal is an embeddable, single-writer, append-only write-ahead log for Rust. Its one job: when commit() returns, the records it covers are durable — they survive a process crash and a power loss on honest hardware. Everything else about the crate follows from taking that one promise seriously.

It was built as the journal for an LMAX-style, event-sourced system: a single writer thread appends opaque event payloads, syncs them in batches, and only then acknowledges or publishes downstream. After a restart, replaying the log rebuilds state exactly. In such a system the log is not a cache or a convenience — it is the source of truth, and the durability boundary is where correctness lives.

Design philosophy

Durability first, honestly. Many logging libraries treat fsync as a configuration knob and crash behavior as a footnote. Here it is inverted: the crash and power-loss behavior is the specified, tested core, and the API is shaped so you cannot easily mistake “written” for “durable”. append is pure memory; only commit — a write plus fdatasync (F_FULLFSYNC on macOS) — makes records durable, and it tells you exactly how far durability reached.

Loud failure over silent loss. A torn write at the tail of the log (the normal result of a crash mid-write) is detected, truncated, and durably invalidated. Corruption in the middle of the log — which would mean losing acknowledged data — is a fatal, distinct error, never a silent truncation. A failed fsync poisons the writer handle rather than pretending a retry can make the data safe. These choices borrow deliberately from PostgreSQL (the fsyncgate lesson), RocksDB (torn-tail tolerance vs. absolute consistency), and Kafka (segmented retention).

Small and predictable. No background threads, no compaction, no manifest files, no hidden I/O. Two runtime dependencies (crc32c, rustix). Payloads are opaque byte slices — serialization is entirely the caller’s concern. The steady-state append/commit path performs zero heap allocations.

Single-writer, enforced. Exactly one writer per log, and the type system plus an OS lock make that a compile-time and open-time guarantee rather than a documentation footnote. This is a feature: it removes locks from the hot path and removes multi-writer interleaving from the correctness argument. See Single-writer by construction.

What it is not

• Not a database. Records are located by log sequence number (LSN) only — no keys, indexes, or queries.
• Not multi-writer. One exclusive writer per WAL directory.
• Not a replication system. The crate provides the durable substrate — ordered records, immutable sealed segments, a durable-watermark hook — but shipping, acknowledgements, and failover belong to the integrator (details).
• No multi-record transactions. The atomicity primitive is a single record; see the durability model for what that means in practice.

How this book relates to the design spec

This book is the human on-ramp. The normative contract — the twelve durability invariants (D1–D12), the exact on-disk byte layout, the recovery algorithm, and the fault-injection test plan that backs the guarantees — lives in the design specification. Where this book says “the log guarantees X”, the spec is where X is stated precisely and mapped to tests. When in doubt, the spec wins.

Status

open-wal is pre-1.0 and under active development. The write, recovery, and checkpoint paths are implemented and tested well beyond unit level — SIGKILL crash matrices, LazyFS power-loss simulation, dm-flakey fsync-failure injection, property tests, a model-based oracle, and continuous fuzzing — but the public API may still see breaking changes before 1.0. Linux is the production target; macOS is supported for development and correctness work only; Windows is out of scope for v1.

Getting started

Every example in this book is compiled and run against the real API as part of the crate’s test suite (tests/book_examples.rs), so the code you see here is the code that works.

Install

[dependencies]
open-wal = "0.2"

MSRV is Rust 1.85. Linux is the production platform; macOS works for development (see Introduction).

The mental model

Three moments matter in the life of a record:

1. append — the record gets the next LSN and is buffered in memory. No syscall happens. It is not durable.
2. commit — everything appended since the last commit is written to the log file and fdatasync’d. commit returns the new durable watermark: every record with an LSN at or below it is now safe against crash and power loss.
3. Replay — after a restart, Wal::open runs recovery and a Reader yields the committed records back, in LSN order, byte-identical.

LSNs are dense (1, 2, 3, …) and the durable log is always a gap-free prefix. That is the shape everything else in this book builds on.

Hello, durability

use open_wal::{Lsn, Wal, WalConfig};

fn main() -> Result<(), open_wal::WalError> {
    let dir = std::path::Path::new("./journal");
    let (mut wal, report) = Wal::open(dir, WalConfig::default())?;
    println!("recovered up to LSN {}", report.durable_lsn);

    // Pure memory: assigns LSNs 1 and 2, buffers the payloads.
    wal.append(b"order-created:42")?;
    wal.append(b"order-paid:42")?;

    // One write + one fdatasync for the whole batch. After Ok(w),
    // every record with lsn <= w survives crash and power loss.
    let durable = wal.commit()?;
    println!("durable up to {durable}");

    // Replay from the beginning (Lsn(0) means "from the start").
    let mut reader = wal.reader_from(Lsn(0))?;
    while let Some(record) = reader.next() {
        let (lsn, payload) = record?;
        println!("{lsn}: {} bytes", payload.len());
    }
    Ok(())
}

Run it twice. The second run’s report.durable_lsn is 2: recovery found the committed records and the log continues from there. Kill the process between append and commit and the buffered records are gone — which is exactly the contract: a crash loses at most the un-committed tail, never anything a returned commit covered.

What’s on disk afterwards is deliberately boring — a LOCK file and one pre-allocated segment file named after its base LSN:

journal/
├── 00000000000000000001.wal
└── LOCK

See the on-disk format if you want to know what’s inside.

Configuration

WalConfig has exactly two knobs:

let config = WalConfig {
    segment_size: 128 * 1024 * 1024, // bytes pre-allocated per segment file
    max_record_size: 1024 * 1024,    // hard cap on a single payload
};
// …which happen to be the defaults: WalConfig::default()

• segment_size — the log is a sequence of fixed-size, pre-allocated segment files. Larger segments mean fewer rolls; smaller segments mean finer-grained space reclamation (checkpointing deletes whole segments).
• max_record_size — payloads above this are rejected at append with RecordTooLarge. No silent truncation, no splitting.

One constraint ties them together: a record never spans segments, so a maximal record (plus headers and padding) must fit inside one segment — max_record_size + 91 <= segment_size. open validates this up front:

let bad = WalConfig { segment_size: 1024, max_record_size: 1024 };
assert!(matches!(
    Wal::open(dir, bad),
    Err(open_wal::WalError::InvalidConfig)
));

Pass the same config every time you open a given WAL directory; the config is not persisted in the log.

Where to go next

The single most important thing to understand before shipping anything on top of this crate is what commit does and does not promise — that’s the durability model.

The durability model

This is the centerpiece chapter. If you read only one, read this one — the misconceptions it corrects are the ones that cause silent data loss in systems built on logs.

Append is memory; commit is durability

append assigns the next LSN and encodes the record into an in-memory staging buffer. It performs no I/O and, in steady state, no allocation. The record is not durable — it isn’t even in the page cache yet.

commit takes everything staged since the last commit, writes it to the active segment file, and calls fdatasync (on macOS, F_FULLFSYNC, because plain fsync there does not flush the drive cache). Only when commit returns Ok(w) may you treat records with lsn <= w as durable — and w, the durable watermark, is exactly the value to gate acknowledgements on:

let (mut wal, _) = Wal::open(dir, WalConfig::default())?;
wal.append(b"durable")?;
wal.commit()?;
wal.append(b"buffered only")?; // never committed

assert_eq!(wal.last_lsn(), Lsn(2));    // assigned...
assert_eq!(wal.durable_lsn(), Lsn(1)); // ...but not durable

// Dropping the handle without committing loses the buffered tail —
// exactly what a crash would do.
drop(wal);
let (_, report) = Wal::open(dir, WalConfig::default())?;
assert_eq!(report.durable_lsn, Lsn(1));

last_lsn() is the highest LSN assigned; durable_lsn() is the highest LSN safe. The gap between them is what a crash may cost you. Never acknowledge, publish, or replicate past durable_lsn.

Group commit: the throughput lever

An fdatasync costs the same whether it covers one record or ten thousand. commit’s batch is simply “everything appended since the last commit”, so the way to trade latency for throughput is to batch:

for event in ["a", "b", "c", "d"] {
    wal.append(event.as_bytes())?; // pure memory, no syscall
}
let durable = wal.commit()?; // one write + one fdatasync for the batch

This is the classic LMAX journaling pattern: drain whatever events are available, append them all, commit once, then acknowledge the batch. Per-event fsync latency disappears; the fsync cost amortizes across the batch.

The non-guarantees — read these twice

commit is not atomic

A commit batch is a performance grouping, not a transaction. The log’s guarantees are per-record durability and a dense LSN prefix — never all-or-nothing durability of a batch.

Concretely: when a batch doesn’t fit in the active segment, commit splits it at whole-record boundaries and syncs each segment separately. A crash (or an I/O failure) between those syncs keeps the first part of the batch — durable, acknowledged by the watermark — and loses the rest. That outcome is contract-compliant: what survives is still a dense prefix, and durable_lsn reported exactly how far durability got.

When a batch happens to fit in one segment it is covered by a single shared fdatasync, so it looks atomic. That is incidental, not guaranteed. Do not build logic that assumes a commit batch is indivisible.

The atomicity primitive is a single record

A single record is all-or-nothing: it never spans segments, it’s covered by one CRC, and recovery either yields it byte-identical or (if it was the torn tail) drops it entirely. So if several events must live or die together, encode them into one record with a compound payload:

// WRONG for atomicity: two appends in one commit batch. A crash (or a
// split across segments) can keep the first and lose the second.
//
// RIGHT: if two events must be all-or-nothing, encode them into ONE
// payload. A single record is the only atomicity primitive.
let mut compound = Vec::new();
compound.extend_from_slice(b"debit:alice:100;");
compound.extend_from_slice(b"credit:bob:100");
let lsn = wal.append(&compound)?;
wal.commit()?;

The encoding of the compound payload is your concern — the WAL stores opaque bytes.

When fsync fails: the handle poisons

A failed fdatasync is not a transient hiccup you can retry. On Linux, a failed fsync may mean the dirty pages were already dropped — the data is gone, and a retried fsync can “succeed” while durable state is missing (the PostgreSQL fsyncgate lesson). For a dense-LSN log there is no safe resume: the API has no way to rewrite a lost LSN slot, and continuing would create a permanent gap.

So there is exactly one policy. On any write/fdatasync failure:

• The failing commit returns an error (FsyncFailed or Io).
• durable_lsn keeps whatever earlier segments in the batch achieved — it is monotonic and never lies.
• The handle is poisoned: every subsequent append/commit/checkpoint returns WalError::Poisoned.

The only way forward is to drop the handle and Wal::open afresh — recovery truncates any torn tail and hands you an honest durable state to rebuild from. Treat a poisoned handle as “this process’s writer is done”, not as an error to swallow.

What this buys you

Summing up the contract in user terms (the spec states these precisely as invariants D1–D12 in §4 of the design spec):

• After commit() returns Ok(w), records <= w survive process crash and power loss.
• The durable log is always a dense, gap-free run of LSNs.
• A crash loses at most the un-committed tail.
• Replay returns exactly what was committed, in order, byte-identical.

How those promises survive torn writes and corruption is the subject of Recovery.

Writing and reading

The writer surface

The whole write API is four methods on Wal:

Lsn is a transparent newtype over u64 (Lsn(pub u64)). Lsn(0) is the reserved “none” value — real records start at Lsn(1) and are dense from there. The newtype exists so an LSN can’t be silently mixed up with a byte offset or a count.

append can fail two ways: RecordTooLarge if the payload exceeds max_record_size (nothing is truncated or split — size your records or raise the limit), and Poisoned if a previous durability failure killed the handle:

let big = vec![0u8; 2048]; // exceeds max_record_size
assert!(matches!(
    wal.append(&big),
    Err(open_wal::WalError::RecordTooLarge)
));

A commit with nothing staged is a cheap no-op that returns the current watermark. Segment management — rolling to a new segment when the active one fills, splitting a large batch across segments — happens entirely inside commit; append neither knows nor cares about segments.

Reading back: the streaming Reader

reader_from(&self, from: Lsn) returns a Reader that yields records with lsn >= from, in order, across segment boundaries. Lsn(0) means “from the beginning”. Reading is how you replay state at startup, and how a shipping consumer pulls newly durable records.

let mut reader = wal.reader_from(Lsn(3))?; // start mid-log
let mut retained: Vec<(Lsn, Vec<u8>)> = Vec::new();
while let Some(record) = reader.next() {
    let (lsn, payload) = record?;
    // `payload` borrows the reader's internal buffer and is only valid
    // until the next `reader.next()` call. Copy it to keep it.
    retained.push((lsn, payload.to_vec()));
}

Two things to notice:

It is a lending iterator, not a std::iter::Iterator. Each call to next() returns Option<Result<(Lsn, &[u8])>> where the payload slice borrows the reader’s reused internal buffer — valid only until the next next() call. The standard Iterator trait cannot express an item that borrows from the iterator per call, which is exactly the shape that makes replay zero-copy and (after warm-up) allocation-free. The practical consequences: you drive it with while let Some(..) = reader.next() rather than a for loop, and if you need to keep a payload past the next call you copy it (.to_vec()). Consumers that ship records over a network pay that copy at the serialization boundary anyway.

Reading below the retention floor is a loud error. After a checkpoint has deleted old segments, asking for records that no longer exist fails with an error rather than silently starting at the oldest available record. A reader silently jumping from LSN 100 to LSN 100000 is the footgun this rule forbids.

The reader holds a shared borrow of the Wal, so the borrow checker prevents appending while a Reader is alive — read, drop the reader, then write. (For readers that run concurrently with the writer — tailing from another thread or process — see External readers.)

LSNs are yours to correlate

The WAL hands out LSNs; it does not interpret payloads. Typical integrations keep a mapping from domain state to “applied up to LSN x”, persist that with their snapshots, and use it to know where replay must start. If you run two WALs (say, an input journal and an output journal), their LSN spaces are completely independent — correlate them inside your payloads if you need to.

Recovery

Recovery is not an error path bolted on afterwards — it is the other half of the durability contract, and most of this crate’s engineering (and testing) lives here. It runs inside every Wal::open, before the handle is returned.

What open does

1. Discover. List the directory, collect *.wal segment files, parse each base LSN from the filename, and sort. (Never trusting directory iteration order — recovery must be deterministic.)
2. Validate headers. Each segment starts with a checksummed header written and synced at creation. A corrupt header on a sealed segment is fatal. One special case is forgiven: a highest-base file with an incomplete header is the residue of a crash during segment creation — it provably contains no durable records, so it is discarded.
3. Scan records. Each segment’s records are walked in order, checking length bounds, CRC-32C, and LSN continuity, until the end-of-records sentinel (an all-zero header in the pre-allocated zero region).
4. Handle the tail — see below.
5. Check cross-segment continuity. The last LSN of each segment must be exactly followed by the next segment’s base. An internal gap is fatal (ContiguityViolation).

The result is a (Wal, RecoveryReport) pair:

use open_wal::TailState;

let (wal, report) = Wal::open(dir, WalConfig::default())?;
println!(
    "recovered LSNs {}..={} across {} segment(s)",
    report.oldest_lsn, report.durable_lsn, report.segments_scanned
);
match report.tail_state {
    TailState::Clean => { /* the common case */ }
    TailState::TruncatedAt { segment_base, offset } => {
        // A torn tail (crash mid-write) was truncated and durably zeroed.
        // Only un-committed records were lost.
        eprintln!("torn tail truncated in segment {segment_base} at byte {offset}");
    }
}

A TruncatedAt tail is normal after a crash and worth logging, but requires no action: nothing at or below any previously returned durable_lsn was lost.

Torn tail vs. mid-log corruption — the crucial distinction

A crash mid-commit leaves a partially written record at the physical tail of the active segment. That record was never covered by a returned commit, so dropping it is correct. Recovery detects it (bad length or CRC), truncates at that point, and then durably zeroes everything from the truncation point to the end of the segment. The zeroing matters: without it, a stale but CRC-valid record from an earlier generation could lurk past the tail and be “resurrected” as if it were live data on some later recovery.

Corruption before the tail is a different animal entirely. If a record fails validation but a valid record still exists after it, the bad record cannot be a torn tail — data after it was genuinely written and acknowledged. Truncating there would silently discard acknowledged records. Recovery instead fails loudly with TornMidLog (or Corruption in a sealed segment, which can never contain a torn tail). This is deliberate: mid-log corruption means the storage lied, and the correct response is to stop and involve the operator — restore from backup or accept explicit, visible data loss — not to quietly shrink the log.

Properties you can rely on

• Deterministic and idempotent. Recovery is a pure function of the on-disk bytes — no clocks, no environment. Opening the same directory repeatedly converges: the second recovery sees what the first left and changes nothing.
• Bounded. Recovery never panics, reads out of bounds, or scans/allocates unboundedly — for any input bytes, including adversarial ones. The parser is fuzzed continuously and cross-checked against an independent reference implementation.
• Frugal with memory. Recovery never loads payloads into memory; it keeps only a small per-segment index. Opening a multi-GB log does not OOM.
• Read-back fidelity. What replay yields after recovery is exactly what was committed: same records, same order, byte-identical.

These are user-facing distillations of invariants D1–D12; the precise normative statements and their test mapping are in §4 of the design spec.

One operational note: reopening after a crash

When a writer process dies, the OS releases its directory lock during process teardown — which can complete slightly after the process is otherwise gone. A supervisor that restarts the writer immediately may see a brief, spurious WalError::Locked on the first open. Retry with a short backoff (up to about a second) before treating Locked as a genuine concurrent writer.

Checkpointing and retention

A WAL grows forever unless something reclaims it. In an event-sourced system the thing that makes old log entries redundant is a snapshot: once you have durably persisted your application state as of LSN s, recovery becomes “load the snapshot, replay the log after s” — and log entries at or below s are no longer needed.

checkpoint(up_to) is the reclamation half of that bargain.

What checkpoint does

// Only ever pass the LSN covered by your latest DURABLE SNAPSHOT —
// never `wal.durable_lsn()`. Recovery is snapshot + replay of the log
// after it; deleting the log past your snapshot caps recovery at the
// stale snapshot. The WAL trusts the caller here.
wal.checkpoint(snapshot_lsn)?;

It deletes whole sealed segments that are fully at or below up_to — oldest first, followed by a directory fsync so the deletions are durable. It never rewrites, compacts, or partially truncates anything, and it never deletes the active segment. Afterwards the oldest available LSN advances; records above up_to are exactly as readable as before. If no segment is fully superseded, the call is a no-op — space is reclaimed at segment granularity, so segment_size is also your retention granularity.

Because deletion is oldest-first and only ever removes a prefix, a crash at any point mid-checkpoint leaves the survivors a contiguous suffix; re-running the checkpoint after recovery is safe and idempotent.

The caller rule — the one way to lose data with a correct WAL

The WAL guarantees it will never delete a record above up_to. What it cannot check is whether you can afford to lose the records below it.

The safe bound is your latest durable snapshot LSN — not durable_lsn(). It is tempting to “clean up everything that’s durable” by calling checkpoint(wal.durable_lsn()), and it is exactly wrong: recovery needs the log between your snapshot and the durable watermark to replay. Delete it and nothing fails today — but the next restart silently comes up with state frozen at the stale snapshot. That is the inverse of the WAL’s own promise: the WAL won’t delete what you kept; you must not ask it to delete what you cannot rebuild.

The discipline that follows:

1. Persist a snapshot of application state, durably, recording the LSN s it covers.
2. Only after the snapshot is safely down: wal.checkpoint(s).
3. On restart: load the snapshot, then replay with reader_from(Lsn(s + 1)).

Retention margin for readers and backups

Checkpointing interacts with anything else reading the log. A backup job or a tailing reader that still needs a deleted segment hits the fatal-gap rule: it fails loudly rather than silently skipping ahead. The writer does not track reader positions in v1 — retention policy is yours. Practical options: keep a margin of N segments behind your snapshot, checkpoint on a schedule that outlasts your slowest consumer, or accept that a lagging consumer re-seeds from a fresh snapshot.

One convenience on the reading side: an already-open file descriptor survives unlink on Linux, so a reader that is mid-segment when a checkpoint deletes that segment finishes it normally — the gap only bites a reader that hadn’t opened the file yet.

Single-writer by construction

open-wal allows exactly one writer per log directory. This is not a limitation to be worked around — it is a load-bearing design decision, and the crate spends type-system and OS machinery making it impossible to violate by accident.

Why single-writer

A multi-writer log needs interior locking, write coordination, and a much larger correctness argument — precisely the machinery an LMAX-style system exists to avoid. With one writer:

• The hot path has no locks, no atomics, no contention — append is a plain memory write into a buffer owned by one thread.
• LSN assignment is trivially dense and ordered.
• The crash-recovery story stays tractable: every on-disk state is the result of one sequential writer stopping at some point, which is what makes the recovery classification (torn tail vs. real corruption) sound.

If you have multiple producers, put a queue in front of the single writer thread — that is the intended integration shape, not a workaround.

Two layers of enforcement

Within a process: the type system. Wal is Send but deliberately not Sync, and every write method takes &mut self. You can move the handle to another thread, but you cannot share it between two: Arc<Wal> gives you no way to call append, and &Wal cannot cross threads. Concurrent writers are a compile error. (This is verified in CI with a compile-fail test.)

Between processes: an OS lock. open takes an exclusive advisory flock on a LOCK file in the WAL directory and holds it for the handle’s lifetime. A second process (or a second handle in the same process) gets a clean error:

let (wal, _) = Wal::open(dir, WalConfig::default())?;
match Wal::open(dir, WalConfig::default()) {
    Err(open_wal::WalError::Locked) => { /* expected: one writer at a time */ }
    Err(other) => panic!("expected Locked, got {other:?}"),
    Ok(_) => panic!("expected Locked, got a second writer"),
}
drop(wal); // releases the lock; now a new writer may open
let (_wal, _) = Wal::open(dir, WalConfig::default())?;

One operational wrinkle: after a writer process crashes, the OS may release its lock a beat after the process disappears, so an immediate restart can see a transient Locked. Retry briefly (up to ~1 s) before concluding another writer is alive — see Recovery.

Note that read-only access is not writer-locked: Readers borrow the writer in-process, and external readers attach without touching the exclusive lock (see External readers).

What the integrator owns

Single-writer scoping means the crate deliberately stops at the durability boundary. Around it, you own:

• The publish/ack barrier. The WAL returns durable_lsn; gating downstream consumers on it (an atomic cursor, a channel — the LMAX daisy-chain) is your code. The DurabilityObserver hook exists to feed it.
• Serialization. Payloads are opaque bytes; encode/decode however you like, subject only to max_record_size.
• Snapshots and the checkpoint trigger. The WAL reclaims what you tell it to; knowing what is safe to reclaim requires your snapshot — see Checkpointing.
• Replication transport and failover, if any — see External readers.

A common deployment runs two independent instances — an input journal and an output journal. Each has its own directory, lock, LSN space, and recovery; the WAL neither knows nor cares that the other exists.

External readers, backup, and replication

The log is single-writer, but its on-disk format is deliberately tailable: records are self-describing, LSN-stamped, and CRC-protected, and sealed segments are immutable — once a newer segment exists, a sealed segment’s bytes never change until a checkpoint deletes the whole file. That immutability is what makes everything in this chapter safe.

This chapter covers what v1 supports directly, the one hazard every external consumer must understand, and where the crate’s responsibility ends.

The durability-visibility gap — the central hazard

The writer’s write(2) makes bytes visible in the OS page cache before the fdatasync that makes them durable. A concurrent reader (another process tailing the files, or even a thread reading between the writer’s write and sync) can therefore see records that are complete and CRC-valid but not yet durable. On power loss, those records vanish.

A valid CRC proves integrity, not durability. Any consumer whose actions must never outrun the primary’s durable state — a replica, a downstream system receiving acknowledgements — must consume only up to the published durable watermark, no matter how many valid-looking records are visible past it. (A pure backup consumer is the exception; see below.)

Publishing the watermark: DurabilityObserver

The writer publishes its watermark through a pluggable observer. Wal is generic over it — Wal<O: DurabilityObserver = NullObserver> — and the default NullObserver compiles to nothing, so you pay for the hook only if you use it. The observer fires on the writer thread at the end of each commit, strictly after durability has advanced, so it can never affect the durability guarantees; it receives a monotonic durable_lsn.

use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use open_wal::{DurabilityObserver, Lsn, Wal, WalConfig};

/// Publishes the durable watermark to an atomic another thread reads.
struct WatermarkPublisher(Arc<AtomicU64>);

impl DurabilityObserver for WatermarkPublisher {
    fn on_durable(&mut self, durable_lsn: Lsn) {
        // Contract: cheap, non-blocking, no I/O, must not panic.
        // A release-store (or a queue push) is the intended shape.
        self.0.store(durable_lsn.0, Ordering::Release);
    }
}

let watermark = Arc::new(AtomicU64::new(0));
let (mut wal, _report) = Wal::open_with(
    dir,
    WalConfig::default(),
    WatermarkPublisher(Arc::clone(&watermark)),
)?;

Keep on_durable to exactly that shape — an atomic store or a queue push. It runs synchronously inside commit, so blocking or doing I/O there puts your observer’s latency on the commit path. The observer carries only the watermark; shipping actual record bytes is a separate consumer’s job, reading via a Reader up to the watermark it was handed.

Backup

Backup needs no watermark and no coordination beyond retention:

1. Sealed segments: copy freely, any time, in any order — they are immutable, so a copy is coherent by construction. You can checksum one once and trust it until it is deleted.
2. Active segment: copy it as-is, even mid-write. The copy may end in a torn record; that is fine, because restoring is just running normal recovery — Wal::open on the copied directory truncates the torn tail and yields a consistent dense prefix.

A backup taken this way may include a few records that were never durable on the primary (the visibility gap above). For backup that is harmless: the restore is still an internally consistent log. Only if a backup must match an exact durability point do you need to trim to the watermark.

Schedule copies against your checkpoint cadence so a slow copy isn’t undercut by segment deletion.

The retention floor: gap is fatal

A checkpoint can delete segments a lagging reader still needs. When a reader’s next-expected LSN is older than the oldest available LSN, it must fail loudly — silently skipping to the oldest available record would be silent data loss on the consumer’s side. The in-process Reader enforces this (reader_from an already-reclaimed LSN errors), and any external reader you write must do the same. Recovery from a gap is operational: re-seed the consumer from a fresh snapshot or backup. The v1 writer does not track or wait for readers; retention margin is integrator policy.

Replication: what v1 gives you and what it doesn’t

v1 supports in-process log shipping: a consumer in the writer’s process learns the watermark from the observer (or your own cursor), reads newly durable records with a Reader, copies them out (.to_vec() — the network serialization boundary pays that copy anyway), and ships them. The one inviolable rule: never ship a record above durable_lsn — otherwise a power loss on the primary leaves the replica ahead of the primary, which is divergence.

What v1 does not include: a cross-process watermark channel (so replication readers in another process are deferred — cross-process backup is fine, as above), the transport, the replica-ack protocol, and failover. The crate’s responsibility ends at providing committed records in order plus a truthful watermark; see §15 of the design spec for the full normative treatment of external access.

The on-disk format

This chapter is a readable summary — enough to understand what the files are, reason about recovery, and write an external reader. The normative byte-level layout is §5 of the design spec, which wins on any disagreement.

Directory layout

<wal_dir>/
├── 00000000000000000001.wal    # segment whose first record has LSN 1
├── 00000000000000100001.wal    # base_lsn = 100001
├── 00000000000000200001.wal    # active segment (highest base_lsn)
└── LOCK                        # advisory exclusive-writer lock file

• Each segment file is named {base_lsn:020}.wal — its first record’s LSN, zero-padded to 20 digits (the width of u64::MAX), so lexicographic and numeric order agree.
• There is no manifest or CURRENT file. The log’s state is derived entirely by listing the directory, sorting by base LSN, and scanning. A manifest would be one more thing that could disagree with reality.
• The active segment is the one with the highest base LSN; every other segment is sealed and immutable until a checkpoint deletes it whole.
• Segments are pre-allocated to segment_size at creation, so the unwritten remainder reads as zeros. During a scan, an all-zero record header is the end-of-records sentinel.

Segments

Every segment starts with a fixed 64-byte header: a magic string (WAL\0SEG1), a format version, the segment’s base_lsn, a creation timestamp (informational only — recovery never uses it), and a CRC over the header. The header is written and synced when the segment is created, before any record — so a corrupt header is never a torn write, and recovery treats it as fatal (with one carve-out for a crash mid-creation; see Recovery).

Records follow the header back-to-back, 8-byte aligned.

Records

Each record is framed as:

Details that carry weight:

• The checksum is CRC-32C (Castagnoli) — hardware-accelerated on modern CPUs, and the same polynomial ext4, RocksDB, and iSCSI use. The crate re-exports the primitive as open_wal::crc32c so an external reader can match it exactly. (Beware: the popular crc32fast crate implements a different polynomial and will reject every record.)
• Padding is inside CRC coverage, so tampered or corrupted padding is detected — there is nowhere to hide bytes in a record.
• A record never spans segments. This is why max_record_size + 91 <= segment_size is enforced at open (64-byte segment header + 20-byte record header + up to 7 bytes padding), and it is what makes each segment independently parseable.
• The end-of-records sentinel is an all-zero 20-byte header — not merely rec_type == 0. A header with a zero rec_type but nonzero CRC is a corrupt record, not the end of the log; the distinction is what lets recovery tell a clean end from a single flipped byte in the middle of acknowledged data.

What a scan looks like

Reading a segment is: skip the 64-byte header, then repeatedly — check for the sentinel, bounds-check length, verify the CRC, verify the LSN is the expected next one, yield the payload, advance by the padded record size. The in-crate Reader, recovery, and any external tailer all follow this same logic; Recovery adds the tail-classification rules on top.

That’s the whole format: sorted filenames, a checksummed header, checksummed LSN-stamped records, zeros at the end. Boring on purpose.

FAQ and gotchas

The recurring misconceptions, in roughly the order they bite.

Is a commit batch atomic?

No. Durability is per-record; the batch is a performance grouping. A crash during a commit that spans a segment boundary can keep the first part of the batch and lose the rest (still a dense prefix — never a gap). If several events must be all-or-nothing, encode them as one record. This is the top misconception; the durability model covers it in full.

I can see the record in the file — is it durable?

Not necessarily. write(2) makes bytes visible before fdatasync makes them durable, so an external reader can observe complete, CRC-valid records that would vanish on power loss. CRC proves integrity; only the durable watermark (durable_lsn, or the DurabilityObserver publication) proves durability. See the durability-visibility gap.

Can I checkpoint(wal.durable_lsn()) to reclaim everything durable?

No — this is the classic silent-data-loss mistake. Recovery is snapshot + replay after it. Checkpoint only up to the LSN your latest durable snapshot covers; deleting the log between snapshot and durable_lsn caps every future recovery at the stale snapshot, and nothing warns you until a restart. See the caller rule.

How do I write from two threads?

You don’t — the handle is Send but not Sync, so sharing it doesn’t compile. Move it into one writer thread and put a queue in front. A second process is excluded by the directory lock (WalError::Locked). See Single-writer by construction.

Why isn’t Reader a std::iter::Iterator?

Because it lends: each yielded &[u8] borrows the reader’s reused buffer and is valid only until the next call — a shape Iterator cannot express. That is what keeps replay zero-copy. Use while let Some(record) = reader.next(), and .to_vec() anything you need to keep. See Writing and reading.

commit failed and now everything returns Poisoned. Can I recover the handle?

No — by design. After a failed fsync the data’s fate is unknowable (the OS may already have dropped the dirty pages), and a dense-LSN log has no safe way to resume past a hole. durable_lsn still tells the truth about what made it. Drop the handle, Wal::open again (recovery truncates any torn tail), and rebuild from your snapshot + the recovered log. See the durability model.

Recovery says TruncatedAt { .. } — did I lose data?

You lost only records that were never covered by a successful commit — the un-acknowledged tail a crash is allowed to take. Log it and carry on. The error cases are different: TornMidLog / Corruption mean acknowledged data is damaged mid-log, and recovery deliberately refuses to continue. See Recovery.

Can I put JSON / protobuf / bincode in records? What about compression?

Yes — payloads are opaque bytes and the WAL never interprets them. The only constraint is max_record_size. Serialization, versioning, and compression are all yours.

Does it work on macOS? Windows?

macOS: yes for development and correctness — the crate correctly uses F_FULLFSYNC (plain fsync on macOS does not flush the drive cache) — but it carries no production durability claim; the hardware fault-injection gate runs on Linux only. Windows: out of scope for v1.

My payload is bigger than max_record_size.

append rejects it with RecordTooLarge — no silent truncation, no splitting. Raise max_record_size (it must satisfy max_record_size + 91 <= segment_size), or chunk at the application level if the pieces don’t need single-record atomicity.

What is the fuzzing Cargo feature I see in the source?

Internal-only: it exposes hooks for the crate’s fuzz targets and differential tests. It is not part of the public API and must never be enabled in a real build. (Same for inject_no_dir_fsync, a deliberately-broken build used as a fault-injection negative control.)

Where are the exact guarantees written down?

§4 (invariants D1–D12) and §5 (on-disk format) of the design spec are normative, and §14 maps every invariant to the tests that enforce it.