The on-disk format

This chapter is a readable summary — enough to understand what the files are, reason about recovery, and write an external reader. The normative byte-level layout is §5 of the design spec, which wins on any disagreement.

Directory layout

<wal_dir>/
├── 00000000000000000001.wal    # segment whose first record has LSN 1
├── 00000000000000100001.wal    # base_lsn = 100001
├── 00000000000000200001.wal    # active segment (highest base_lsn)
└── LOCK                        # advisory exclusive-writer lock file

• Each segment file is named {base_lsn:020}.wal — its first record’s LSN, zero-padded to 20 digits (the width of u64::MAX), so lexicographic and numeric order agree.
• There is no manifest or CURRENT file. The log’s state is derived entirely by listing the directory, sorting by base LSN, and scanning. A manifest would be one more thing that could disagree with reality.
• The active segment is the one with the highest base LSN; every other segment is sealed and immutable until a checkpoint deletes it whole.
• Segments are pre-allocated to segment_size at creation, so the unwritten remainder reads as zeros. During a scan, an all-zero record header is the end-of-records sentinel.

Segments

Every segment starts with a fixed 64-byte header: a magic string (WAL\0SEG1), a format version, the segment’s base_lsn, a creation timestamp (informational only — recovery never uses it), and a CRC over the header. The header is written and synced when the segment is created, before any record — so a corrupt header is never a torn write, and recovery treats it as fatal (with one carve-out for a crash mid-creation; see Recovery).

Records follow the header back-to-back, 8-byte aligned.

Records

Each record is framed as:

Details that carry weight:

• The checksum is CRC-32C (Castagnoli) — hardware-accelerated on modern CPUs, and the same polynomial ext4, RocksDB, and iSCSI use. The crate re-exports the primitive as open_wal::crc32c so an external reader can match it exactly. (Beware: the popular crc32fast crate implements a different polynomial and will reject every record.)
• Padding is inside CRC coverage, so tampered or corrupted padding is detected — there is nowhere to hide bytes in a record.
• A record never spans segments. This is why max_record_size + 91 <= segment_size is enforced at open (64-byte segment header + 20-byte record header + up to 7 bytes padding), and it is what makes each segment independently parseable.
• The end-of-records sentinel is an all-zero 20-byte header — not merely rec_type == 0. A header with a zero rec_type but nonzero CRC is a corrupt record, not the end of the log; the distinction is what lets recovery tell a clean end from a single flipped byte in the middle of acknowledged data.

What a scan looks like

Reading a segment is: skip the 64-byte header, then repeatedly — check for the sentinel, bounds-check length, verify the CRC, verify the LSN is the expected next one, yield the payload, advance by the padded record size. The in-crate Reader, recovery, and any external tailer all follow this same logic; Recovery adds the tail-classification rules on top.

That’s the whole format: sorted filenames, a checksummed header, checksummed LSN-stamped records, zeros at the end. Boring on purpose.