LimniFS v0.3.9 — Nine Releases Later: Every Hot Path Parallel, Every Path Gated

v0.3.0 was about the read path: seekable drops, SIEVE caching, a 98× first-read win. Since then we shipped nine patch releases in about thirty-six hours. This post is the whole arc — what changed, what it’s worth, and — the part we’re most proud of — how every claim below is enforced by a red-or-green CI gate instead of a benchmark run nobody reproduces.

The arc, in nine releases

Three serial tails, one lesson

Each of the last three performance passes found the same bug wearing different clothes: a step that ran on one core while every rayon worker idled, invisible on the workloads we benchmarked because something else was parallel.

• Phase 1 hashed every FastCDC chunk sequentially. Many-file trees hid it — files pack in parallel, so the cores were busy. Pack one 5 GB file and the hash pass was a single-lane BLAKE3 conveyor with N−1 spectators. Now par_iter, byte-identical output, pinned by a pack-twice determinism test.
• Dictionary re-compression ran after the parallel phase ended. A second full-tree compression pass — the most expensive codecs, on one core, precisely when every worker had gone home. Now parallel, with the per-drop plaintext clone turned into a borrow.
• Extracting a single large file decoded its slices one at a time. Cross-file parallelism hid it on many-file images; a multi-GB weights file decoded on one core. Multi-slice files now decode in bounded 64-slice batches — peak extra memory stays ~16 MiB, not file-sized.

The lesson we now apply on every audit: when a profile says “X is hidden by parallelism elsewhere,” find the workload where that elsewhere doesn’t exist. It’s usually a single large file.

Gates, not benchmarks

A performance number without an enforcement mechanism is marketing. Every claim in this post has a hard gate in the benchmark canary that runs on every release tag:

The create gate exists because the read gates caught nothing on the write side: the v0.3.4 mmap-materialization bug (a full memcpy per file) would have sailed through CI. Now that class of regression is a red job, not a user report.

Two more gates worth naming:

• Windows runs the whole test suite, not just cargo check. Its first run caught a real bug immediately — three test helpers opened /dev/urandom directly and failed on MSVC. Production signing was already portable; the tests weren’t. Compile-green ≠ behavior-green, and now we test the difference.
• The release pipeline heals itself. A transient GitHub 5xx on one asset upload used to fail the release and need a manual re-run (it happened on v0.3.6). The publish step now re-uploads everything idempotently with retries and verifies the asset count before declaring success.

omnizip 0.21.7 rides in

v0.3.9 updates all 19 omnizip codec dependencies to the 0.21 line — brotli’s quality-tier overhaul (every level 14–31% under the reference implementation, with a determinism fix and emit-path speedups), plus the rest of the family. LimniFS compiled against it with zero API drift, the full suite passed on the first run, and the create canary moved 85 → 99 MB/s from the dependency change alone — no LimniFS code changed in v0.3.9.

Compatibility note: compressed outputs may differ from 0.16-era images (new encoders); decoding handles frames from both. The wire format is unchanged — still a single v1.

What’s deliberately not here

The one remaining serial term on the create path is FastCDC’s gear-hash cut itself — a loop-carried recurrence no SIMD can touch without changing chunk boundaries. The leap-based CDC reformulation is documented in docs/fastcdc-simd-proposal.md and parked: it changes dedup behavior, and that’s a writer-semantics decision, not a refactor. Every gate above will tell us if the trade ever becomes worth reopening.