LimniFS v0.3.0 — The Read Path Catches Up: 48 GiB of Wasted Decode, Gone by Construction

v0.2.51 was about the write path: codecs, ratios, create speed. It turns out that was only half the filesystem. This release is about the other half — what happens when someone actually reads an image — and about a number that embarrassed us into it:

Reading a 19.5 MB file through 8 KiB windows on a mounted image decompressed ~48 GiB of data to serve 19.5 MB.

That is ~2,500 windows × one full-drop decode each. Nineteen seconds of pure lz4 work, ~1 MB/s effective throughput, for an operation that should be a memcpy. tebako hit it in the wild and patched around us with a client-side memo — which was the actual insult, and the actual signal: when your users memo around your library, the library is wrong.

Why this had to be fixed, and why a cache wasn’t enough

The write side of LimniFS is configurable to a fault — codec tournaments, trained dictionaries, nine profiles, a streaming pipeline. The read side was a raw codec call per window. Every windowed read paid the whole drop’s decode cost, again and again.

The obvious first fix is a cache, and we shipped a good one. But a cache alone has a hole a user can fall through: the first touch of any byte range still decodes everything behind it. A 19.5 MB drop, cold, costs 19.5 MB of decode whether you want 8 KiB or the whole file. Random access into compressed data shouldn’t have a story where the word “amplification” appears with a three-digit multiplier. That hole isn’t a cache problem — it’s a format problem, and it needed a format answer.

There’s prior art for the answer. The zstd Seekable Format (frames plus a trailing index) and Fuchsia’s BlobFS seek tables both encode random-access structure into the compressed layout. We adopted the idea, codec-agnostically, and since LimniFS is alpha software with no compatibility debt worth keeping, we made it the format rather than an opt-in: one image format, one drop-record layout, no version history to lug around. Making a v2 without ever having shipped a real v1 would have been borrowing formality we haven’t earned.

What shipped, layer by layer

1. Seekable drop containers — the format fix

Drops over 1 MiB (general codecs: LZ4, ZSTD, XZ, Brotli, bzip2, …) are now stored as independent 256 KiB frames plus a footer index: per-frame (uncompressed_len, compressed_len) pairs and a fixed 10-byte tail with the LMSK magic, a version, and the frame count (positioned so the footer parses back-to-front — you learn the count from the tail, then walk the table; same trick as zstd’s). A drop record’s new trailing flags byte marks it SEEKABLE.

The reader binary-searches the footer and decompresses only the frames covering the window. A cold 8 KiB read anywhere in a 19.5 MB drop costs one 256 KiB frame — 1.3% of the drop — instead of 100% of it. Edge frames are sliced after decode, so the bound holds for any window shape.

Crucially, DropId is still BLAKE3(plaintext). Identity, dedup, and Merkle verification are untouched: a seekable drop and a monolithic drop with the same bytes are the same drop.

2. SIEVE caches — the memory fix

Repeat accesses need to be free, not just cheap. Decoded drops (the small ones) and decoded frames (the 256 KiB ones) now live in two SIEVE-evicted caches — one FIFO queue and one visited bit per entry, O(1) throughout, scan-resistant, and evaluated across 6,594 block traces to beat LRU/ARC-class policies. Values are shared Arc<[u8]> plaintexts: a cache hit is a refcount bump, and callers that only borrow never copy. Both caches are bounded by entries and bytes (64 MiB + 32 MiB defaults), and a value larger than the whole byte budget bypasses the cache instead of evicting the working set — one huge drop can no longer flush everything else.

3. O(1) reads, mmap’d slabs — the architecture fix

An audit of what a single 8 KiB window actually paid found that every lookup re-parsed the entire slab — its record table, every drop, on every read, twice (once to ask “is this drop seekable?”, once to read it). Slab record tables are now parsed once at open into an owned index; a read is a hash lookup plus a bounds-checked slice. And ImageReader::open mmaps the slab sidecars, so pages enter RSS on demand through the kernel’s page cache instead of an eager read() of the whole image.

Two quieter fixes fell out of the same audit: the [chunking] config section was being parsed, validated… and silently ignored (the chunker was hardcoded; it’s wired now, with defaults that keep output byte-identical), and large-drop emission grew a seekable_drops knob for the ratio-over-read-cost crowd (max-ratio opts out; the container’s frame independence costs 1–3%, and on our benchmark fixture the measured cost was 0.2%).

4. A public reader API — the ergonomics fix

The efficient thing should be the easy thing:

use limnifs_core::read::{ImageReader, ReadConfig};
use std::io::Read;

let reader = ImageReader::open(manifest_path.into(), ReadConfig::default())?;

let file = reader.file("/usr/bin/app")?;
let n = file.read_at(offset, &mut buf)?;      // positional, bounded decode
file.read_to_end(&mut sink)?;                 // std::io::Read, streaming

// bulk extraction: drops are independent — decode on rayon
limnifs_core::read::extract_file(&manifest, "/usr/bin/app", &mut w,
    ReadConfig { parallel_decode: true, ..Default::default() })?;

tebako’s client-side memo can be deleted. A frames_decoded() counter is exposed so integrators can assert bounded work in their own CI — we do in ours.

The numbers

Same 19.5 MB file, same 8 KiB windows, packed both ways (limnifs-bench readcompare):

metric	monolithic	seekable	delta
first 8 KiB window	48.7 ms	0.57 ms	85×
cold windowed	0.2 MB/s	14.0 MB/s	84×
warm windowed	13038 MB/s	8091 MB/s	0.62×
sequential extract	376 MB/s	400 MB/s	1.06×
image size	14.52 MiB	14.49 MiB	1.00×

Cold work per window: one 256 KiB frame (1.03 on average across random offsets), never the drop. The warm-windowed line is the one honest loss — a monolithic drop that’s already decoded is a single giant Arc you slice for free, and nothing beats a slice — but both sides are gigabits-class, and the monolithic column is the one carrying a 58 ms cold read.

The CI-gated canaries (limnifs-bench readperf, hard gates: windowed ≥ 200 MB/s, extract ≥ 100 MB/s) measure 3163 MB/s sequential / 11999 MB/s warm-windowed on a quiet machine. The warm number climbed in two steps — the construction-time index and caches took it 4627 → 7793 before the format change was even exercised, and the zero-copy read_at_into follow-up (decoding straight into the caller’s buffer, no intermediate allocation per window) took it to ~12000.

Kept honest by CI

Performance fixes rot silently unless something fails when they regress. The Benchmark workflow’s read-canary job now runs the gated readperf and the A/B readcompare on every push, and the test suite asserts the invariant directly: cold windows decode exactly one frame (frames_decoded() deltas), repeat windows decode zero, and a hand-corrupted footer fails closed. The 48 GiB class of bug now has to get past three locked doors to come back.

The alpha tax, paid once

One format means exactly that: images written by ≤ 0.2.64 are not readable by 0.3.0+. We cut this as a minor version bump so downstream users see the incompatible image-format change immediately. If you have cached images — tebako runtimes, CI artifacts — re-pack them once and never think about it again. We’d rather spend our compatibility budget before 1.0 on making the format right than on carrying a prerelease layout nobody depends on. The spec now specifies the 50-byte drop record and the seekable container bit-for-bit.