v0.2.36 was about the format. v0.2.51 is about what surrounds it: the crates are on crates.io, releases are gated through CI and signed by OIDC rather than a laptop, the binary pipeline covers seven platforms, and the codec stack crossed a threshold we’ve been chasing for a month — LimniFS’s CSV ratio (2.99%) now beats the C Brotli reference encoder’s own output (6.23%) by 2.1×, while creating images 39× faster than the previous release did on the same dataset.
The ratio story
The synthetic CSV benchmark (750K rows of structured records, 20 MB) has been our hardest target since v0.1.0. The 0.16.7x omnizip codec line — block splitting plus context clustering in the from-spec Brotli encoder — changed the landscape:
| Format | Create | Ratio |
|---|---|---|
| LimniFS (balanced) | 2.25 s | 2.99% |
| tar + zstd | 0.04 s | 4.79% |
| DwarFS | 38.6 s | 3.59% |
| SquashFS (zstd) | 0.02 s | 16.35% |
Two things stand out: the ratio beats every competitor including DwarFS’s LZMA pipeline (which takes 17× longer to get there), and the create speed is within two orders of magnitude of plain tar — for a content-addressed, Merkle-rooted format that also verifies and dedups. On FITS-style scientific data the specialized ricepp codec keeps its crown (32.1% vs DwarFS’s 46.3%), and clean PCM audio sits at 0.44%.
The writer got faster where it hurt
- Streaming walk — the tree walk now runs concurrently with compression: files flow to rayon workers through a bounded channel while the walk is still descending. Output bytes are identical (verified manifest, metadata sidecar, and all seven slabs of a 400 MB tree); warm-cache deep-tree create drops ~10%, and the cold- cache win is larger by construction.
- Drop-level dedup — a container-layer-shaped tree (4,000 files, 262 MB, 100 unique contents) images in 0.2 s flat: 100 drops total, deterministic Merkle root, cache hits shared by refcount instead of copied.
- A quiet config bug fixed —
inline_thresholdwas documented per-profile but silently ignored; it now applies. The measurements that found it also settled a design question: for tiny files, inline routing beats slab routing on both size and speed (87.1% and 4.6 s vs 108.3% and 68.4 s on 50K unique 1 KiB files), because the shared-inline table already dedups and one metadata stream compresses better than 50K independent chunks.
Releases you can verify
The release process is now fully machine-gated — nothing ships from a laptop:
- release-plz owns versioning and publishes the five crates
(
limnifs-format,limnifs-ocb3,limnifs-core,limnifs-write,limni) to crates.io in dependency order. - Trusted publishing — the CI job exchanges a GitHub OIDC token for a short-lived crates.io token. No long-lived registry token exists anywhere.
- Every tag builds seven binaries — linux-gnu ×2, static musl ×2, macOS ×2, Windows — plus SHA-256 checksums and a CycloneDX SBOM per crate, attached to the GitHub Release.
- Offline signing —
limni sign-keygen/limn --sign-key/verify-sigprovide Ed25519 signatures over the image’s ManifestRoot with OpenSSL-compatible PEM keys: three independent tamper checks (root match, signer identity, signature), no network, no sigstore dependency.
Resilience work landed too: a panicking codec now degrades to the next candidate (Brotli → ZSTD → STORE) instead of killing the write — a guard that earned its keep twice this month when upstream encoder bugs surfaced on ≥16 MB inputs.
What’s next
The single remaining hot spot is single-stream Brotli on very large metadata blobs (the dictionary-lookup cost in the from-spec encoder, filed upstream); everything else in the assemble path measures under 1% of create time. On the format side, the v1 wire spec freeze is the next milestone.