LimniFS v0.2.36 — Pure-Rust Codec Stack, Read-Write Images, 18 Codecs

Four weeks ago we shipped v0.1.0: seven codecs, BLAKE3 content addressing, pure Rust. Today v0.2.36 brings the format to a different class — read-write images, 18 codec variants competing in a tournament selector, 9 profiles spanning the speed/ratio/read-heavy/write-heavy tradeoff space, a streaming write API, mmap-bounded peak memory, cross-file compress cache, and a state-of-the-art benchmark suite that runs the full matrix against DwarFS, SquashFS, and tar+zstd.

The codec stack is now pure-Rust end-to-end — 16 of 17 omnizip crates have zero external dependencies. The only remaining external codec dep is brotli (via omnizip-brotli, awaiting Phase C of the full in-house port).

The headline: LimniFS now beats every existing tool on ratio across most workload classes, with 5–8× create-speed wins across the benchmark suite — while keeping the pure-Rust, content-addressed, Merkle-rooted design from v0.1.0.

Cumulative speedups since v0.1.0

Same ratios, same output bytes. Every speedup is from removing redundant work or absorbing upstream omnizip improvements.

What’s new since v0.1.0

1. Read-write images with crash safety

v0.1.0 was read-only — pack once, read forever. v0.2.28 introduces RwImage: open a .lim file, add/update/delete files, commit. A write-ahead log (LIMWAL) survives crashes mid-commit; an atomic rename(2) swap ensures the manifest is either fully old or fully new.

Three turnover modes:

• CopyOnWrite (max-write-rw): fast updates, unreferenced blocks reclaimed on turnover.
• UpdateInPlace (max-read-rw, balanced-rw): full history kept for audit. Turnover compacts when the history threshold hits.

limni open my-image.lim --rw
limni add my-image.lim new-file.txt
limni update my-image.lim config.toml
limni commit my-image.lim

Crash at any point: re-open rolls forward through the WAL or rolls back to the last commit. No partial state ever escapes.

2. Tournament codec selector with short-circuit

The v0.1.0 categorizer picked one codec per content class (text → Brotli, binary → LZ4, audio → FLAC, etc.). v0.2.28 introduces a tournament: every chunk gets compressed by each codec in the profile’s tournament list, smallest wins.

The catch: a tournament with 5 codecs is 5× slower than single-codec. We added short-circuit — when any codec achieves the profile’s target ratio (default 25% for balanced, 0% for max-ratio, 50% for max-write), the tournament accepts and stops. On highly compressible text chunks, LZ4 typically hits under 10% ratio in microseconds and we accept it without running the slower Brotli/ZSTD passes.

The same short-circuit pattern now applies to both the chunk path and the whole-file categorizer path. CSV files compress 8× faster; WAV files 11× faster; ratios unchanged.

3. Nine profiles

Different workloads need different tradeoffs. v0.2.28 ships nine built-in profiles:

Switch via limni limn --profile max-ratio ./input -o out.lim.

4. The codec library

The headline change. LimniFS now ships 18 codec variants end-to-end through omnizip:

Each codec has strongly-typed tunables (quality, memory budget, context order, dictionary size) plumbed through the profile system.

5. State-of-the-art benchmark suite

limnifs-bench is a Rust workspace crate (no Python subprocess overhead) that runs LimniFS via direct library calls and external tools via subprocess. Multi-profile support:

limnifs-bench run --profile balanced,max-write,max-ratio --all

Each profile produces a separate row in the report. The win/loss matrix renders every (dataset, operation, format) triple with the winner flagged and slow formats tagged with the multiplier (e.g., 8.4× means 8.4× slower than the winner).

6. Other improvements

• FastCDC 4× unroll in the gear hash inner loop
• Parallel slab assembly — slabs encoded in rayon workers
• madvise(MADV_WILLNEED) prefetch on slab files
• Hot slab LRU cache with polymorphic SlabSource trait
• Cross-image Bloom filter sparse dedup index (opt-in feature)
• Per-class ZSTD dictionary training (FrequencyTrainer + FastCoverTrainer)
• AEAD: ChaCha20-Poly1305 (default), AES-OCB3, AES-GCM, Ascon
• Key wrap: X25519-HKDE (HPKE-style)
• Signing: Ed25519 + optional sigstore
• Reed-Solomon erasure coding per slab (systematic Vandermonde)
• Locators: file, HTTP range, S3, IPFS gateway + CAR
• 585 workspace tests passing, 0 warnings, clippy::pedantic clean

7. v0.2.29: ZSTD 7.5× encode speedup

The single biggest external win of the cycle: omnizip-rs landed a cached Huffman encode table (TODO 152) that lifts ZSTD-1 from 11.5 to 85.8 MB/s. ZSTD is in 4 of 9 LimniFS profiles (max-read, max-read-rw, balanced-rw text path, max-ratio tournament), so the speedup shows up across many datasets. FITS saw the largest swing because its 47 MB payload runs through ZSTD L6 baseline in process_whole_file_drop: 26.9 s → 3.7 s (7.3×) on the balanced profile.

8. v0.2.30: Streaming write, mmap input, FSST baseline

Three LimniFS-side improvements that don’t depend on omnizip:

• write_stream<R: Read> — new entry point that packs a single named stream from any reader. Internal buffering bounded at max_chunk_size + 64 KiB. Lets callers pipe from network sockets, pipes, or generators without a temp file:

  let reader = std::io::Cursor::new(bytes);
  let artifact = limnifs_write::write_stream("name.bin", reader, &config)?;

• Memory-mapped input — process_file memmaps files above 1 MiB instead of std::fs::read-ing them. Peak RSS drops from total_input_size to roughly unique_chunks × avg_chunk_size because chunk compressors see borrowed slices into the mmap. Below 1 MiB plain read is faster.

• FSST+Brotli pre-computed baseline — process_whole_file_drop now passes its already-computed Brotli result into FSST+Brotli’s comparison check, eliminating one full Brotli pass per FSST-routed file. New public API: limnifs_core::codec::fsst_brotli::compress_with_baseline.

9. v0.2.31–v0.2.32: Snappy encoder, LZMA reusable state

Two more upstream wins absorbed:

• CODEC_SNAPPY (0x06) gains a from-spec encoder with full wire-format compatibility (snap-compat). Previously Snappy was decode-only in LimniFS — round-trip with externally produced Snappy streams (Parquet, ORC, Avro, SQLite WAL) now works in both directions.
• XzCodec accepts LIMNIFS_XZ_REUSE_STATE env var — opt into LzmaCompressor::ResetMode::ReuseState for ~5–10% LZMA encode speedup at the cost of run-to-run byte determinism. Default behaviour (deterministic ResetMode::Full) is unchanged.

10. v0.2.33–v0.2.34: LimniFS-discovered LZ4 bug, fixed upstream

We found a bug in omnizip-lz4 0.14.18’s new from-spec encoder: it produced output its own decoder rejected on inputs with match/literal lengths exactly at the code-nibble-15 boundary. Filed upstream as docs/omnizip-proposals/lz4-from-spec-broken.md. omnizip-rs PR #115 fixed it within one release cycle and added 5 regression tests. v0.2.34 absorbs the fix and removes the Cargo.lock pin.

Side effect: lz4_flex is no longer in the dep tree. All LZ4-using paths (including omnizip-blosc and omnizip-filters) now use the in-house from-spec encoder.

11. v0.2.35: libdeflate dynamic-Huffman fix

omnizip-rs TODO 116: the broken package-merge in omnizip-libdeflate was replaced with correct standard Huffman + zlib CPI length limiting. Dynamic-Huffman path re-enabled. CODEC_LIBDEFLATE (0x14) now produces 10–20% better ratio on text and binary inputs. Round-trip safety verified by the existing cross_decodes_with_deflate test against the miniz_oxide-backed CODEC_DEFLATE (0x05).

12. v0.2.36: Cross-file compress cache

Each rayon worker thread now carries a thread-local HashMap<DropId, (codec_id, compressed_bytes)>. When two files share a chunk — common in source trees with vendored dependencies, container layer stacks, and successive build outputs — the second file hits the cache and skips the tournament compress pass entirely. Cache is bounded at 100K entries; output bytes are unchanged.

Where we are vs the field

Pure-ratio comparison on synthetic datasets (balanced profile, lower is better):

We win or tie ratio on every synthetic dataset. Create speed is competitive with SquashFS on most workloads and substantially faster than DwarFS. Extract speed is dominated by decompression cost; we win on WAV (FLAC) and FITS (ricepp) by 5-50×.

What’s next

• omnizip-side work: ZSTD SIMD per-se, Brotli Phase C, FLAC FFT (other half), ricepp SIMD (other half). Each closed item directly improves LimniFS without code change here.
• Website polish: more docs, an interactive drop-explorer, a Docker image, package manager recipes.
• FUSE mount stability: production hardening of limnifs-fuse.
• ComposeFS path: kernel-mountable via composefs-style path.

Get started

git clone https://github.com/limnifs/limnifs
cd limnifs/limnifs
cargo build --release -p limni

# Your first .lim image
./target/release/limni limn ./my-dir -o my.lim
./target/release/limni verify my.lim
./target/release/limni ls my.lim

We’re on GitHub at limnifs/limnifs. Issues, PRs, and benchmark results from your own workloads all welcome.

— The LimniFS team, 2026-08-05