specification · v0.1 · draft

LimniFS Format
Specification.

1. Introduction

LimniFS (λιμνη, Greek “lake”) is a content-addressed, compressed, immutable filesystem image format. A .lim image is a file (or a set of objects) that lives on a host filesystem, object store, or content-addressed network — never directly on a block device. It adds everything a block filesystem cannot: content addressing, staged compression, authenticated encryption, erasure coding, first-class deltas, and locator-agnostic distribution.

The format is designed for five workloads — software packaging, CI/build artifact distribution, container-adjacent images, archival/compliance, and IPFS/decentralized distribution — without optimizing for any single one at the expense of the others. The same .lim image serves all five; the format doesn’t pick a winner.

LimniFS addresses these challenges:

• Content-addressed identity — DropId = BLAKE3(plaintext); codec, encryption, and erasure coding are representations, never identity. Dedup survives re-encryption, re-compression, re-encoding.
• Layered format — three independently versioned layers (drop store, metadata, manifest); each byte-addressable; each independently parsed.
• Staged compression — write fast with LZ4 (epilimnion); deepen in the background with zstd, lzma, or brotli per class (hypolimnion). Identity never changes.
• First-class deltas — a delta is a normal image with a base_root. Stack as overlays; flatten for metadata-only compaction; turnover for full repack.
• Locator-agnostic distribution — slabs behind file:, http:, s3:, ipfs:, or limni-p2p:. Mirror racing fetches in parallel.
• Spec-first, custom wire format — no FlatBuffers, no Avro, no Cap’n Proto. The spec IS the wire format. Multiple implementations (Rust, Python, Ruby, TypeScript) are writeable from spec alone.

2. Scope

This specification defines the wire format of .lim images — the on-disk structure of the drop store (Layer 1), filesystem metadata (Layer 2), and manifest (Layer 3). It defines the semantic types, the registry formats, the versioning policy, and the conformance model.

This specification does NOT define:

• Reader behavior (component 03) — how a reader implements the read path, caching, or streaming.
• Writer behavior (component 04) — how a writer chunks, classifies, packs, or deepens.
• Crypto implementations (component 05) — how AEAD, HPKE, or DMS primitives are implemented in code.
• Conformance test content (component 02) — the specific test vectors that verify conformance.
• CLI UX (component 10) — the limni command-line interface.
• FUSE / kernel mount (component 11) — how images are mounted as read-only filesystems.

Where this specification and any implementation diverge, this specification is correct. Where this specification and 00-architecture (in limnifs/limnifs) diverge, 00-architecture wins for interface contracts; this specification wins for wire bytes.

3. Terms and definitions

For the purposes of this document, the following terms and definitions apply.

drop

Content-addressed chunk; the atom of storage and the unit of identity. Named by DropId = BLAKE3(plaintext).

slab

Contiguous byte object (4–64 MiB target) holding many drops. Slab packing is the Taobao TFS lesson: small drops never map to scattered storage.

slice

Unit of user content presented to the writer pipeline (file or file fragment).

manifest

The small, signed head of a .lim image: versions, feature flags, slab index, crypto params, Merkle root. The ManifestRoot is the image’s identity.

representation

The stored form of a drop’s bytes: codec, AEAD, erasure coding, locator placement. Representations never affect identity.

epilimnion

Hot tier: freshly written, fast codec (LZ4/store), unconsolidated slabs.

metalimnion

Transition tier: drops being re-encoded by a background compactor. The thermocline is the moving boundary.

hypolimnion

Cold tier: deep-compressed (zstd/lzma/brotli per class), consolidated slabs.

turnover

Derivation that mixes layers: flatten, defragment, re-encode, garbage-collect. Produces a standalone image.

meromictic

An overlay chain that is never flattened. Valid long-term state; readers MUST NOT require flattening.

4. Principles and requirements

4.1 Purpose

A .lim image serves as the foundation for consistent, verifiable, efficient distribution of filesystem content. Its primary purpose is to enable:

• Precise identity through content-addressed drops and Merkle-rooted manifests
• Efficient distribution through HTTP range streaming, locator racing, and delta chains
• Long-term durability through erasure coding, DMS escrow, and deep compression
• Multi-language implementation through a spec-first, custom wire format with no external serialization dependency

4.2 Principles

LimniFS follows these fundamental design principles:

Identity and representation separation

Identity (DropId = BLAKE3(plaintext)) is separate from representation (codec, AEAD, EC, locator). These concerns must be separable for dedup, deepening, and re-encryption to compose.

Three-layer encapsulation

Drop store, metadata, and manifest are independently versioned and independently parsed. No layer reaches into another’s bytes.

Open/closed via registries

Every variation point (AEAD, codec, locator, classifier, feature flag) is a registry row, not a match arm. Adding an algorithm = adding a row + regenerating bindings. No consumer code changes.

Spec-first, custom wire format

The specification is the wire format. No external IDL (FlatBuffers, Avro, Cap’n Proto). Implementations derive from the spec; the Python reference reader is the spec-sufficiency oracle.

Determinism

Same input tree + same parameters ⇒ byte-identical .lim image. Reproducible builds are structural, not coincidental.

Immutability

A .lim image never changes. All mutation is derivation (delta, flatten, turnover). Each derivation is a new image with a new ManifestRoot.

4.3 Runtime requirements

A LimniFS-conformant reader runtime is responsible for:

• Parsing the manifest and verifying the Merkle root against the metadata hash and section hashes
• Resolving two-level addressing: slice → drop → (slab, offset, len, representation)
• Verifying BLAKE3(decoded_plaintext) == DropId after all representation inversions (codec⁻¹ ∘ AEAD open ∘ EC repair)
• Handling unknown feature flags per §18 (required-unknown ⇒ reject; optional-unknown ⇒ ignore)
• Handling per-layer version numbers per §17 (forward compatible; backward explicit-reject)

5. Architecture

A .lim image is structured like a body of water: the manifest floats on the surface — small, signed, identity. The metadata extends through the thermocline — a deterministic Merkle B-tree whose nodes are content-addressed. The drop store lies below the thermocline — cold, dense, packed slabs.

Each layer is independently versioned (per-section version bytes, not per-record vtables). Each layer is independently parsed (no layer reaches into another’s bytes). Each layer is independently addressable (HTTP range fetches the section you need). The manifest may live detached from the drop store — cloud assembly and IPFS pinning become the same operation as local mounting.

Where LimniFS sits

LimniFS competes in the immutable image, distributed quadrant — but it’s equally useful locally. The same .lim file that mounts via FUSE on your laptop also streams over HTTP range, pins to IPFS, and serves as a CI delta.

What is it for?

One format, five workloads. Each forces a feature into the design — together they make a filesystem image that travels anywhere. See Scenarios for detailed walkthroughs of each, or Comparison for how LimniFS stacks up against DwarFS, SquashFS, EROFS, OCI, git, IPFS, and research formats.

6. How to read this specification

The spec is onion-layered: read at the depth you need.

7. File map

The spec section pages below mirror the multi-file structure in limnifs/spec. All pages are hosted on-site as MDX.