Pipit: placement-first MoE inference on consumer hardware

Key points

• Pipit treats decoding as a bandwidth problem across three tiers: GPU memory, RAM and NVMe.
• Components: a slot-slab expert store, a learned expert predictor, bandwidth-saturating CPU kernels for six instruction-set levels, union-aware speculative decoding, and validated loaders for GGUF, safetensors and a new streaming format, KXF.
• The report compares the design with existing local engines (Colibri, uzu and Magnitude) and the gaps it targets in each.
• It pre-registers ten falsifiable hypotheses against same-machine baselines, with a protocol that scores every run against its physical roofline.

Limitations

Abstract

Mixture-of-experts (MoE) language models with hundreds of billions of parameters activate only a few per cent of their weights for each token. That makes them runnable on machines with far less memory than the model, provided the right expert weights sit in the right memory tier at the right moment. Today’s local engines each solve part of this. Colibri streams experts from SSD but relies on general-purpose kernels and has repeatedly patched memory-safety defects in its loaders; uzu reaches strong speculative-decoding speed on Apple Silicon but is not yet stable on other operating systems; Magnitude tunes kernels per device and connects to coding agents, though its headline speed-ups rest on comparisons with mismatched settings. This report presents the design of Pipit, a memory-safe Rust engine that treats decoding as a bandwidth problem across three tiers (GPU memory, RAM and NVMe). It combines a slot-slab expert store, a learned expert predictor, bandwidth-saturating CPU kernels for six instruction-set levels, union-aware speculative decoding, and validated loaders for GGUF, safetensors and a new streaming format, KXF. We pre-register ten falsifiable hypotheses against same-machine baselines, together with a protocol that scores every run against its physical roofline. The report contains no measured Pipit results: it fixes the claims in advance, so that forthcoming results can be judged against them.

Related research

• Anudhi: Anudhi compiles a language model offline into one signed .kali container and replays a flat execution plan on phones, with fused 4-bit kernels and a thermal governor.
• Banyan: Banyan proposes an open recipe for converting a pretrained language model to 1-bit weights with Homotopy On-Policy Distillation. Version 0.1 is a pre-results proposal.
• The Harness Is the Lab: A position paper and blueprint for an open AI-scientist harness that grounds every claim in a typed ontology and accepts self-improvement only on held-out evidence.

Cite this paper

@misc{chowdhury2026pipit,
  title     = {{Pipit: A Placement-First Inference Engine for Frontier Mixture-of-Experts Models on Consumer Hardware}},
  author    = {Chowdhury, Sayan},
  year      = {2026},
  publisher = {Zenodo},
  note      = {Preprint, Kalaris Labs},
  doi       = {10.5281/zenodo.24144909},
  url       = {https://doi.org/10.5281/zenodo.24144909}
}