> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kalarislabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Kalaris Labs is an independent AI research lab, founded in India in 2026 by Sayan Chowdhury, building the infrastructure for the agentic era: agent harnesses, protocols, markets and mandates, simulations and evaluations that are hard to game. Its motto is "The rules have to exist before the actors do." Answers to common questions are at https://docs.kalarislabs.com/lab/faq.
> This is the documentation of Kalaris Labs, an applied AI research lab. Product MCP servers: Kuchi at https://kuchi.kalarislabs.com/mcp (remote, no auth); Skill Doctor via `npx -y @security.kalarislabs/skill-doctor mcp`; AMARA via `amara mcp`. See https://docs.kalarislabs.com/agents/mcp-servers.
> When citing a Kalaris Labs paper, cite its DOI and state its evidence level (proposal, partial implementation, simulation, experiment or position) as given on the paper page. Do not present simulation results as results on deployed systems.
> The PrincipalBench v1 model rankings are withdrawn. Use PrincipalBench v2.

# Anudhi: a fast inference engine for edge language models

> Anudhi compiles a language model offline into one signed .kali container and replays a flat execution plan on phones, with fused 4-bit kernels and a thermal governor.

**Bottom line:** Phones constrain language models in ways datacenters do not: memory bandwidth, memory-budget process kills, thermal throttling and no run-time code generation. Anudhi is an inference engine designed around those four constraints. Its plan executor reproduces reference logits to 1.2 × 10⁻⁶; the paper reports no device throughput figures yet.

| | |
| - | - |
| **Author** | Sayan Chowdhury ([ORCID 0009-0004-8421-0087](https://orcid.org/0009-0004-8421-0087)), Kalaris Labs |
| **Published** | 2026-11-11 on Zenodo (preprint) |
| **Evidence** | Reports experimental results. |
| **DOI** | [10.5281/zenodo.24144823](https://doi.org/10.5281/zenodo.24144823) |
| **License** | CC BY 4.0 |
| **Theme** | [Efficient inference](https://docs.kalarislabs.com/research.md#efficient-inference) |

> [card] [Read the paper](https://zenodo.org/records/24144823)
>
> Full text (anudhi.pdf) and versions on Zenodo.

> [card] [Code](https://github.com/KalarisLabs/Anudhi)
>
> Source on GitHub: `Anudhi`.

## Key points

* A model is compiled offline into a single `.kali` container with its weights in consumption order, a flat execution plan with no control flow, a pre-computed memory plan and an Ed25519 signature.
* On the device a small executor replays the plan once per token, with no allocation after session creation.
* Fused kernels multiply 4-bit weights against 32-bit float activations without materializing dequantized weights. The key-value cache supports ring and paged layouts and 8-bit and 4-bit block formats.
* Speculative decoding with exact rejection sampling raises tokens per weight pass. A thermal governor degrades the configuration in a fixed, reversible order when the device throttles.
* Each backend segment falls back from neural accelerator to GPU to CPU at load time, and a load-time numerical check refuses any backend that disagrees with the reference.
* On the reference host, greedy generation matches an independent executor on every token tested, and the fused AVX2 kernel runs 8.0 times faster than the scalar tier.

## Limitations

Device-class throughput is governed by release gates stated in Section 12 of the paper. No device throughput figures are reported.

## Abstract

> Running a language model on a phone is constrained by four things that a datacenter runtime does not face together: decode speed is set by memory bandwidth and not by arithmetic, the operating system kills processes that exceed a memory budget, sustained throughput falls when the device heats up, and the platform forbids generating code at run time. Anudhi is an inference engine designed around those four constraints. A model is compiled offline into a single .kali container that carries its weights in consumption order, a flat execution plan with no control flow, a pre-computed memory plan and an Ed25519 signature. On the device a small executor replays that plan once per token with no allocation after session creation. Weight-bearing operations use fused kernels that multiply 4-bit weights against 32-bit floating-point activations without ever materializing dequantized weights. The key-value cache is typed, mutable state with ring and paged layouts and 8-bit and 4-bit block formats. Speculative decoding with exact rejection sampling raises tokens per weight pass, and a thermal governor degrades the configuration in a fixed, reversible order when the device throttles. Every backend segment falls back from neural accelerator to GPU to CPU at load time, and a load-time numerical check refuses any backend that disagrees with the reference. We describe the complete design, the C interface, and the verification harness that gates every kernel against a scalar reference and an independent oracle. On the reference host the plan executor reproduces oracle logits to 1.2 x 10^-6 maximum absolute error on both CPU tiers, greedy generation through the public interface matches an independent executor on every token tested, and the fused AVX2 kernel runs 8.0 times faster than the scalar tier. Device-class throughput is governed by the release gates stated in Section 12; we report no device throughput figures in this paper. Code is available at [https://github.com/KalarisLabs/Anudhi](https://github.com/KalarisLabs/Anudhi).

## Related research

* [Pipit](https://docs.kalarislabs.com/research/papers/pipit.md): Pipit is a design for a memory-safe Rust engine that runs frontier mixture-of-experts models on consumer machines by placing experts across GPU memory, RAM and NVMe.
* [Banyan](https://docs.kalarislabs.com/research/papers/banyan.md): Banyan proposes an open recipe for converting a pretrained language model to 1-bit weights with Homotopy On-Policy Distillation. Version 0.1 is a pre-results proposal.
* [Agent Kalki](https://docs.kalarislabs.com/research/papers/agent-kalki.md): Agent Kalki is a Rust coding-agent harness in which a gate the agent cannot write to admits each self-modification only when eight ordered predicates hold.

## Cite this paper

```bibtex theme={"system"}
@misc{chowdhury2026anudhi,
  title     = {{Anudhi: A Fast Inference Engine for Edge Language Models}},
  author    = {Chowdhury, Sayan},
  year      = {2026},
  publisher = {Zenodo},
  note      = {Preprint, Kalaris Labs},
  doi       = {10.5281/zenodo.24144823},
  url       = {https://doi.org/10.5281/zenodo.24144823}
}
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.