> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kalarislabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Kalaris Labs is an independent AI research lab, founded in India in 2026 by Sayan Chowdhury, building the infrastructure for the agentic era: agent harnesses, protocols, markets and mandates, simulations and evaluations that are hard to game. Its motto is "The rules have to exist before the actors do." Answers to common questions are at https://docs.kalarislabs.com/lab/faq.
> This is the documentation of Kalaris Labs, an applied AI research lab. Product MCP servers: Kuchi at https://kuchi.kalarislabs.com/mcp (remote, no auth); Skill Doctor via `npx -y @security.kalarislabs/skill-doctor mcp`; AMARA via `amara mcp`. See https://docs.kalarislabs.com/agents/mcp-servers.
> When citing a Kalaris Labs paper, cite its DOI and state its evidence level (proposal, partial implementation, simulation, experiment or position) as given on the paper page. Do not present simulation results as results on deployed systems.
> The PrincipalBench v1 model rankings are withdrawn. Use PrincipalBench v2.

# PrincipalBench: a benchmark for LLM orchestrators in multi-agent pipelines

> PrincipalBench measures how well a language model orchestrates worker agents. Version 2 uses a sealed two-phase protocol, clean control episodes and chance-corrected failure detection.

**PrincipalBench measures a model as an orchestrator: the principal that decomposes work, supervises worker agents and
synthesises their outputs.** Use version 2. Version 1 cannot support model comparisons, and its published rankings are
withdrawn.

> [!WARNING]
>
> The repository README still shows v1 results, including a ranking with Llama-3.1-70B first. Those rankings are
> withdrawn by the [PrincipalBench v2 paper](https://docs.kalarislabs.com/research/papers/principalbench-v2.md). Do not cite them.

## Why v1 was withdrawn

An audit of v1 against its own code and data found that a model-free policy that flags every output scores 0.52
PrincipalScore.

| v1 problem | Consequence |
| - | - |
| 433 of 500 tasks were templated placeholders without real questions or gold answers | Most of the dataset could not be scored |
| Every episode contained an injected failure | Flagging everything gave perfect failure detection |
| Only three distinct failures were ever injected | Detection measured three cases, not 105 |
| Context coherence was computed from worker outputs | The metric was identical for every model |
| Two-thirds of injected failures came from another domain | Failures were easy to spot for the wrong reason |

## How v2 works

**1. Planning phase**

The orchestrator decomposes the task from the question alone, and is scored on that plan.

**2. Supervision phase**

The orchestrator reviews gold-aligned worker reports. Half of the episodes are clean controls. Failures are derived
from each hop's own gold output.

**3. Synthesis**

The final step is withheld, so the orchestrator must synthesise the answer itself.

**4. Scoring**

Detection is chance-corrected with Youden's J. Coherence is measured on the orchestrator's own ledger.

## v2 dataset

| Split | Tasks | Subtasks | Episodes |
| - | - | - | - |
| Core | 67 hand-authored | 289 | 335 over five seeds |

## Reference policies

v2 behaves as a benchmark should: degenerate strategies score 0.00 on detection and at most 0.50 PrincipalScore, and
only a full oracle reaches 1.00. The paper reports reference policies, not model rankings. It releases v2 as the basis
for future, reproducible orchestrator evaluations.

## Get the code

```bash theme={"system"}
git clone https://github.com/KalarisLabs/Principal-Bench
cd Principal-Bench
pip install -r requirements.txt
```

> [card] [PrincipalBench v2 paper](https://docs.kalarislabs.com/research/papers/principalbench-v2.md)
>
> The audit, the v2 protocol and the reference-policy results.

> [card] [Repository](https://github.com/KalarisLabs/Principal-Bench)
>
> Harness, tasks, failure library and scoring code.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.