Kalaris Labs benchmarks and evals for AI agents

Benchmarks we maintain

Benchmark	Measures	Status	Docs
PrincipalBench v2	How well an LLM orchestrates worker agents: planning, failure detection, synthesis and coherence	v2 core split released; v1 rankings withdrawn	PrincipalBench
HiddenLaw	Whether a learner can introduce a variable its vocabulary lacks (Level-3 discovery)	Released with transcripts and reproduction script	HiddenLaw
Research Agent Skills benchmarks	Skill quality, trigger routing, with-vs-without-skill task outcomes, citation verification and writing revision	Pilot measurements	Skill benchmarks
Skill Doctor attack fixtures	Detection of malicious skill files across 11 threat classes	14 curated fixtures in v0.1.0	Detection rules

Benchmark	Used in	Result
KernelBench	TailHorizon	fast_1 of 48% / 78% / 22% on Levels 1–3, against 43% / 72% / 18% for the strongest baseline (three seeds)
RE-Bench	TailHorizon	Ahead of the best baseline in six of seven environments; not significant (p = 0.0625)

Benchmark

Used in

Result

KernelBench

TailHorizon

fast_1 of 48% / 78% / 22% on Levels 1–3, against 43% / 72% / 18% for the strongest baseline (three seeds)

RE-Bench

TailHorizon

Ahead of the best baseline in six of seven environments; not significant (p = 0.0625)

Evaluation protocols in our papers

Several papers define how a system will be evaluated before any result exists. Pre-registering the protocol stops us from choosing the test after seeing the outcome.

Paper	Protocol
Agent Kalki	Eight ordered admission predicates, k live runs per case, Bonferroni-corrected flake margin on a protected set
Viveka	Re-test each change on unseen seeds, replace lucky scores, remove changes that do not matter
The Harness Is the Lab	Held-out self-improvement curve at three nested levels
Pipit	Ten pre-registered hypotheses scored against same-machine baselines and each run’s physical roofline
Anudhi	Every kernel gated against a scalar reference and an independent oracle
VeriClust	Planned measurements on mathematical reasoning, software verification and fact-checking

Evaluation principles

The rules we apply to every result we publish, drawn from what went wrong in PrincipalBench v1.

Benchmarks and evals

Kalaris Labs benchmarks and evals for AI agents

Benchmarks we maintain

External benchmarks we report on

Evaluation protocols in our papers

Evaluation principles