The Harness Is the Lab: measured self-improvement for AI scientists

Key points

• Generic coding harnesses are now commodities. Research scaffolds evolved for specific tasks are not.
• Research memory should be a typed ontology of hypotheses, runs and results, not a chat log.
• A claim of recursive self-improvement without a held-out improvement curve is marketing.
• The proposed harness keeps its evaluators sealed and improves at three nested levels (search policy, harness skills, model weights). Each level accepts a change only on held-out evidence.
• The paper situates this against Karpathy’s autoresearch, Sakana’s Agentic Scientist, Aster, Recursive’s Eureka Machine, Prime Intellect’s Prime Agent and Edison’s Kosmos.

Limitations

Abstract

Every serious AI-scientist system released in 2026 has converged on the same lesson: generating hypotheses, code and papers is no longer the hard part; deciding which results deserve to be believed is. Karpathy’s autoresearch showed that a sealed evaluator, a fixed budget and a git ratchet can run a hundred useful experiments overnight. Sakana’s Agentic Scientist, Aster, Recursive’s Eureka Machine, Prime Intellect’s Prime Agent and Edison’s Kosmos each add a missing piece: evidence generation, massive parallelism, a model-harness flywheel, a self-editing harness, and a structured world model. This paper is a position and a blueprint. We argue seven positions, among them that generic coding harnesses are now commodities while task-evolved research scaffolds are not; that research memory must be a typed ontology rather than a chat log; and that any claim of recursive self-improvement without a held-out improvement curve is marketing. We then describe an open research harness that grounds every claim in an ontology of hypotheses, runs and results, keeps its evaluators sealed, and improves itself at three nested levels (search policy, harness skills, model weights), each accepted only on held-out evidence. We define the evaluation protocol we will hold ourselves to, including a self-improvement curve no public system has yet reported. We report no experimental results here; we state falsifiable predictions instead.

Related research

• Agent Kalki: Agent Kalki is a Rust coding-agent harness in which a gate the agent cannot write to admits each self-modification only when eight ordered predicates hold.
• Viveka: Viveka is a checking layer that re-tests a self-improving agent’s changes on fresh seeds, so that changes kept by chance do not become the bar for real progress.
• Inventing the Variable: The HiddenLaw benchmark tests whether AI can introduce a variable its vocabulary lacks. Passive data blocks every learner; with interventions, Claude Opus 5.5 succeeds in 9 of 9 worlds.

Cite this paper

@misc{chowdhury2026theharnessisthelab,
  title     = {{The Harness Is the Lab: Measured Self-Improvement for Ontology-Grounded AI Scientists}},
  author    = {Chowdhury, Sayan},
  year      = {2026},
  publisher = {Zenodo},
  note      = {Preprint, Kalaris Labs},
  doi       = {10.5281/zenodo.24144800},
  url       = {https://doi.org/10.5281/zenodo.24144800}
}