Glossary

The terms used across the experiment pages, defined once.

Experiment

One app built from one frozen spec and milestone sequence — e.g. Slotbook, a scheduling app. An experiment compares how different contestants handle the same work.

Run

One complete pass of an experiment's milestone sequence. Everything published so far is from Slotbook run 1.

Harness

The coding-agent product that drives a model: Claude Code, Codex. The harness decides how the model sees the workspace, what tools it gets, and how sessions and context are managed — so two harnesses on comparable models can behave very differently. (Separately, "the harness" in this repo's code refers to the acceptance-test suite in autocode/harness — an unrelated sense of the word.)

Model

The LLM the harness drives, including its settings — e.g. Opus 4.8, or gpt-5.6-sol at high reasoning effort.

Iteration

One complete, independent pass by the same harness + model combination: fresh workspace, same frozen prompts, same rules. Multiple iterations show how much a contestant's results vary when nothing else changes. With 2–3 iterations the pages can show spread, not statistical significance — differences between iterations are observations.

Contestant

One competing instance: harness × model (with settings) × iteration. Each contestant has its own workspace, repo, and database, and appears as its own column or bar on the results pages — e.g. CC-Opus4.8-i1 is Claude Code driving Opus 4.8, iteration 1.

Milestone

One frozen prompt in the experiment's sequence, given to every contestant verbatim. Each milestone is scored independently; the acceptance suite is cumulative, so earlier behavior keeps being checked.

Acceptance suite

The independent Playwright test suite that scores each milestone. Written from the spec before contestant sessions start, frozen from then on, and never shown to contestants. Details under methodology.

Fix-it prompt

The standardized message returning acceptance-suite failures to a contestant after it declared a milestone done — at most two rounds per milestone. Defined under methodology.