Slotbook

Coding agents build a Calendly-style scheduling app from the same seven frozen milestone prompts, on the same stack (TypeScript, Next.js, Postgres). An independent Playwright test suite scores each milestone: every implementation must satisfy a fixed testability contract — URL routes, data-testid attributes, seed fixtures, and a minimal JSON API — so one suite drives any conforming UI.

Scheduling was chosen because it's easy to demo and hard to get right: timezones, DST, double-booking races, buffer and notice rules. Later milestones also change earlier assumptions, so the experiment shows how each agent handles reworking its own code.

A contestant instance is a harness × model × iteration: an iteration is a fresh run of the same frozen prompts with the harness version pinned, measuring run-to-run variation (see the glossary). Iterations catch up from milestone 1 and appear on each milestone page as their runs complete.

Protocol, prompts, harness, and raw results: martinpllu/autocode

Contestants

Milestones