Methodology: metrics

What each number on the results pages means, how it's measured, and what it doesn't tell you. The same tools and configs are run on every contestant at every milestone, so numbers are comparable across columns and over time. Terms are defined in the glossary.

Comparison happens in three layers. Acceptance tests (a Playwright suite the agents never see) measure whether the software works. Static analysis (this page, mostly) measures properties of the code mechanically. A structured review of architecture, code quality, and test quality — with every claim citing specific code — runs at milestones 2, 4 and 6. Static numbers are published for every milestone; only a small subset where "better" is unambiguous contributes to scoring.

Process metrics

Acceptance tests

An independent Playwright suite written against the product spec, which the agents never see. It drives each implementation through the same fixed contract every repo must satisfy — URL routes, data-testid attributes, seed fixtures, and a minimal JSON API — through real browser flows (as a New York visitor booking with a London host) and direct API checks, including a DST-transition check and a concurrent double-booking check. The suite is cumulative: after milestone k, all tests for milestones 1…k run, so a regression keeps costing until it's fixed. The score is passes / total after any fix-it prompts.

Fix-it prompts

When the acceptance suite fails after an agent declares a milestone done, the agent gets the failure output back in a standardized message — always the same wording: "The acceptance test suite reports the following failures. Please investigate and fix." followed by the verbatim test output. At most two rounds per milestone; after that the milestone is scored as it stands and the run moves on. The count is a measure of how often "done" wasn't done.

Wall-clock

Time from the milestone prompt being sent to the agent declaring it done. Excludes the operator's verification afterwards. Fix-it rounds, when they happen, are reported separately.

Turns

The number of request/response rounds between the harness and the model. Each turn re-sends the whole conversation so far, which is why input tokens are much larger than the actual content (see input tokens). Claude Code reports this directly; the Codex CLI does not expose it.

Actions

Tool calls counted from the session transcript: shell commands, file reads/writes/edits, browser automation, and other tool use. The categories differ per product (Claude Code reports named tools; Codex reports command executions, file changes, and MCP calls), so the breakdown is stated per column rather than forced into one scheme.

Output tokens

Text and code the model actually generated across the session, as reported by each product — the best proxy for work performed and the number used when efficiency is scored. Costs are stated in tokens, not currency: both agents run on subscription plans, so dollar figures would be synthetic.

Input tokens

Everything sent to the model across all turns, as reported by each product. Because the full conversation is re-sent every turn, input grows roughly with turns × context size, and most of it is a replay — the cached share is the replayed portion served from prompt cache. Descriptive context for output tokens.

Diff size

Lines added in the milestone diff (git diff m<k-1>..m<k>) — the change's raw footprint. Descriptive only; its trend across milestones feeds the marginal-cost signal in SCORING.md category E.

Self-verification

How the agent verified its own work before declaring the milestone done, as observed in the session transcript. The cell is a brief comma-separated list of the methods seen, drawn from a stable vocabulary where it fits: browser click-testing, browser checks (MCP), API probes, concurrency test, own test suite, typecheck, build. Descriptive only — whether any of that verification survived as committed tests is the committed tests review criterion.

Committed work

Whether the agent committed its milestone work to git during the session, unprompted — the milestone prompts never mention version control. Cells are yes/no, with a parenthetical when history helps (e.g. 'no (did at m1)'). Descriptive: a signal of engineering habit, not a scored behavior.

Code at this milestone

Reference links, not a measurement: each contestant's repository at the milestone tag (tree/m<k>) and the milestone diff (compare/m<k-1>...m<k>) — the exact code every number and review cell on the page describes.

Code metrics

Measured by the toolchain in analysis/, run against each repo at each milestone tag. Frozen configs apply identically to both contestants; whatever lint or format setup the agents built for themselves is ignored. Generated code (.next/, build output) and type declaration files are excluded everywhere.

Source size

Count of TypeScript source files and their non-blank lines, excluding tests and generated code. Descriptive only — it is never scored, because more code is not worse by itself. Where a rate matters (lint findings, complex functions), it's stated per unit of code so a larger codebase isn't penalized for its size.

Own tests

Tests the agent wrote for itself, counted by scanning test files (*.test.ts, *.spec.ts, __tests__/ and similar) for test cases (it(…) / test(…)). Also recorded: whether a test runner is installed and whether package.json has a test script. This counts existence, not quality — whether the tests are any good is a structured-review question (including, for each known trap in the scoring probe list: would this suite catch it?).

Type coverage

Measured with type-coverage: the percentage of identifiers whose TypeScript type is something more specific than any. 100% means every expression has a real type; numbers drop when any flows through code, silently switching type checking off for everything it touches.

Duplication

Measured with jscpd: the percentage of lines that appear in copy-paste clones at least 50 tokens long. The 50-token threshold is arbitrary but identical for both repos. The number says how much duplication exists, not whether it matters — duplicated domain logic is dangerous (fix a bug in one copy, miss the other), duplicated boilerplate is merely noise. That distinction is judged in the structured review.

Most complex function (cognitive complexity)

Cognitive complexity (SonarSource's measure) scores how hard a function is to follow: nesting, branching, and flow breaks add up; a score under 10 is easy to hold in your head, over 15 is generally considered in need of refactoring. We report the worst function in the repo. It's measured for every function via ESLint's sonarjs plugin, so full distributions (mean, p90) are in the published metrics.json too.

Functions over complexity 15

How many functions exceed the conventional refactor-me threshold of 15. One monster function and a codebase full of them read the same in a "max" number; this count distinguishes those cases.

Bug-prone lint findings

A frozen set of ESLint rules that flag likely-bug patterns rather than style: things like eqeqeq (loose equality), switch fallthrough, identical branches or sub-expressions, self-comparison, empty catch blocks, and misused loop conditions (full list in the frozen config). The agents' own lint setups are ignored; this rule set is applied uniformly. Findings are counted per rule.

Escape hatches

Occurrences of as any, @ts-ignore, @ts-expect-error, and eslint-disable — places where the author switched off a safety mechanism rather than satisfying it. A fenced one at a library boundary can be reasonable; the count says how often the codebase reaches for the override, and the review judges whether each use was justified.

Circular dependencies

Import cycles between modules (A imports B imports A), detected with dependency-cruiser. Cycles make code harder to understand, test in isolation, and split; they're also a common early sign of eroding module boundaries. The full module graph is captured as graph.dot alongside the metrics.

Orphan modules / unused exports

Two flavors of dead code. Orphan modules (dependency-cruiser): files nothing imports. Unused exports (knip): exported functions, types, or values nothing consumes. Both accumulate as a codebase evolves and earlier structures get abandoned rather than removed — one of the concrete forms "cognitive debt" takes. Caveat: framework entry points (pages, route handlers, config files) are recognized by the tools' Next.js support, but a small false-positive rate is possible; counts are best-effort and identical in method for both repos.

Churn (from milestone 2 onward)

Per milestone, from git history: gross lines changed (added + deleted) versus net lines added. A churn factor near 1 means new code was mostly laid down beside existing code; a high factor means the codebase was substantially reworked to absorb the change. Neither is good or bad alone — some milestones are designed to force rework, and how each agent absorbs them (clean refactor vs bolt-on) is exactly what the structured review examines at that point. Milestone 1 is the baseline: everything is new, so the factor is 1 by definition.

Task checks

Task-specific correctness, stated as plain criteria ("no double booking under concurrent requests", "times stable across a DST transition") and verified mechanically — each is backed by acceptance tests or probe runs against the live app, not by opinion. They differ from the generic process metrics only in being about this product's hard parts.

M1 acceptance checks m1 pre-registered

The frozen acceptance criteria for milestone 1 (public booking flow) — one card per check on the milestone page. harness/tests/m1-booking-flow.spec.ts + harness/tests/m1-slots-api.spec.ts — committed before the M1 contestant sessions ran (harness git history is the evidence); the cumulative suite re-runs them at every later milestone.

M2 acceptance checks m2 pre-registered

The frozen acceptance criteria for milestone 2 (event types, dashboard, auth) — one card per check on the milestone page. Scope note: the management UI for availability and event types has no testability-contract hooks at this milestone; those behaviors are exercised by the adversarial probes after M6 and the judged review. harness/tests/m2-auth.spec.ts + harness/tests/m2-dashboard.spec.ts — committed before the M2 contestant sessions ran (harness git history is the evidence); the cumulative suite re-runs them at every later milestone.

M3 acceptance checks m3 pre-registered

The frozen acceptance criteria for milestone 3 (manage, cancel, reschedule, notifications) — one card per check on the milestone page, verified by the frozen M3 suite additions including the email outbox capture. Scope note: host-initiated cancellation with an optional message has no testability-contract hooks at this milestone; it is exercised by the adversarial probes after M6 and covered in the review. harness/tests/m3-manage.spec.ts + harness/tests/m3-notifications.spec.ts — committed before the M3 contestant sessions ran (harness git history is the evidence); the cumulative suite re-runs them at every later milestone.

M4 acceptance checks m4 pre-registered

The frozen acceptance criteria for milestone 4 (correctness under load) — one card per check on the milestone page. Extended and re-frozen at M4 pre-flight before any session runs. harness/tests/m4-double-booking.spec.ts — committed before the M4 contestant sessions ran (harness git history is the evidence); the cumulative suite re-runs them at every later milestone.

Adversarial probes

The "demos well but wrong" traps, registered in the criteria registry and scored as SCORING.md category B. Pass/fail, run once against each contestant's final (post-M6) build running locally. The list may be extended before it freezes at M6 pre-flight, never during a run.

DST spring-forward

Slots for a London host on a US-mismatch week (Oct 25 – Nov 1 2026) render at correct wall-clock times for a New York visitor.

DST wall-clock stability

A London host's 09:00–17:00 window still yields 09:00-start slots in the week after clocks change.

Double-booking race

10 concurrent booking requests for the same slot → exactly one 201.

Cross-event-type conflict

Booking on event type X removes the overlapping slot from event type Y for the same host.

Minimum-notice boundary

A slot exactly at the notice boundary is handled consistently (no off-by-one exposing a bookable-in-the-past slot).

Buffer enforcement

Back-to-back bookings respect before/after buffers across different event types.

IDOR — manage link

Invitee A's manage token cannot cancel/reschedule invitee B's booking; tokens are not enumerable (not sequential IDs).

AuthZ — host boundary

Logged-in host A cannot read or cancel host B's bookings via direct URL or API.

Stale-slot submit

Submitting a booking for a slot taken after page load → clean 409/error UI, no double booking, no 500.

Daily-cap boundary

The cap counts bookings on the host's calendar day (timezone-aware), not UTC day.

Cancelled slot release

Cancelling a booking makes the slot bookable again; rescheduling releases the old slot atomically (no window where both or neither are held).

Round-robin fairness

6 sequential round-robin bookings distribute 2-2-2 across 3 hosts with identical availability (M6 feature).

Review

Everything above is mechanical. The review layer is where someone actually reads the code and forms a judgment — both generic (is this well built? what decisions were made?) and task-specific (is DST correct by design, or does it just pass today's tests? when a milestone invalidated an earlier assumption, was the model reworked or worked around?).

Two depths. A light review after every milestone: findings good and bad, each naming the file it's grounded in, organized as a side-by-side table — one row per recurring theme (domain core, time & DST, double-booking guard, error handling, tests…), one cell per contestant describing what it did, marked as a strength, a risk, or mixed. Cells are descriptive, never scored, so iterations can be compared without pretending to grading rigor the light review doesn't have. A full review at milestones 2, 4 and 6: structured scored forms covering architecture, code quality, and test quality, with every claim required to cite specific code — including the probe-catch rate (of the known traps, how many would the agent's own tests catch?). The light review's themes track the same form dimensions. Form definitions: JUDGING.md.

Disclosure: the reviewer is Claude (the agent orchestrating this experiment) while Claude Code is a contestant. Mitigations are structural — every claim must cite code anyone can check, full reviews run on blind-labeled checkouts, and all review text is published verbatim. Replication by a second, non-contestant model is planned for the full reviews.

Review criteria

Every theme row on a milestone page is a registered criterion from protocol/criteria.json — the build fails on an unregistered theme, so this list and the tables cannot drift apart. Each entry records when it was registered and how: pre-registered criteria are committed before the milestone's contestant sessions run (git timestamps prove the criteria predate the code); emergent criteria are added during a review because the findings demanded a row no one predicted; retrospective criteria are back-filled onto already-run milestones. Honest caveat: the criteria for milestones 1–3 were organized retrospectively from the review findings; pre-registration at prompt freeze starts at milestone 4. Where a criterion tracks a scored dimension of the judging forms, the dimension is noted.

Method, common to all criteria: Codebase read at the milestone tag; one descriptive cell per contestant, marked strength/risk/mixed, citing the file each claim is grounded in. Cells are one or two sentences, roughly 15–40 words — they must fit a table column. Use '—' where the review didn't assess a criterion for that contestant. Entries below note only what an individual criterion adds to that.

Standing — assessed at every milestone

Domain core tracks A3 retrospective 2026-07-21

Where the scheduling rules live and what shape they take: one composed engine or logic re-derived per endpoint; pure and unit-testable, or entangled with IO and framework code.

Time & DST tracks A4 retrospective 2026-07-21

How wall-clock ↔ instant conversion is handled: UTC at rest with conversion at the edges, and DST transitions handled by design (library or principled hand-rolling) or by luck.

Double-booking guard tracks A5 retrospective 2026-07-21

The mechanism that makes double-booking impossible — database constraint, lock, serialized transaction, or app-level check-then-insert — and whether it still holds as the schedule model grows. Also stress-tested mechanically by the concurrency acceptance tests and probes.

Boundary validation tracks A6 retrospective 2026-07-21

Validation at the API edges: request schemas, format checks, length caps, and whether malformed input is rejected before it reaches domain code.

Infra-failure handling tracks B3 retrospective 2026-07-21

Whether infrastructure failures are distinguished from domain outcomes: what a database outage or mail-adapter failure looks like to the caller, and whether anything is swallowed or misreported.

Committed tests tracks C1, C2, C3, C4, C5 retrospective 2026-07-21

Whether tests exist in the repository at the milestone tag. In-session verification that was discarded does not count; test quality is scored separately at the judged checkpoints. Repository inspection at the tag; cross-checked against static analysis's 'own tests' count.

Milestone-scoped

Guard's known gap m1 tracks A5 retrospective 2026-07-21

The specific weakness each contestant's double-booking guard leaves open, stated as the scenario that would defeat it.

Auth design m2 tracks A7 retrospective 2026-07-21

Session mechanics and credential handling introduced by milestone 2: what is stored at rest, how sessions are validated and expire, and where authorization is enforced (chokepoint vs per-handler).

Delete-as-archive m2 tracks A2 retrospective 2026-07-21

How event-type deletion preserves booking history: soft-delete mechanism, foreign-key policy, and whether historical records survive and stay accurate.

Reschedule write path m3 tracks A5 retrospective 2026-07-21

Atomicity of the reschedule swap: no window where both or neither slot is held, concurrent takers of the target slot resolved, and the new slot revalidated server-side.

Notification semantics m3 tracks B3 retrospective 2026-07-21

Exactly-once email behavior around booking mutations: no notification without a state change, no double-fire on repeats, and mail failure isolated from the committed write.

Email copy m3 retrospective 2026-07-21

Human-facing notification copy: whether times render as wall-clock in the relevant timezone or leak the contract's raw UTC instant format, and whether subjects identify the event and action. Also inspects the outbox capture (var/outbox.jsonl) from harness runs.

Manage token & authz m3 tracks A7 retrospective 2026-07-21

Manage-link token hygiene (entropy, storage, format gating, behavior on miss) and authorization scoping of cancel/reschedule actions across hosts and states.

Picker reuse m3 tracks B2 retrospective 2026-07-21

Whether the slot-picker experience required by the manage flow was reused as a shared component or duplicated — the run's clearest materiality test for code duplication. Read alongside the jscpd duplication delta between milestones.

DB-level invariant m4 tracks A5 pre-registered 2026-07-21

Whether at-most-one-confirmed-booking is now enforced by the database itself (exclusion constraint, unique index, or equivalent) or remains app-side discipline every write path must remember — and what happens when one forgets. The gap all four repos carried through m1–m3.

Cross-type conflict model m4 tracks A8 pre-registered 2026-07-21

How host-wide occupancy is modeled once a booking on any event type must block overlapping slots (including buffers) on all of them: the conflict rule reworked into one engine shared by GET /api/slots and POST /api/bookings, or patched separately per endpoint.

Concurrency proof m4 tracks C2, C3, C4 pre-registered 2026-07-21

The test the prompt demands: does it fire genuinely concurrent requests at a real database and assert exactly one confirmation — a test that would catch a regression — or does it approximate concurrency with sequential or mocked calls? Committed test read at the tag; where practical, re-run against the tagged app.

Judged dimensions

The dimensions of the full scored review, registered in the criteria registry and scored as SCORING.md category C2, at the checkpoint tags (m2, m4, m6; m7 for the epilogue). Scored 1–5 (3 = competent professional baseline; 5 = notably good; 1 = would not survive review by a competent team) on blind-labeled checkouts, fresh judge session per repo, mandatory cited evidence. Procedure and conflict-of-interest mitigations: protocol/JUDGING.md.

Architecture (Form A)

A1 · System map

Layers and dependencies: where domain logic lives (route handlers? server actions? a lib core?), what depends on what. One paragraph plus a module diagram, informed by the dependency-cruiser graph.

A2 · Data model fit

Reconstruct the ERD from the schema. Does it model the current milestone's reality, or fossilize a superseded one (e.g. availability still weekly-only with overrides bolted alongside; event types still host-owned with team fields nulled in)? Migration hygiene.

A3 · Domain core

Is slot computation a single engine or re-derived per endpoint? Are the rules (duration grid, buffers, notice, caps, overrides, cross-event conflicts) composed in one place? Pure and unit-testable, or entangled with IO/framework?

A4 · Time discipline

UTC at rest, timezone conversion only at edges? A deliberate approach (library or principled hand-rolling), or ad-hoc Date arithmetic scattered around? DST handled by design or by luck?

A5 · Write-path integrity

How is double-booking made impossible — DB exclusion constraint, serializable transaction, advisory lock, or app-level check-then-insert (i.e. hope)? Is reschedule atomic? What happens on retry?

A6 · Boundaries & contracts

Validation at the edges, coherent error taxonomy, correct status codes, API handlers thin vs fat.

A7 · AuthN/Z design

Session mechanics, manage-token generation (entropy, storage), authorization enforced at a chokepoint or re-remembered per handler?

A8 · Evolution response

When the milestone broke an earlier assumption: was the model migrated, or was a parallel structure / conditional layer bolted on? Quote the seam. (m4/m6 checkpoints only.)

A9 · Proportionality

Over-engineering is a defect too: abstraction layers nobody needs, premature generality, "enterprise cosplay". Equally: under-structure where duplication is already hurting. Judge fit to the problem as specified so far.

A10 · Consistency & idiom

Does the codebase read like one plan? Naming, file organization, framework idiom. Or does each milestone look like a different author visited?

Code quality (Form B)

B1 · Readability

Naming, function size, control-flow clarity. Would a new maintainer orient quickly?

B2 · Duplication (material)

jscpd gives the number; judge the materiality — is the duplicated code load-bearing domain logic (dangerous) or boilerplate (annoying)?

B3 · Error handling

Deliberate taxonomy vs try/catch-and-pray; conflicts, invalid input, and infra failures distinguished? Anything swallowed?

B4 · Dead code & vestiges

Informed by knip/orphan metrics; judge what the vestiges are — abandoned experiments, superseded endpoints, columns nothing reads.

B5 · Type discipline

Informed by type-coverage and escape-hatch counts; judge escape hatches in context — a fenced as any at a library boundary differs from any flowing through domain logic.

B6 · Comments & docs

Do comments state non-obvious constraints, or narrate the obvious / go stale? Is there a usable README/setup path?

Test quality (Form C)

C1 · Behavioral coupling

Do tests assert observable behavior, or mirror implementation structure so tightly that any refactor breaks them?

C2 · Boundary appetite

Do the agent's tests probe edges — DST weeks, the notice boundary, buffer adjacency, cap rollover, concurrent booking — or only the happy path?

C3 · Probe-catch rate

For each registered adversarial probe: would this suite have caught that bug? Score = caught/total, plus the checklist. The headline test-quality number and an objective anchor for the judged layer.

C4 · Test honesty

Tautological assertions, snapshot dumps nobody reads, mocked-to-meaninglessness (test passes with the domain logic deleted).

C5 · Test maintainability

Fixtures/factories vs copy-paste setup, isolation (order-dependence, shared mutable DB state), determinism (real clock? real randomness?).

Scoring

Most numbers on this page are descriptive and never scored. The scored static subset — type coverage, duplication, functions over complexity 15, bug-prone lint findings, escape hatches, circular dependencies, orphan modules / unused exports (the entries flagged scored: C1 in the registry) — is compared head-to-head as win/tie/loss at each checkpoint, avoiding fake precision. Full definitions and weights: SCORING.md.