coder-eval 0.12.2__tar.gz → 0.12.4__tar.gz

coder_eval-0.12.4/.claude/harness-candidates.md

@@ -0,0 +1,1011 @@
+# Harness Candidates
+
+Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
+(or a test) when picked up.
+
+## From code review 260701-1954 (fix-review-top5 run) — deferred to a dedicated guardrail plan
+
+> Numbering note: these are *proposed* ids. `CE024` (discriminated-unions) and
+> `CE025` (live-verdict consistency) have since been **implemented** for other
+> rules, so the candidates below were renumbered to the next free ids. Always
+> claim the next unused number in `tests/lint/rules/` — the id-uniqueness assert
+> in `tests/lint/runner.py` is the source of truth.
+
+- **CE026** — workflow-YAML rule: forbid any `uses:` step pinned to a floating ref
+  (`@v3`, `@main`) rather than a 40-hex commit SHA. Would have caught
+  `mxschmitt/action-tmate@v3` (fixed manually in this run). >30 min: needs a
+  non-Python file-walk branch in `tests/lint/runner.py`.
+- **CE027** — retired-token grep gate: fail when a removed-subsystem token
+  (`LLMGW_`, `API_BACKEND=proxy`, `uipath_llmgw_client`) reappears outside an
+  allowlist across docs/config/src. Would have caught the LLM-Gateway residue
+  swept in this run. >30 min: needs an allowlist + repo-wide text scan.
+- **CE028** — assert the Makefile `lint:` help does not hardcode a stale `CE0NN`
+  upper bound (use `CE001+`). Would have caught the `CE001–CE005` drift fixed here.
+- **docs-vs-harness smoke test** — execute the CI tutorial's `coder-eval run`
+  command against a NoOp task and assert the produced tree matches the documented
+  globs. Would have caught the `--run-dir runs` layout bug. Not statically
+  reachable (needs a live run).
+- [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CExxx at the next free id), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
+
+## From 2026-07-23 stop_when:auto early-stop review
+
+- [ ] CE-rule: the early-stop watcher stop rule must decide polarity via the
+  resolved `_armed_polarities`, never a raw `criterion.stop_when` comparison —
+  forbid `.stop_when` attribute reads inside `EarlyStopWatcher._evaluate` /
+  `_resolve_armed_polarities`'s callers in `orchestration/early_stop.py`. This
+  diff *was* the fix for exactly that class of bug (the old rule compared
+  `stop_when in ("pass","decided")` and so vetoed every mixed `auto` pass-stop).
+  Deferred, not cheap: existing CE rules scope by file/module, not by a specific
+  method, so a method-scoped attribute-ban needs a new AST-walk shape (and risks
+  false positives on the legitimate `is not None` membership reads elsewhere in
+  the file). Claim the next free id in `tests/lint/rules/`. Caught in the
+  2026-07-23 stop_when:auto review; the behavior itself is guarded by
+  `test_auto_mixed_pass_stops_ignoring_undecided_distractors` +
+  `test_mixed_static_arming_pass_stops_ignoring_fail_armed`.
+
+## From 2026-07-24 publish-github-releases review
+
+- [ ] **sdist-contents assertion** — build the sdist and assert it contains only
+  intended paths. `pyproject.toml` declares no `[tool.hatch.build.targets.sdist]`
+  section, so hatchling's default selection honors only the **root** `.gitignore`
+  and sweeps in everything else sitting in the tree at build time. Two distinct
+  consequences, worth keeping apart:
+  - **What actually reaches PyPI today: nothing unintended.** A local `uv build`
+    in a developed worktree produces a 135 MB sdist carrying
+    `evalboard/node_modules/**` (8520 files) and `evalboard/.next/**` (190),
+    because `evalboard/.gitignore` is nested and therefore not honored. CI is
+    spared only incidentally — `release.yml` never runs `npm`/`pnpm install`, so
+    those paths do not exist on the runner at `uv build` time. Verified against
+    the published artifacts: the 0.8.9 and 0.8.2 sdists on PyPI are ~7.5 MB /
+    ~550 files with **zero** `node_modules` entries. (A dirty-tree release would
+    not silently ship JS either — 135 MB exceeds PyPI's 100 MB per-file limit, so
+    it fails at upload. The real exposure is a broken release, not a stealth one.)
+  - **The live hazard is untracked files a workflow leaves in the tree**, which
+    hatchling *does* package: a `release-notes.md` written at the repo root by a
+    CI step landed in `coder_eval-X.Y.Z/release-notes.md` (verified by building
+    it). This is why the "Publish GitHub Release" step writes to
+    `${RUNNER_TEMP}` — a convention no check enforces.
+
+  Not cheap: needs a real `uv build` inside the test suite (slow) plus a decision
+  on whether to add an explicit sdist include/exclude allowlist, which changes
+  published artifacts. Worth pairing with the allowlist so the contract is
+  declared rather than inferred from hatchling's defaults — caught in the
+  2026-07-24 ci/publish-github-releases review.
+- [ ] **CE032 — run the existing AST lint rules over Python embedded in
+  `.github/workflows/*.yml`.** CE008/CE009/CE010 already forbid unencoded
+  `read_text`/`open`/`subprocess.run`, but `tests/lint/runner.py::check_paths`
+  walks only `*.py` under `src/`, so Python inside a `run:` heredoc is invisible
+  to ruff, pyright, pytest, coverage *and* the CE runner. Would have caught the
+  four unencoded `read_text`/`write_text` calls fixed by hand in this review
+  (`release.yml` ×2, `publish-testpypi.yml` ×2). Needs a heredoc extractor
+  (`python3 - <<'PY' … PY` → dedent → `ast.parse`) with line-number mapping back
+  to the YAML; wire as a `tests/test_custom_lint.py` class like CE027–CE031
+  rather than a `BaseRule`. Also consider extending CE008 to `write_text` (it
+  matches only `read_text` today, though `src/` happens to be clean).
+- [ ] **CE033 — interpreter heredocs in `.github/workflows/**` must use a quoted
+  delimiter** (`<<'PY'`, not `<<PY`). With a bare tag the shell expands `$VAR`
+  into the *program text* before the interpreter parses it, so a value containing
+  a quote or newline breaks out of the string literal it lands in. Fixed by hand
+  in `publish-testpypi.yml` in this review (it interpolated `${DEV_VERSION}` into
+  Python source); regex-detectable in ~10 lines, and CE032's `ast.parse` is only
+  sound on quoted bodies, so the two ship together.
+- [ ] **`actionlint` + `zizmor` over `.github/workflows/**`.** No static analysis
+  whatsoever runs over workflow YAML today (`make verify` never looks at it), so
+  every workflow finding in the 2026-07-24 review was caught by a human reading
+  it. `actionlint` runs shellcheck over `run:` bodies; `zizmor`'s
+  `excessive-permissions` / `artipacked` / `template-injection` rules cover the
+  credential-scoping and `${{ }}`-into-`run:` classes reviewed by hand. Subsumes
+  the CE026 SHA-pinning candidate above. Start as a non-blocking annotation job.
+
+## From 2026-07-03 open-source docs cleanup
+
+- [ ] **Dead-relative-link checker for `docs/**/*.md`** — resolve every relative
+  `](target.md)` link against the tree and fail on a missing target. During the
+  docs/features purge, the literal `git grep "docs/features"` gate missed 3
+  dangling links written in relative form (`](features/...)` in
+  TASK_DEFINITION_GUIDE.md ×2 and DOCKER_ISOLATION.md ×1); only a reviewer sweep
+  caught them. The cleanup plan explicitly deferred this as YAGNI for the
+  one-time purge, but any future doc rename/deletion re-opens the same blind
+  spot — caught in the 2026-07-03 open-source-docs-cleanup implementation run.
+
+## From PR #77 (command-executed shell-normalize) — CE030-to-criteria deferred
+
+- [ ] **Extend CE030 doc/schema-parity to the `SuccessCriterion` union** so a new
+  criterion (or field) can't ship undocumented. Attempted in PR #77 and reverted:
+  CI installs `--extra uipath`, and in that environment `coder_eval.models.criteria`
+  gains a `CliCalledCriterion` (fields `log`/`positional`) that is NOT present in a
+  plain checkout (it did not reproduce on macOS, whose lockfile resolution omits the
+  contributing linux-only component). It defeated every discriminator tried — union
+  membership, a `__module__` string filter (it is spoofed to `coder_eval.models.criteria`),
+  a genuine-module-attribute scan (it is `setattr` onto the module), and even an AST
+  parse of the `SuccessCriterion` union literal in `criteria.py` source (CI's imported
+  criteria module resolves to a file whose union literal already contains it). No
+  runtime OR source signal available in the lint could separate the injected criterion
+  from an in-tree one. Revisit only with a way to identify the in-tree criterion set that
+  is provably immune to the uipath integration — e.g. a hardcoded name allowlist of the
+  in-tree criteria (losing auto-coverage of new ones), or first understanding exactly how
+  that environment injects the criterion. Until then CE030 stays scoped to the four
+  top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is
+  documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless.
+
+## From the evalboard Path-to-GA de-tag / mature-passes fix (4e5bbc4…dd5f7e9) — TS-side guards deferred
+
+Context: the CExxx harness is a **Python** AST runner over `src/coder_eval/`, so none
+of the invariants below are mechanizable in it. Each would need a TypeScript lint
+harness (eslint config + custom rules) that `evalboard/` does not have today —
+standing one up for three call sites fails the KISS/YAGNI gate. Deferring rather
+than dropping; promote if a fourth TS-side invariant appears, and stand up the
+harness once for all of them.
+
+> **Update (PR #94 review round 2).** The *execution* half of this gap is closed:
+> `evalboard/` is now gated by the `evalboard` job in `.github/workflows/pr-checks.yml`
+> and reachable locally via `make evalboard-verify`, so the vitest suite (including
+> the pricing drift guard) is enforcement rather than documentation. What remains
+> deferred below is the *static-analysis* half — eslint has still not been stood up.
+> The review that prompted this round names four more candidate TS rules (raw
+> `status === "SUCCESS"` outside `lib/status.ts`; DOM-global shadowing in props;
+> inline copies of the tag predicate; per-run tooltip copy reused on aggregate
+> surfaces), which meets the "fourth invariant" promotion bar stated above —
+> **stand up eslint next time `evalboard/` is touched substantively.**
+
+- [ ] **"Every consumer of `RunOverviewTask.matureSkipped` must decide explicitly
+  whether a carry-forward row counts."** Four consumers now, and they deliberately
+  DISAGREE: `lib/trends.ts` and `app/runs/[id]/run-view.tsx` count a mature skip as
+  a pass; `lib/overview.ts::buildTagTaskRows` excludes it from both terms
+  (`/path-to-ga` is a GA-readiness page). A new consumer silently inheriting either
+  convention is a real hazard. Guard shape: flag a file that reads `.matureSkipped`
+  without a nearby comment naming its convention — weak, hence the deferral. Closed
+  for now by unit tests that assert the exclusion from BOTH numerator and denominator
+  (`lib/__tests__/overview.test.ts` → `describe("buildTagTaskRows")`).
+
+- [x] ~~**`taskCarriesRepoTag` is the single repo-provenance tag predicate — but one
+  duplicate survives.**~~ **RESOLVED in PR #94 review round 2.** The predicate moved to
+  a dependency-free `lib/tags.ts` (structurally typed on `{skill, tags}` so
+  `RunOverviewTask`, `TaskResultSummary` and `TaskTrend` all satisfy it), re-exported
+  from `lib/overview.ts` for existing callers. Both inline copies now import it:
+  `app/runs/[id]/run-view.tsx` (the `"use client"` one that could not before) and
+  `lib/trends.ts::trendMatchesTag` (a third copy the original deferral missed).
+  Still worth a lint rule ("no inline `tags.includes(x) || skill === x`") to catch
+  future copies — folded into the eslint promotion noted above.
+
+- [ ] **The de-tag rule fails CLOSED on a newest run that loads fine but stamps no
+  `tags`** (`lib/overview.ts::buildTagTaskRows`): every tagged task would read as
+  de-tagged and the table would empty, rendering an empty state indistinguishable from
+  a genuine full de-tagging. Its sibling failure mode (`overview == null`, a transient
+  blob read failure) IS guarded, with exactly this rationale. Currently unreachable —
+  0 of ~116k date-shaped non-ad-hoc task rows in `runs-remote/` lack `tags`, and the
+  six zero-tag runs found are all ad-hoc (filtered upstream by id shape + `meta.adhoc`)
+  — so the barrier is two upstream filters rather than a check at the seam. Left
+  unguarded on purpose: a `if (taggedInRun.size === 0) skip the de-tag signal` guard
+  would also mask a real, total de-tagging. Revisit if the pipeline ever stops
+  stamping tags, or if a non-ad-hoc run legitimately carries zero tagged rows.
+
+- [ ] **Discriminating-test discipline for predicate narrowings.** Two tests in this
+  change passed for the wrong reason — a downstream rule (the de-tag drop) masked the
+  mutation they claimed to catch — and the plan leaned on a `grep` acceptance criterion
+  that CI never runs. Both were found by mutation-testing the suite and fixed. No
+  mechanizable guard; the durable lesson is: when a test names a narrowing, construct
+  the fixture so the row SURVIVES every other rule, or the assertion proves nothing.
+
+- [ ] **CE038 — runner-label registry + dogfood runner parity** over
+  `.github/workflows/*.yml`. Two clauses: (a) every label a job can land on must appear
+  in `.github/actionlint.yaml`'s `self-hosted-runner.labels` or a stock GitHub-hosted
+  allowlist — including *both* branches of an expression-valued `runs-on:`, which
+  actionlint treats as opaque; (b) `action-dogfood`'s label must equal the one the
+  consumer snippet in `docs/tutorials/02-ci-pipeline.md` advertises. Nothing guards
+  either today: actionlint is not wired into `make verify` or pre-commit (grep: the
+  config file is its only mention), and CE026 parses that job's prerequisite *steps*
+  but never its `runs-on:`. Why it matters: an undeclared label is not a runtime error,
+  the job queues until GitHub cancels it hours later — indistinguishable from a pool
+  outage; and a repo-wide `runs-on:` migration has twice swept up `action-dogfood`
+  (#306, then 027121e in this PR), which exists precisely to prove the published Action
+  works on the image external integrators use. Implemented and verified once (both
+  clauses caught their regression class on the real tree) but reverted as out of
+  proportion to a 16-line runner migration — ~240 lines including tests. Note when
+  writing it: discriminate labels from expression operands structurally, on the
+  preceding `&&`/`||`, NOT on the string's shape — a "contains 'ubuntu'" heuristic
+  silently fails on `uipath-ubunut-latest`, the exact transposition typo the rule is
+  for. Caught in the multi-model review of PR #86.
+
+## From the 2026-08-04 Claude Code plugin marketplace run
+
+- [ ] **Plugin skills must not name a file that exists only in THIS repo** — the
+  `test_bundled_files_reference_no_repo_paths` denylist (`docs/`, `src/`,
+  `.claude/shared/`, `.claude/commands/`, `uv run`, `../`) deliberately allows
+  `tasks/` and `.claude/skills/`, because those are user-workspace paths the
+  skills legitimately scan and scaffold. So a skill body naming a specific repo
+  file (e.g. `tasks/hello_date.yaml`) would slip past the guard even though an
+  installed plugin is copied to `~/.claude/plugins/cache/` without it. The
+  obvious rule — "extract path-shaped tokens, fail if the path exists at the repo
+  root" — is NOT cheap: `init` legitimately tells users to scan `pyproject.toml`
+  and `package.json`, and `pyproject.toml` exists here, so the heuristic
+  false-positives on correct prose. Needs a token classifier that distinguishes
+  "a file to look for in the user's repo" from "a file in ours", which is a
+  design problem, not a 30-minute one. No skill violates it today (grepped) —
+  caught in the 2026-08-04 claude-code-plugin-marketplace implementation run.
+  *Update (2026-08-04, plugin-audit-p0-p1 run): the guard was renamed and widened
+  from `skills/*/SKILL.md` to every shipped text file under `plugins/coder-eval/`
+  (`PLUGIN_TEXT_FILES`), which closed the coverage half of this gap — a bundled
+  reference now cannot name a repo path either. The token-classifier problem
+  described above is unchanged and still deferred.*
+
+## From 2026-08-04 plugin-audit-p0-p1 run
+
+- [ ] **A skill's advertised `description` must not promise a check that no bundled
+  reference declares.** `lint-tasks` ships a user-facing description claiming it
+  finds "prompts that give away the answer", but that check was declared only in
+  `skills/task/SKILL.md` prose — a file `lint-tasks` never reads — so the two
+  rubric readers had already forked on it before the skill shipped. Caught by a
+  reviewer, not by a test; fixed by promoting it to rubric check 7. A guard would
+  have to map claim-phrases in a description onto declarations in
+  `reference/task-rubric.md`, which is natural-language matching, not a token
+  grep — the phrasings are deliberately different (a description sells, a rubric
+  check instructs), so any cheap version either misses the real case or fails on
+  correct prose. Needs a fixed vocabulary of claim tags shared between the two
+  files to become mechanical, which is a design change rather than a 30-minute
+  rule — caught in the 2026-08-04 plugin-audit-p0-p1 implementation run.
+
+## From the PR #82 review follow-up (2026-08-10)
+
+- [ ] **CE039 — documented `coder-eval` invocations must be executable as written.**
+  `init/SKILL.md` told the agent to run `coder-eval plan <task-directory>` and
+  "iterate until it exits 0", which the CLI rejects outright (`plan` takes files;
+  a directory argument exits 1 with a hint) — an unreachable loop condition
+  shipped in a skill. A rule would scan inline-code spans and fenced `bash` blocks
+  across `README.md`, `docs/**/*.md` and `plugins/**/*.md`, assert the subcommand
+  exists in the Typer app, and — the harder half — that the *argument shape* is
+  one the command accepts. The subcommand check is cheap and would not have caught
+  this; the argument-shape check is what matters and needs either a real
+  invocation (see the live-smoke candidate below) or a per-command arity model
+  that duplicates the CLI signature. Deferred on that split — caught in the PR #82
+  review, fixed by hand in `init/SKILL.md`.
+
+- [ ] **Documented-CLI live smoke.** The behavioural counterpart to CE039: in a
+  `-m live`/`-m slow` test, materialize a fixture repo with one task YAML and
+  execute every fenced `coder-eval …` command extracted from the shipped skills
+  and docs, asserting exit 0 (or an explicitly-expected non-zero). This is the
+  only form that proves argument shape rather than command existence. Not
+  statically reachable, hence separate from CE039 — proposed in the PR #82 review.
+
+## From the 2026-08-11 plugin generic-adopter run
+
+- [ ] **`working-directory` input on `action.yml`.** A repository whose eval tree is
+  nested (`tests/tasks/…`) has no way to tell the composite action to run from that
+  subdirectory, so every path in every input has to be spelled from the repo root. The
+  fix is a new input, and that is why it is deferred rather than cheap: `action.yml`'s
+  inputs are a **published API** — CE026 clause 4 asserts every `with:` key across four
+  onboarding surfaces is a real input, so adding one means updating those surfaces (the
+  `ci` skill among them, whose output lands in other people's repositories), and it
+  carries action tag/release implications. Out of scope for the plan that surfaced it,
+  which worked around it in the `ci` skill's prose instead.
+
+- [ ] **`shopt -s globstar` (or quoting `$CE_TASKS`) in `action.yml`'s run step.** The
+  real fix for a degradation the `ci` skill currently works around in prose:
+  `args+=($CE_TASKS)` is deliberately unquoted so a caller can pass several patterns, but
+  with `globstar` off `a/**/*.yaml` expands to `a/*/*.yaml` and **silently drops every
+  top-level task** — reproduced with `a/top.yaml` + `a/sub/deep.yaml`, which yields
+  `deep.yaml` alone. `nullglob` is off too, so an unmatched pattern reaches the CLI
+  literally and exits 1 (`Error: Task file not found: …`). One line in the action fixes
+  the first half; the second half is arguably correct-as-is (failing loudly beats
+  silently running nothing). Deferred alongside `working-directory` because both change
+  the action's observable contract and belong in one considered change.
+
+## From the final review of the 2026-08-11 plugin generic-adopter run
+
+Two **pre-existing `action.yml` defects** surfaced by an external reviewer during that
+run's final review. Neither is caused by the change, and `action.yml` was explicitly out
+of that plan's scope, so both are recorded here rather than fixed in passing. They belong
+with the two `action.yml` items above — one considered change to the action's contract.
+
+- [ ] **The score gate silently drops a malformed `weighted_score`.** `action.yml`'s
+  minimum-task-score step filters `task_results` rows down to usable floats; a row whose
+  score is a string, a bool, `NaN`/`inf`, or out of `[0, 1]` is omitted from the
+  comparison rather than failing it. So a `run.json` carrying one corrupt row **and** one
+  valid row above the floor gates **green**, which contradicts the fail-closed intent
+  stated in that step's own comment. The fix is to error on a present-but-invalid score
+  while still skipping `None` (errored tasks are already covered by coder-eval's exit
+  code). Wants a test over a synthetic `run.json` per bad-value class, which is why it is
+  not a five-minute change.
+
+- [ ] **`tasks:` is declared optional but omitting it cannot work.** The input defaults to
+  empty and the run step then appends no path arguments, so `coder-eval run` is invoked
+  bare — and zero-argument discovery resolves against the *installed package's* location,
+  finds nothing, and exits 1. The input is therefore effectively required, and the action
+  advertises otherwise. Either mark it `required: true` (a published-input contract change,
+  see the `working-directory` item) or fail with a clear message instead of an obscure
+  discovery error.
+
+## From the coder-eval-code-review of fix/antigravity-wait-for-wakeup (2026-08-12)
+
+- [ ] **A retry/poll loop's continuation state must derive from a stable per-entity
+  key, never a mutable monotonic counter used as an id fallback.** `_AntigravityTurnState._handle_tool_call`
+  minted a synthetic tool-call id from `f"{raw_name}_{self._next_seq}"` when the SDK's
+  `call.id` was falsy; since `_next_seq` advances between a tool call's ACTIVE and DONE
+  emissions, the DONE step computed a *different* fallback id than the ACTIVE step,
+  stranding the ACTIVE entry as a permanent orphan and stalling `communicate()`'s new
+  poll loop for its full `_MAX_BACKGROUND_POLLS` budget on every id-less turn. Fixed by
+  deriving the fallback from `(step.step_index, call_index)` instead (stable across a
+  step's own re-emissions, per this class's own docstring) -- then, in the same PR,
+  further folded in `step.trajectory_id` (falling back to bare `step_index` when it's
+  empty, mirroring the SDK's own `trajectory_id:step_index` id scheme), since a
+  sub-agent trajectory can reuse the same low `step_index` values as the main one and
+  two id-less calls across trajectories would otherwise collide. Not promoted to a CExxx rule:
+  this is the only id-fallback-driving-control-flow site in the codebase today (a
+  single call site, not a recurring class per the existing "single call-site fix, no
+  recurring pattern to guard" convention) — a mechanical AST rule for "no mutable
+  counter in a dict-key fallback" would need real design work to avoid false-positiving
+  on ordinary sequence-numbering counters elsewhere in the file. Caught by two
+  independent reviewers (Opus fallback pair) in this run's final code review.
+
+- [ ] **A `while` loop built around a cooperative-cancellation watchdog should read the
+  watchdog's own "already decided to fire" flag in its condition, not rely solely on a
+  later exception handler to notice.** The antigravity poll loop's condition checked
+  `not state.stopped_early_hit and state.has_orphaned_tool_call() and poll_count < cap`
+  but not `state.timeout_hit`, so if `ThreadedWatchdog`'s background thread set the flag
+  before its `task.cancel()` actually landed on this coroutine, the loop kept
+  sleeping/re-draining for up to the full poll budget before the pre-existing
+  post-loop `if state.timeout_hit:` check ever got a chance to run. Fixed by adding
+  `and not state.timeout_hit` to the condition, plus a mid-body early exit right after
+  the sleep (`if state.timeout_hit: break`) so a flag landing DURING the sleep skips
+  the following re-drain too, instead of waiting for the loop's next head check. Not
+  promoted: `ThreadedWatchdog` + a bespoke poll loop reading its own state flag is a
+  one-off shape unique to this agent; no second instance exists to generalize a rule
+  from. Caught in the same
+  final review as above.
+
+- [ ] **A regression test's fake dependency must model every layer the fix under test
+  actually touches, not just the outermost one.** `_drain()`'s cooperative-stop path
+  wraps a real SDK call (`Conversation.receive_steps()`) that is itself a delegating
+  async generator over an inner, connection-layer generator holding the real
+  re-entrancy guard. The first regression test written for this fix used a
+  single-layer fake (the guard lived on the SAME generator `_drain()` iterated), which
+  passed against an incomplete fix (`contextlib.aclosing` on the outer generator only)
+  that does not work against the real two-layer SDK shape — confirmed live that the
+  inner generator's cleanup is deferred to a LATER event-loop turn, not synchronous
+  with the outer's `aclose()`. Caught by a reviewer re-deriving the real dependency's
+  shape from its installed source, not by the test itself. Not promoted: detecting "a
+  test double is missing a delegation layer the source has" is a semantic match
+  against third-party source, not an AST pattern in our own code — no cheap mechanical
+  check exists. Caught in the round-3 coder-eval-code-review of this same branch.
+
+- [ ] **An agent's internal sleep-and-retry loop must derive its own exit bound from
+  the turn's actual `timeout`, never a fixed cycle count picked independently.** The
+  poll loop's own graceful exit path (force-close a never-resolving orphan as
+  unresolved, finalize and grade normally) was bounded by `_MAX_BACKGROUND_POLLS * _BACKGROUND_POLL_INTERVAL_SECONDS`
+  (120 × 5s = 600s) — DOUBLE `experiments/default.yaml`'s own default `turn_timeout: 300`.
+  Since the pre-existing `ThreadedWatchdog` enforces `timeout` by cancelling the whole
+  turn, it always won that race under default settings, making the graceful path dead
+  code: a tool call spuriously left ACTIVE with no real background job behind it (a
+  real, observed case — see the final validation run) went from "finalizes immediately,
+  graded on whatever the agent wrote" pre-fix to "burns the full 300s, then crashes as
+  `TurnTimeoutError` with zero criteria graded" post-fix — a strict regression for that
+  input class. Fixed by deriving a `poll_deadline` from a fraction (0.8x) of the actual
+  `timeout` passed to `communicate()`, falling back to the cycle cap only when
+  `timeout is None`. Caught independently by two reviewers (`bai-uipath`, `uipreliga`)
+  on the PR, both citing the exact same arithmetic mismatch. **Not promoted in this
+  pass**, but a stronger candidate than most entries here: `uipreliga` proposed a
+  generic whole-tree rule (proposed as CE035, renumbered CE042 here — CE035 shipped as
+  the workflow-outputs resolver on the published-action branch) — for every sleep-loop under
+  `src/coder_eval/agents/**`, assert its own cycle-count × interval either references a
+  timeout-derived name or is provably below `experiments/default.yaml`'s baseline — that
+  would catch this class of bug in ANY agent, not just this one (confirmed zero
+  violations on `main` before this bug, one on this PR). Worth a real look next time
+  `agents/` is touched, since a second agent adding its own disconnected sleep-loop
+  constant would reintroduce the exact same shape.
+
+## From 2026-08-04 published-action verification review
+
+- [ ] **CE041 — `VAR=$(… | grep …)` under `set -e` followed by an emptiness check
+  is a dead diagnostic.** With `set -euo pipefail`, a pipeline whose `grep` matches
+  nothing exits 1, so the assignment aborts the step *before* the
+  `if [ -z "$VAR" ]; then echo "::error::…"` branch that was written to report it —
+  the operator gets a bare exit 1 with no message. Also applies to `head -1`
+  closing the pipe early (SIGPIPE 141). Fix is `|| true` on the substitution,
+  letting the emptiness check own every failure mode. Detectable by matching
+  `\w+=\$\(.*\|\s*(grep|head)\b` inside a `run:` body whose script sets `-e`, then
+  requiring `|| true`/`|| :` on the same logical line. Caught by a reviewer in
+  `verify-published-action.yml`; **`actionlint` + shellcheck do NOT flag it**
+  (verified against the exact snippet), so the actionlint candidate above does not
+  subsume this one.
+- [ ] **CE036 — ban the skipped-green job gate.** Fail a job-level `if:` in
+  `.github/workflows/**` whose only discriminator is an emptiness/equality test on
+  `needs.<job>.outputs.<key>`. A lost output on a partial "Re-run failed jobs" resolves
+  the job to SKIPPED-**green**, so an operator sees a green re-run while nothing ran.
+  Fixed by hand twice now: `promote` was designed around the hazard, and
+  `publish-pypi`'s `if: needs.release.outputs.version != ''` (dead *and* dangerous — a
+  skipped publish also skipped `promote`) was removed in the follow-up review. CE035
+  catches the *typo* class; this catches the *shape*. Escape hatch: inline
+  `# noqa: CE036 — <reason>` for value-driven gates that cannot strand a release.
+- [ ] **CE037 — `if: failure()` is wrong in a job containing a `continue-on-error`
+  step.** Require `always()` (or a reference to the tolerated step's
+  `steps.<id>.outcome`) on diagnostic/upload steps in such a job. Fixed by hand in
+  `verify-published-action.yml`: the run dir was discarded in exactly the tolerated-red
+  case the gate is designed around, because a tolerated red leaves the job green and
+  `failure()` never fires. Pure YAML shape check, ~30 lines.
+- [ ] **CE040 — cap inline `run:` bodies; oversized decision logic belongs in
+  `.github/scripts/`.** `verify-published-action.yml`'s parity step (~70 lines, 7
+  decision points) and its e2e gate (~66 lines, switching from bash to a `python3`
+  heredoc mid-step) are 10-20-branch units invisible to `make check`, `make lint`,
+  `pyright` and coverage — which is the structural reason the `steps.parity.outputs.version`
+  bug survived to `main`. Analogous to CE022's statement cap; composes with CE032/CE033.
+  Deferred as a refactor, not a fix: extraction touches all 423 lines of a workflow that
+  cannot be exercised before merge, and CE035 + `tests/test_verify_published_workflow.py`
+  now cover the specific failure classes. Precedent for the extraction:
+  `.github/scripts/release_notes.py` + `tests/test_release_notes.py`.
+- [ ] **Exercise the Action's score gate in the FAILING direction.** Both
+  consumer-simulating jobs pass `minimum-task-score: "0.0"`
+  (`verify-published-action.yml`'s `e2e`, `pr-checks.yml`'s `action-dogfood`), so the gate
+  is only ever proven to *pass*. The new exit-contract assertion catches a gate that
+  wrongly fails; nothing catches one that wrongly passes — the direction that silently
+  disables every consumer's quality gate. Needs a second invocation with an unmeetable
+  score floor, i.e. a second paid agent run per nightly; deferred on cost, and better
+  placed in `action-dogfood` (PR-time, already paying) than in the cron.
+- [ ] **Extend CE026's `REQUIRED_PREREQ_TOKENS` anchor to the `e2e` job.** The lint pins
+  the documented Node + `@anthropic-ai/claude-code` prerequisite steps to a single
+  executable reference (`action-dogfood` in `pr-checks.yml`, via
+  `tests/lint/action_docs.py::DOGFOOD_JOB`). `verify-published-action.yml`'s `e2e` job is
+  now a third copy of the same two steps — and the truer consumer proof (no checkout,
+  published action, default pin) — so the two can drift while the docs follow only one.
+- [ ] **Runtime-key parity for `run.json` consumers outside `src/`.** The e2e gate in
+  `verify-published-action.yml` reads `task_results[*].status` / `weighted_score` /
+  `total_tokens`, and `action.yml`'s score gate reads `weighted_score` / `task_id`.
+  These are string keys in shell/YAML that no test or type-checker binds to
+  `eval_result_to_task_dict` (`run_record.py`), so renaming a key there
+  silently turns an external gate into a no-op — a reviewer here proposed
+  `final_status`, which does not exist in `run.json` and would have made a new
+  assertion dead on arrival. Guard: assert the key set that non-Python consumers
+  depend on, mirroring how CE030 pins doc/schema parity.
+- [ ] **A probe task's detector must not be satisfiable from the sandbox-readable task
+  YAML.** `docker_runner._stage_inputs` serialises the post-override `TaskDefinition` to
+  `/work/input/task.yaml` and mounts `tasks/` again at `/work/task_dir`, both agent-readable.
+  Two probes now depend on NOT being satisfiable from that text — `anti_cheat_reference` via a
+  regex that cannot match its own source, `record_cli_responses` via a log-derived detector —
+  and nothing enforces it. `record_cli_responses` originally shipped (in review) with
+  `file_contains` needles that were verbatim in its own YAML, which would have let a
+  transcribing agent pass while dispatch was dead. Guard: for every `smoke-pass` task, assert no
+  `file_contains` needle / `file_matches_regex` pattern on a must-match criterion appears in the
+  serialised task YAML. Deferred: needs per-criterion-type handling and a real false-positive
+  pass (paths and generic words will collide), so well over 30 min.
+- [ ] **A new shim failure mode must still RECORD the invocation.** A generated shim that dies
+  before `record()` leaves a log byte-identical to "the agent never ran it", which passes a
+  `max_count: 0` guard. The sidecar import was exactly that, caught only in review. Guard:
+  render each shim shape, break each external dependency in turn, assert the log is non-empty.
+  Deferred: "each external dependency" has no enumeration today, so the rule needs a seam
+  (a declared list of what a shim depends on) before it can be mechanical rather than a
+  hand-maintained list that decays.
+- [ ] **A generated-artifact invariant must be asserted against a real run of that artifact,
+  not against the config that produced it.** `TestRecordCliProbeIntegrity` first shipped
+  comparing the task YAML with itself and hardcoding `"rule": 0` — a spelling `json.dumps`'s
+  default separators own — so a separator change would have left it green while the blocking CI
+  probe failed. Now fixed for this case by running a real shim. Deferred as a general guard:
+  "derives its expectation from the thing it checks" is not mechanically detectable; it belongs
+  in the review rubric rather than a lint rule.
+
+- [ ] A `_*TurnState` (agent turn-state) attribute that is written but never read
+  outside its own assignment — CE037-class dead accumulator. Surfaced during the
+  Pi harness port: `_PiTurnState.turns_finished` was copied from OpenCode's
+  `steps_finished` (which drives that agent's `finished_without_tokens` guard) but
+  Pi deliberately dropped that guard, leaving the counter dead. Fixed by hand this
+  run. Guard would need cross-method dataflow over each `Agent`-subclass turn-state
+  class (write sites vs read sites), which the AST-only CExxx runner can't express
+  in ~30 min — deferred. Caught in: Pi harness Phase 2 quality review.
+
+- [ ] A duration/count aggregate over run-task rows that fails to exclude
+  `mature_skipped`. Nothing guards it: the CExxx runner is Python-AST only and
+  this defect class lives in the evalboard's TypeScript. A codex nightly
+  rendered "1300 tasks · 15h 29m" for the 397 tasks that actually ran. Fixed
+  for the two whole-run sites by extracting `deriveRunDuration`; a general
+  guard needs a TS lint surface the harness does not have.
+  Caught in: timing-capture Phase 1 review.
+
+- [ ] Subtracting a SUM of intervals from a wall span where the intervals can
+  overlap. Semantic, not syntactic — an AST rule cannot tell a sum of
+  durations from a union. Antigravity shipped it: four concurrent 400ms tool
+  calls inside a 1000ms window summed to 1600ms and clamped generation to the
+  0.0 the change existed to remove. The real guard is the replay-based
+  `assert_timing_captured` golden sensor, which now exists; a static rule
+  would not have caught it. Caught in: timing-capture Phase 3 review.
+
+- [ ] A test fixture whose field name does not exist on the type it models.
+  `parseMessages`'s `CommandEntry` keys on `tool_id`; a fixture using
+  `tool_use_id` never resolves, params fall back to `{}`, and every tool
+  weighs exactly 1 — which silently turned a "split by content size" test
+  into a 99%-thinking assertion that passed. TypeScript accepts it because
+  the fixtures are untyped object literals. Typing the fixture factories
+  against the real interfaces would guard the whole class; that is a
+  sweep across the evalboard test suite, not ~30 min.
+  Caught in: timing-capture Phase 5 review.
+
+### Deferred timing divergences (not guardrails — accounting gaps)
+
+Recorded here as well as in `docs/agents/HARNESS_PARITY.md` § Known
+divergences, so the deferred-work record is one place. Measurements in
+`c/time-bugs-audit.md`.
+
+- [ ] **Antigravity books orphan-poll waiting as agent duration** (audit P2-1).
+  A task can spend `0.8 × turn_timeout` waiting on a tool call that never
+  reaches DONE — 14 tasks, 9.6h of one 83h run. Only CLOSED tool intervals are
+  subtracted from a generation window, so that wait stays inside whichever
+  window contains it; the force-close records `execution_completed_at` while
+  leaving `duration_ms` as `None`. Deliberately out of scope of the timing
+  work: closing it means deciding whether a backgrounded tool's elapsed time
+  is model time (the model IS generating while it runs), which is a semantic
+  question, not a bug fix.
+
+- [ ] **Delegate (`delegate-sdk`) records no execution bounds** (audit P3-1).
+  It reports `duration_ms` but neither `execution_started_at` nor
+  `execution_completed_at`, so its tool calls cannot be placed on a timeline.
+  Coverage is ~88%, so it is not urgent. The agent lives in the separate
+  `coder_eval_uipath` repo; mirror the Codex change there
+  (`_item_timing` + threading the SDK stamps through the telemetry builders).
+  **Consequence, as of the turn-timing consolidation (2026-09-12):** such a
+  call now contributes to no bucket on EITHER surface — the evalboard's
+  `toolExecutionMs` lost its `durationMs` fallback, matching the `is not None`
+  filter Python has always had — so the time reads as Unaccounted rather than
+  as tool execution. Pinned by `unbounded_cases` in
+  `tests/_fixtures/timing_union_cases.json`, which both suites replay.
+
+- [ ] **Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens`
+  double-counts the cache buckets.** The validator copies a legacy record's
+  full-prompt `input_tokens` straight into `uncached_input_tokens`, and the
+  computed `input_tokens` then adds `cache_creation` + `cache_read` again. A
+  legacy record with `input_tokens=1000`, `cache_creation=200`, `cache_read=150`
+  reloads as 1350 prompt tokens and bills 1000 at the uncached rate instead of
+  650. Affects every report, budget check and detached regrade over a
+  pre-split run that used prompt caching. NOT touched by the timing work
+  (token accounting was explicitly out of its scope) and not a guardrail
+  candidate — a real bug needing its own change, with a decision about
+  whether legacy records can be distinguished from current ones at all.
+  Caught in: timing-capture final review (gpt-5.6-sol).
+
+- [x] ~~No golden-corpus assertion of the four-bucket identity.~~ **DONE.** The
+  fixture clocks were unified (`_rebase_notifications` / `_rebase_lines` shift
+  codex's 2027 base and opencode's month-old base onto the replay's own clock,
+  keeping every derived duration exact) and `assert_timing_captured` now
+  asserts `Σ generation + ∪ tool + head + tail` against `duration_seconds`.
+  Mutation-verified: reintroducing the defect fails
+  `test_antigravity_golden[d_orphaned_tool]`, which previously passed.
+  22 of 27 scenarios are checked. The remaining 5 are exempt via
+  `FICTIONAL_DURATIONS` for a reason rebasing cannot fix: they inject SDK
+  stamps in integer MILLISECONDS (17-900 ms of declared item time) while the
+  replay runs in well under one, so closing that last gap needs the agent's
+  own clock faked, not the fixtures' rebased.
+
+- [x] ~~No TypeScript counterpart to CE058.~~ **DONE.**
+  `evalboard/lib/__tests__/no-zero-coalesce.test.ts` is a vitest source scan
+  (there is no eslint in `evalboard/`) over `lib/runs.ts`, `lib/timing.ts` and
+  `_sections.tsx`. It is an ALLOWLIST rather than a ban, exactly because the
+  residual arithmetic uses `?? 0` correctly — each of its 14 entries carries a
+  one-line reason, and a new occurrence fails until its author justifies it or
+  keeps the value null. It is keyed on the codebase's own `…Ms` naming
+  convention rather than on every `?? 0`: a blanket scan matches 58
+  occurrences, ~40 of them token buckets where zero is a fine answer, and an
+  allowlist that long is one nobody reads. Blind spots are declared in the
+  file. Two meta-tests keep it honest — a negative control (so the scan cannot
+  pass by matching nothing) and an assertion that every allowlist entry is
+  still present, so an entry cannot outlive its reason.
+  Caught in: turn head/tail timing final review.
+
+- [x] ~~**`timing.py::decompose_turn` raises an uncaught `TypeError` on a
+  naive/aware datetime mix**~~ **DONE.** `timing.py::_require_same_awareness`
+  now raises from five call sites (`decompose_turn`'s head and tail,
+  `busy_ms`'s window bounds and each span's two ends) with one message template
+  naming the field and which side is aware. Deliberately a GUARD and not a lint
+  rule: the invariant is still unviolated in-tree, and the exposure that
+  actually matters is a third-party agent registered through the
+  `coder_eval.plugins` SPI, which lives outside `src/coder_eval/agents/` and
+  which a rule scoped to that directory could never see — so the message
+  addresses that reader directly. An empty span list is checked not at all,
+  bounds included: nothing is compared, so there is no pair to be about.
+  Caught in: turn head/tail timing final review.
+
+- [x] ~~**`claude-code` does not subtract tool execution from its generation
+  windows.**~~ **FIXED** in `_ClaudeTurnState._subtract_tool_time_from_windows`,
+  which runs at finalization (it cannot run at flush time — a tool issued by an
+  earlier emission is still running when the next window closes). Re-measured
+  on the same task: 481 ms / 2.691% -> **1.4 ms / 0.006%** over four turns that
+  all carried overlapping tool calls. Original report kept below for the
+  reasoning.
+
+  ORIGINAL: The other four
+  harnesses subtract the union (`timing.py::busy_ms`); claude-code is exempted
+  on the reasoning that it "marks the end of the previous SDK event and reads
+  again when the next message arrives, so a tool's execution falls between two
+  windows rather than inside one". But a tool's timer starts at the **emission**
+  carrying its `tool_use` block, and one assistant turn spans several emissions,
+  so a later emission's window runs concurrently with a tool already timing.
+  Measured live on a task with five parallel writes, five reads and two
+  concurrent `Bash` calls: the generation/tool overlap was **482 ms and 340 ms**
+  on two ~18-25 s turns, and the four-bucket residual came out at exactly
+  `-481 ms` / `-339 ms` — the overlap accounts for it to within 1.4 ms. The
+  other four harnesses overlapped by ~2.0-2.3 s on the same task and reconciled
+  to within 1.2 ms. Two claude-code turns with <1 ms of overlap reconciled to
+  within 0.1 ms, so the fault is precisely the missing subtraction.
+  Fix is to apply `busy_ms` in `on_assistant_message` as the other four do, but
+  it changes a PUBLISHED `generation_duration_ms` on the most-used harness, so
+  it needs its own golden regeneration and live pass. NOT introduced by the
+  head/tail work — generation-vs-tool timing predates it — but that work's
+  four-bucket identity is what made it visible.
+  Caught in: post-merge live verification of the head/tail buckets.
+
+### Deferred lint-rule widenings
+
+- [ ] **CE058 and CE059 still match `AssistantMessage` by a hardcoded constructor
+  NAME LIST** (`_MESSAGE_CONSTRUCTORS`), where CE060 derives the set from each
+  module's own `coder_eval.models` imports. The weakness is live, not
+  theoretical: `claude_code_agent.py` binds *only*
+  `AssistantMessage as AssistantMessageTelemetry` and never the bare name, so
+  the two shipped rules guard that file's two construction sites purely because
+  somebody wrote the current alias into a different file's frozenset — rename
+  the alias and both go silently blind there — and an arbitrary
+  `AssistantMessage as Msg` is missed outright by both. Adopting CE060's
+  alias-resolving `check()` pre-pass is about ten lines per rule, but it widens
+  two SHIPPED rules whose firing sets are load-bearing (CE058's constructor set
+  is a different, wider one: `CommandTelemetry`, `SlowestCommandInfo`,
+  `TurnRecord`), so it needs its own mutation check per rule and a re-measured
+  firing set over all of `src/`, not a drive-by edit. If a fourth same-scope
+  kwarg rule ever lands, extract `tests/lint/rules/_message_calls.py` at that
+  point rather than sooner.
+  Caught in: the CE060 / antigravity `message_id` run.
+
+- [ ] **Nothing pins that `message_id` is only ever a WITHIN-TURN identity.** Ids
+  repeat across retry attempts of one turn on every synthetic-id harness —
+  `Agent.discard_pending_turn` rolls the iteration counter back, so a crashed
+  partial and its retry both emit `<harness>-1-msg-0` (antigravity, codex, and
+  the out-of-tree delegate agent alike). Harmless today, and verified so: the
+  evalboard declares its grouping list INSIDE the per-turn loop
+  (`runs.ts:1822`, flushed at `:2217`) and only ever compares adjacent raws, and
+  no Python consumer reads the field at all. It stops being harmless the moment
+  anything joins on the id run-wide (a React key across turns, a cost join, a
+  dedup) — which is a natural thing to reach for once every harness populates
+  it. No cheap guard exists: the property to assert is "no consumer treats this
+  as run-unique", which is a negative over two languages, and asserting
+  within-turn uniqueness instead would pass today and catch nothing. Cheapest
+  real option is a comment on the model field; the durable one is a run-level
+  id if a consumer ever needs one.
+  Caught in: the CE060 / antigravity `message_id` final review.
+
+- [ ] **No evalboard test is fed by a Python golden snapshot.** The two halves of
+  a capture fix are pinned by two hand-written fixtures that never meet: the
+  golden (`tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json`)
+  pins what the reducer emits, and `evalboard/lib/__tests__/parseMessages.test.ts`
+  pins what the consumer does with a fixture an author typed from the same
+  understanding. Nothing feeds a real recorded shape through `parseMessages`, so
+  a reducer change that makes the TS fixture unrepresentative breaks no test on
+  either side. Deferred as architectural: it needs a loader, a scrub-aware
+  timestamp story (the goldens mask exactly the stamps the grouping reads), and
+  a convention for which snapshots the JS suite owns — well over 30 min, and
+  wider than any one capture fix.
+  Caught in: the CE060 / antigravity `message_id` final review.
+
+- [x] ~~**`AssistantMessage.message_id`'s field description names one harness of
+  five**~~ **DONE.** It said "Anthropic API message_id … when the Claude Code
+  CLI splits one API response", while five backends write the field and four
+  synthesize it — so `docs/agents/HARNESS_PARITY.md`'s row was the real SSOT
+  and the model, which this project's DRY principle designates as
+  authoritative, described claude-code only. Rewritten agent-agnostically: what
+  the id MEANS (the generation an emission belongs to), that all five write it
+  and four synthesize it, each scheme named, a pointer to the per-harness row,
+  and the fact that it is a WITHIN-TURN identity that repeats across retry
+  attempts. No mechanical guard was added and none is obvious — "a field
+  description must not name a single harness when the union has five writers"
+  needs a writer census per field, which is CE054-shaped but over a `str`
+  description rather than a key; the cheap version was exactly this, fixing the
+  sentence in the next change that touches the model.
+  Caught in: the CE060 / antigravity `message_id` final review.
+
+- [ ] **The golden corpus pins that a timing value EXISTS, never what it is.**
+  `tests/_fixtures/golden_streams/_scrub.py::SCRUB_KEYS` masks
+  `generation_duration_ms`, `started_at`, `completed_at` and both
+  `execution_*_at` to a placeholder, and the one assertion that does look at
+  magnitudes (`assert_timing_captured`'s four-bucket check) is an UPPER BOUND —
+  it catches a bucket claiming more time than the turn contains and says
+  nothing about one claiming less. So the committed suite cannot see a
+  per-harness generation number move at all, in either direction. Not
+  hypothetical: a whole phase of the timing plan was written on the premise
+  that changing those numbers would turn the golden master red, and it never
+  did. The two-sided check exists (`scripts/timing/decompose_run.py
+  --max-residual-pct`) but runs only against live `task.json` files, by hand.
+  Not cheap to guard: porting the two-sided residual into `_scrub.py` means
+  deciding a per-scenario tolerance for replays whose real wall clock is under
+  a millisecond while their SDK stamps declare hundreds — the same problem
+  `FICTIONAL_DURATIONS` already exempts six scenarios from, so the honest
+  version needs those scenarios to fake the agent's own clock too, not just
+  their item stamps. Interim cover is the per-reducer ms-exact identity test
+  added on pi and opencode
+  (`test_the_four_bucket_identity_closes_exactly_across_the_boundary`).
+  Caught in: the timing-architecture-standardization final review.
+
+## From the turn-timing P0–P3 run (2026-09-12)
+
+- [ ] **A golden scenario's justification comment can contradict its own
+  snapshot, and nothing notices.** Three did in this run: two orphan-tool
+  comments asserted bounds the committed JSON plainly carries (`pi_d`,
+  `opencode_d`), and `opencode_c`'s exemption claimed "the snapshot still
+  records the tiling" while `SCRUB_KEYS` masks both bounds and the duration.
+  Each was found by a human/model reading the JSON beside the prose — nothing
+  mechanically ties an exemption's stated reason to what its snapshot contains.
+  A rule would have to parse prose, so this is probably not guardable; the cheap
+  substitute is the review instruction that already exists ("read every new
+  snapshot before committing") plus the habit of quoting the actual JSON in the
+  comment. Caught in: turn-timing P0–P3, phases 2 and 5.
+
+- [ ] **A rationale comment asserting a now-false premise survives a ripple that
+  updated its siblings.** The "in-process SDK" claim was corrected in six files
+  and left standing in two (`test_event_collector.py`,
+  `message-timeline.test.tsx`), one of them directly beside a sibling that WAS
+  updated. Same shape as CE026/CE047 (doc-surface parity) but over a PHRASE
+  rather than a symbol, so a rule would be a phrase blocklist with an
+  ever-growing allowlist. Deferred on cost, not on value — a grep for the retired
+  phrase in the acceptance criteria is what actually caught these, and that is
+  cheap to write into a plan.
+
+- [ ] **`EventCollector` retains `_commands` and `_turn_starts` across a retry's
+  `AgentStartEvent`**, which resets only `_agent_end`. Pre-existing and NOT
+  introduced by the timing work. Blast radius is narrower than it first looks:
+  the persisted record, the reports and `max_turns` all read the AGENT's
+  collector, which is fresh per `communicate()`. Only `EarlyStopWatcher`'s
+  long-lived collector accumulates — where carrying a turn's whole engagement
+  across retry attempts is arguably what a live "did it engage the skill"
+  verdict wants, and `_check_round`'s docstring already reasons about crashed
+  attempts. Needs a decision on intent before any guard. Caught in: turn-timing
+  P0–P3 final review.
+
+- [x] ~~**claude-code has no `TurnClock`.**~~ **RESOLVED.** `_ClaudeTurnState`
+  now takes an injected `TurnClock` and every wall stamp the turn records
+  derives from it — both window bounds, the tool span
+  `_resolve_pending_command` stamps (which takes the reading as an argument, so
+  the span and the bounds it is clipped against cannot end up on two clocks),
+  and the fallback tool timestamp. One raw `datetime.now()` is deliberately
+  left, on the synthesized sub-agent terminal message: those bounds are an
+  admitted placeholder that `subtract_tool_time` and the head/tail bracket both
+  exclude, so no arithmetic reads them and there is no basis to share. The
+  ms-exact sensor was re-pointed at the injected clock rather than the module
+  `datetime` — a derived stamp escapes a monkeypatch, so the old patch would
+  have left `tests/test_timing_identity_contract.py` measuring the real clock
+  and passing by accident; reverting the conversion now fails it by ~10^7 ms.
+
+- [x] ~~**`pi_agent` publishes a `duration_ms` and a subtracted tool SPAN for an
+  UNRESOLVED orphan.**~~ **RESOLVED.** `_close_tool` now stamps
+  `execution_completed_at` and derives `duration_ms` only when the status is
+  not `UNRESOLVED`; the guard the old comment claimed is the guard the code
+  has. `execution_started_at` is kept (the CLI really did emit that start) and
+  one bound alone forms no span, so the orphan no longer has time subtracted
+  from a generation window it never occupied. `pi_d_orphaned_tool.json` now
+  records both fields as `null`.
+
+  **Sibling, NOT fixed:** `antigravity_agent` does the same thing at its own
+  orphan sweep (`tel.model_copy(update={..., "execution_completed_at":
+  self.clock.now()})`), though it stops short of a `duration_ms`. Deliberately
+  left: `timing.decompose_turn`'s docstring reasons about that stamped
+  completion landing inside the tail, and the `antigravity_d_orphaned_tool`
+  residual was measured against it, so changing it is a separate piece of work
+  with its own fixture to re-derive — not a ride-along.
+
+- [x] ~~**`pi_agent` republishes a turn's content on a duplicate `turn_end`.**~~
+  **RESOLVED.** `turn_text_parts` / `turn_tool_ids` are now cleared in
+  `on_turn_end` beside `turn_started_at`, on the argument that comment already
+  made: all three have been SPENT into the message just appended.
+  `pi_f_duplicate_turn_end.json` now records the second message with an empty
+  `content_blocks` and no `tool_use_ids` — it books the duplicate's own usage
+  and nothing else. The timing half had a unit test that stayed green while the
+  content half was broken, so the two are now asserted separately
+  (`test_a_duplicate_turn_end_does_not_republish_the_previous_content`).
+
+## From the turn-timing consolidation (2026-09-12)
+
+Two findings from `c/turn-audit.md` were CUT during planning, on evidence. They
+are registered here with their corrected cost/benefit and the trigger that would
+reopen them — not because they are cheap guards waiting to be written, but
+because the reason they were cut is the part a later reader will otherwise
+re-derive from scratch.
+
+- [ ] **A2-full — reducers publish BOUNDS only; the collector derives the raw
+  window.** The audit justified this partly as retiring two lint rules. Neither
+  holds. `CE058` form 1 is a generic keyword rule over five constructors
+  (`ce058_no_timing_literal.py:71-76`) and stays live whatever a reducer
+  publishes. `CE059` keys its exemption on `generation_duration_ms=None` being
+  PRESENT at the call site (`ce059_generation_window_is_two_reads.py:68`), so
+  removing the kwarg makes `claims_a_window` true at the three legitimate
+  placeholder sites and forces a rule REWRITE rather than a retirement. Net
+  cost: five reducers, a regeneration of every golden, and a CE059 rework; net
+  benefit: SSOT alone. **Deferring it is safe because the seam assertion in
+  `timing.subtract_tool_time` now checks the property at runtime** — a group's
+  raw total must equal the span its own bounds describe — which also covers a
+  third-party agent registered through the `coder_eval.plugins` SPI, where no
+  lint rule scoped to `agents/` reaches. REVISIT IF: that assertion ever has to
+  be relaxed for a legitimate reducer, which would mean the equality is no
+  longer the contract and storage has stopped paying for itself.
+
+- [ ] **C2 — a persisted `clock_inversions` counter.** The audit wanted the
+  number of clamped negatives recorded on the turn. CE064 removed the reachable
+  cause (the cross-basis head/tail comparison), and a field nothing may ever
+  read is YAGNI. The DOC half was done instead: the two contradictory clamp
+  docstrings now state one position — a measured inversion IS a real zero,
+  because both ends were observed (`timing.decompose_turn`), and claude-code's
+  case was never about the clamp but about the head being measured against the
+  wrong instant. REVISIT IF: an inversion is observed on a live run after
+  CE064, which would mean a basis is still mixed somewhere the rule cannot see
+  (the plugin SPI, or a harness whose spans come from a CLI).
+
+- [ ] **`test_codex_golden[a_agent_message_only]` is FLAKY, ~5% — measured, and
+  pre-existing.** Forty consecutive runs on an unmodified tree (`-n 0`): 2
+  failures, `"no assistant message reports a positive generation window with
+  bounds that span it"` with `(0.0, '...164797', '...164797')`. The replay
+  finishes faster than `datetime`'s 1 us resolution, so codex's rebased item
+  stamps can collapse to one instant and the window rounds to `0.0` — which
+  `assert_timing_captured`'s `expect_generation_window` arm then correctly
+  refuses. Surfaced (not caused) by the turn-timing consolidation, which runs
+  that file repeatedly. Not fixed here because the fix is in the codex fixture's
+  stamp rebasing (`_rebase_notifications`), which is its own change with its own
+  risk of ratifying whatever it then produces; the honest options are to give
+  the fixture's items a floor above the clock's resolution, or to give the
+  scenario `expect_generation_window=False` and say why. Caught in: the
+  turn-timing consolidation, Phase 6.
+
+- [ ] **CE058 misses a sixth form: `<expr> if <test> else <numeric literal>`.**
+  Its five forms are a zero constructor keyword, `x or 0`, `x if x is not None
+  else 0.0`, an `if x is None: x = 0.0` assignment, and a `model_copy(update=)`
+  dict. Form 3 keys on an `is None` / `is not None` COMPARISON, so the shape the
+  single production writer of `tool_union_ms` actually uses —
+  `union_ms(tool_spans) if tool_spans else None`, a truthiness test on a list —
+  is invisible to the rule in either polarity. Nothing ships wrong today (that
+  line correctly writes `None`, and `test_a_turn_with_no_bounded_span_records_none_not_zero`
+  covers it), but an author flipping it to `else 0.0` would publish "measured,
+  and instant" with the rule silent. Candidate: a form that fires on an
+  `ast.IfExp` whose `orelse` is a numeric literal and whose assignment target —
+  or enclosing timing-constructor keyword — matches `_TIMING_NAME`. Deferred
+  because the target-name resolution is new machinery rather than a variant of
+  an existing form. Caught in: the turn-timing consolidation, Phase 5 review.
+
+- [ ] **Pre-existing, surfaced by the turn-timing final review:
+  `stats.regularized_incomplete_beta` clamps an out-of-domain `x`
+  instead of raising.** Its docstring says "Raises ValueError outside that
+  domain — returning NaN would let a bad input render as a real-looking
+  statistic downstream", and it does raise for a non-finite `a`/`b`/`x` and for
+  a non-positive `a`/`b`. But the boundary branches are `if x <= 0.0: return
+  0.0` / `if x >= 1.0: return 1.0`, so a NEGATIVE `x` or one above 1 silently
+  becomes a valid-looking probability — exactly the outcome the docstring says
+  it prevents. `x == 0.0` and `x == 1.0` are legitimately in the domain, so the
+  fix is to split the equality from the inequality, not to tighten the branch.
+  The internal Student-t callers construct an in-range `x`, so nothing ships
+  wrong today; the exposure is a future or external caller. NOT touched by the
+  timing work (the function is zero lines of its diff) and not a guardrail
+  candidate — a small real bug needing its own change. Caught in: the
+  turn-timing consolidation final review (gpt-5.6-sol).
+
+- [ ] **A comment line that opens mid-sentence directly after one that ended.**
+  The residue of a block replacement whose anchor matched the wrong line: the
+  tail of the replaced prose survives as a severed fragment. The exact-form half
+  of this — a `Rationale:` pointer that is not the last line of its block — was
+  PROMOTED in the prose-mass-reduction run and now ships as
+  `prose_budget.check_pointer_placement`. What remains is the general case,
+  where no pointer is involved, and it is heuristic: a legitimately wrapped
+  sentence looks identical to a severed one, so it needs an allowlist (a
+  continuation opening with a backtick, a quote, or a list marker is usually
+  fine). ~30 min plus the false-positive triage. Caught in: prose mass
+  reduction, Phases 4-7 review.
+
+- [ ] **Two `.claude/notes/` sections covering ONE topic under different
+  headings.** The single-home rule is the load-bearing invariant of the notes
+  tree and nothing enforces it. `check_pointers` proves a pointer resolves;
+  nothing proves the topic is not also argued three files away. It bit every
+  phase of the prose-mass-reduction run, including once against a file the
+  phase never opened (`reporting.md` vs `orchestration.md` on
+  `nothing_was_measured`). Needs a similarity measure over section bodies —
+  shared rare tokens, or a shared symbol name appearing as the subject of two
+  headings — so it is real work rather than a regex. Deferred on cost, not on
+  value: this is the highest-value unbuilt guard in the notes design. Caught
+  in: prose mass reduction, all phases.
+
+- [ ] **A `Rationale:` pointer that resolves to a heading which does not hold
+  the rationale that left the site.** The weaker sibling of the above and the
+  same shape of miss: the gate goes green while the reader arrives somewhere
+  unhelpful. Five instances in Phase 5 alone, all fixed by hand. Probably not
+  mechanizable without a semantic check, but worth recording as a known blind
+  spot of `check_pointers` so nobody reads its green as "the pointers are
+  good". Caught in: prose mass reduction, Phases 5-6 review.
+
+- [ ] **A criterion-class first docstring line changing without
+  `make plugin-reference` in the same commit.** CE033 already diffs the
+  generated `plugins/coder-eval/reference/criteria.md`, so drift IS caught —
+  but only for classes that reach the generated file, and only as "the
+  generated file is stale" rather than "you edited a generated surface". A
+  commit-scoped guard would name the cause. Deferred because the lint harness
+  has no access to a commit-scoped diff today; the ad-hoc version
+  (AST-comparing every `ClassDef` first line against a base ref) was written
+  and used throughout Phase 6 and is the thing to promote if that access
+  appears. Caught in: prose mass reduction, Phase 6.
+
+- [ ] A generated surface (`*.generated.*`) has no mechanical guard against being hand-edited
+  — CE065/CE033/CE028 all catch *drift* (source changed, output not regenerated) but an edit
+  to BOTH passes cleanly. Guarding it needs a checksum or a git-attribute gate, not a diff,
+  so it is a different shape of sensor. — caught during the reports consolidation (CE065).
+- [ ] No rule resolves file paths named in PROSE (comments, docstrings, Markdown) across
+  `src/`, `evalboard/`, `litellm/` and `.github/`. That consolidation hand-fixed ~25 stale
+  module references across five phases, and two reviewers each found more the greps missed.
+  The plan's Open Questions measured and declined the CLAUDE.md-only variant (its stale refs
+  live in an ASCII tree, not backticks); a wider variant has the same parsing problem plus
+  legitimate non-resolving refs (container paths, plugin-relative paths). Recorded because
+  the recurrence is now the argument, not the idea. — caught during the reports consolidation.
+- [ ] The anchored package regex `(?:^|[/\\])src[/\\]coder_eval[/\\]` is compiled
+  independently across the rule tree — `ce050_no_union_getattr_probe.py:101`,
+  `ce051_no_driver_override.py:60`, `ce052_process_lethal_must_be_container_gated.py:78`,
+  `ce053_run_record_filename_literal.py:65`, `ce054_env_info_key_round_trip.py:66`,
+  `ce056_no_container_env_literal.py:51` and `ce058_no_timing_literal.py:116` — seven
+  rule modules, to which `_layers.py` adds one more (its `_CLI` and `_REPORTS` derive from it), with the
+  `agents/`-suffixed variant of the same idiom in
+  `_model_ctor.py:28` and `ce059_generation_window_is_two_reads.py:45`, plus a near-variant
+  in `ce037_no_dead_private_helper.py:61` and a `cli/`-suffixed one in
+  `ce048_no_in_process_typer_command_call.py:68`. `_layers.py` is the designated shared rule-helper
+  module, though `_model_ctor.py` is an equal peer and a generic src-path regex arguably
+  belongs in a neutrally named helper rather than one named `_layers`. Not hoisted here
+  because retargeting seven unrelated rules needs a per-rule verification that its scope
+  did not shift — a second refactor inside a review-fix plan. The new copies were written
+  in the established *spelling* deliberately: the defect being fixed was a regex that
+  disagreed with its siblings, so a new variant would be that defect again. — caught during the reports-consolidation review fixes, Phase 1.
+- [x] ~~**CE004 inherits CE066's `reports/` exemption because the two rules share one
+  predicate.**~~ **DONE.** `_layers.py` now shares the package anchor and the `cli/`
+  boundary (`is_package_path`, `is_cli_path`) rather than one exemption set. CE066 keeps
+  `is_core_path` (`{cli, reports}` exempt); CE004's scope is the package minus `cli/`.
+  Widening CE004 to `reports/` found 0 violations. `test_the_reports_package_is_in_scope`
+  and `TestCoreLayerMembership.test_ce004_scope_is_every_module_outside_cli` both fail if
+  CE004 goes back to the core predicate; `test_the_reports_package_itself_stays_exempt`
+  pins that CE066's scope did not widen with it. — caught in the reports-consolidation
+  review fixes, Phase 1 quality review.
+
+- [ ] **A CLAUDE.md Directory Structure bullet naming a path that no longer exists.**
+  Shipped briefly as CE067 over the fenced `coder_eval/` tree, then removed when that
+  tree was replaced by `ls` plus selective bullets — the exhaustive half of the rule
+  became false by design. The surviving half is still real: the bullets name modules
+  (`result_metrics.py`, `reports/html.py`, `models/container_paths.py`) and a rename
+  leaves them stale with nothing failing. Needs a backtick-path extractor scoped to
+  one section, which is the narrow case of the prose-path candidate above. — caught
+  during the reports consolidation rebase.
+
+- [ ] **A prose "see X's docstring" citation whose target no longer holds the
+  claim.** Moving rationale out of a docstring leaves every citation of that
+  docstring pointing at text that is gone, and `check_pointers` cannot see it
+  because the citation is prose, not a `Rationale:` pointer. Not mechanised: the
+  match is heuristic ("see the module docstring", "see CE063's docstring",
+  "see that class's docstring" all read differently), and most citations are
+  in-file and still valid, so a rule would need per-site triage rather than a
+  regex. Instances found (line numbers at 946ca968):
+  `src/coder_eval/harbor/packager.py:382` and `:386`,
+  `tests/test_harbor_packager.py:163` and `:187`,
+  `tests/lint/rules/ce064_turn_bracket_on_the_clock.py:40`. Caught in: tests
+  prose slimming, Phases 3 and 7.
+
+- [ ] **Narrative after a docstring's `Args:`/`Returns:` block that follows a
+  `Rationale:` pointer.** `check_pointer_placement` accepts any tail that STARTS
+  with a trailing section, so pointer → `Args:` → an indented entry → a new
+  base-indented paragraph passes although the pointer is no longer the last prose
+  line. Guarding it means walking section blocks by indentation (the shape
+  `prose_words` already uses), plus false-positive triage over every src/ and
+  tests/ docstring that ends in a section — more than a quick add. Caught in:
+  tests prose slimming, final review (gpt-5.6-sol).
+
+- [ ] **A `Rationale:` pointer whose target is not under `.claude/notes/`.**
+  `check_pointers` joins the captured path onto the repo root, so an absolute
+  path, a `..` segment or any other Markdown file resolves, while CLAUDE.md and
+  `.claude/notes/README.md` define the pointer as a repo-relative notes path. No
+  such pointer exists today. Deferred because restricting the target is a design
+  decision (a `docs/` guide heading is a plausible SSOT target) rather than a
+  mechanical guard. Caught in: tests prose slimming, final review (gpt-5.6-sol).
+
+## From container-contract-and-command-surface (2026-09-15)
+
+- [ ] **A field-name assertion against CLI output can pass vacuously through `tmp_path`.**
+  `assert "variant_id" in result.output` matched the echoed `context.json` path, because
+  pytest names each `tmp_path` after the test (`test_a_non_string_variant_id_i…`). Nothing
+  guards it today; the fix was to assert the pydantic `loc` line (`"\nvariant_id\n"`).
+  Deferred because telling a vacuous substring from a real one needs to know the test's
+  own name and what the command echoes — a convention for reviewers, not an AST pattern.
+  Caught in: Phase 1 quality review.
+- [ ] **A path interpolated into Rich markup without `escape()`.** A run directory name is
+  untrusted, and `[/y]` in it raised `rich.errors.MarkupError` (not an `OSError`) inside
+  `evaluate`'s best-effort refresh, after the verdict printed. Nothing guards it; the fix
+  escaped every new console line. Deferred because the rule needs type information (which
+  f-string placeholders are `Path`s) that an AST-only rule does not have, and the existing
+  CLI has many pre-existing unescaped lines a literal rule would flag at once. Caught in:
+  Phase 5 quality review.