coder-eval 0.12.2__tar.gz → 0.12.3__tar.gz

{coder_eval-0.12.2 → coder_eval-0.12.3}/.claude/harness-candidates.md

@@ -962,3 +962,50 @@ re-derive from scratch.
   leaves them stale with nothing failing. Needs a backtick-path extractor scoped to
   one section, which is the narrow case of the prose-path candidate above. — caught
   during the reports consolidation rebase.
+
+- [ ] **A prose "see X's docstring" citation whose target no longer holds the
+  claim.** Moving rationale out of a docstring leaves every citation of that
+  docstring pointing at text that is gone, and `check_pointers` cannot see it
+  because the citation is prose, not a `Rationale:` pointer. Not mechanised: the
+  match is heuristic ("see the module docstring", "see CE063's docstring",
+  "see that class's docstring" all read differently), and most citations are
+  in-file and still valid, so a rule would need per-site triage rather than a
+  regex. Instances found (line numbers at 946ca968):
+  `src/coder_eval/harbor/packager.py:382` and `:386`,
+  `tests/test_harbor_packager.py:163` and `:187`,
+  `tests/lint/rules/ce064_turn_bracket_on_the_clock.py:40`. Caught in: tests
+  prose slimming, Phases 3 and 7.
+
+- [ ] **Narrative after a docstring's `Args:`/`Returns:` block that follows a
+  `Rationale:` pointer.** `check_pointer_placement` accepts any tail that STARTS
+  with a trailing section, so pointer → `Args:` → an indented entry → a new
+  base-indented paragraph passes although the pointer is no longer the last prose
+  line. Guarding it means walking section blocks by indentation (the shape
+  `prose_words` already uses), plus false-positive triage over every src/ and
+  tests/ docstring that ends in a section — more than a quick add. Caught in:
+  tests prose slimming, final review (gpt-5.6-sol).
+
+- [ ] **A `Rationale:` pointer whose target is not under `.claude/notes/`.**
+  `check_pointers` joins the captured path onto the repo root, so an absolute
+  path, a `..` segment or any other Markdown file resolves, while CLAUDE.md and
+  `.claude/notes/README.md` define the pointer as a repo-relative notes path. No
+  such pointer exists today. Deferred because restricting the target is a design
+  decision (a `docs/` guide heading is a plausible SSOT target) rather than a
+  mechanical guard. Caught in: tests prose slimming, final review (gpt-5.6-sol).
+
+## From container-contract-and-command-surface (2026-09-15)
+
+- [ ] **A field-name assertion against CLI output can pass vacuously through `tmp_path`.**
+  `assert "variant_id" in result.output` matched the echoed `context.json` path, because
+  pytest names each `tmp_path` after the test (`test_a_non_string_variant_id_i…`). Nothing
+  guards it today; the fix was to assert the pydantic `loc` line (`"\nvariant_id\n"`).
+  Deferred because telling a vacuous substring from a real one needs to know the test's
+  own name and what the command echoes — a convention for reviewers, not an AST pattern.
+  Caught in: Phase 1 quality review.
+- [ ] **A path interpolated into Rich markup without `escape()`.** A run directory name is
+  untrusted, and `[/y]` in it raised `rich.errors.MarkupError` (not an `OSError`) inside
+  `evaluate`'s best-effort refresh, after the verdict printed. Nothing guards it; the fix
+  escaped every new console line. Deferred because the rule needs type information (which
+  f-string placeholders are `Path`s) that an AST-only rule does not have, and the existing
+  CLI has many pre-existing unescaped lines a literal rule would flag at once. Caught in:
+  Phase 5 quality review.

{coder_eval-0.12.2 → coder_eval-0.12.3}/.claude/notes/README.md

@@ -15,6 +15,7 @@ rule docstrings in `tests/lint/rules/`, then the guides under `docs/`.
 - [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity
 - [contracts.md](contracts.md) — criteria, datasets, aggregation, judging
 - [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading
+- [lint-rules.md](lint-rules.md) — why each CE lint rule exists
 - [orchestration.md](orchestration.md) — config merge, resume, early stop, execute vs. run
 - [permissions.md](permissions.md) — the chmod window and the reference anti-cheat
 - [persistence.md](persistence.md) — atomic writes and judge persistence
@@ -46,12 +47,31 @@ Path relative to the repo root, then `§`, then the target `##` heading text ver
 There is no other accepted form: `tests/lint/prose_budget.py` parses this one and fails
 `make docs-budget` when the file or the heading does not exist.
 
-## The prose budget is one number, not a lint rule
-
-`make docs-budget` reports the standing total and fails when it grows. It is deliberately
-**not** a `CE` rule: `tests/lint/rules/` polices per-pattern invariants one AST at a time,
-while this is a single whole-tree total. Making it a rule would mean a rule class, a rule
-test and a catalogue entry to enforce one integer — enlarging the harness the budget
+## The prose budget is not a lint rule
+
+`make docs-budget` runs `tests/lint/prose_budget.py` over `src/coder_eval` and `tests`.
+It enforces three rules with no baseline to maintain: no docstring over 150 prose words
+(an `@abstractmethod` and the Typer commands are exempt), no own-line comment run over 8
+lines, and a file's own-line comments within `MAX(20, 0.15 × lines)`. It also fails when a
+`Rationale:` pointer does not resolve or is not the last prose line of its block.
+
+The run cap is the primary rule and the file total is the backstop under it. The order
+matters, because the total alone was inverted: it blocked `isolation/docker_runner.py` at
+229/229 for carrying 62 short annotations pinned to the lines they explain, while a 16-line
+essay in `tests/test_regrade.py` sat at 29% of its budget. Four files had settled at exactly
+100% of the cap — a budget with an allowance becomes a target. The run cap has no allowance
+and catches the shape the total could not see; the total still catches a file that is mostly
+commentary however it is broken up.
+
+Measured when the cap was chosen: 78.5% of comment runs in the tree are 1-2 lines, 2.5% are
+6 or more, and the longest was 16. After the essays moved out, 21 runs sat at exactly 8
+lines. They were cut to 7 or fewer so that no run starts at the cap, and an ordinary edit
+to one of them does not fail the gate. The four files above still sit at 100% of their total
+budget, so a comment added to one of them has to be paid for by deleting another.
+
+It is deliberately **not** a `CE` rule: `tests/lint/rules/` polices per-pattern invariants
+one AST at a time, while this measures prose across whole trees. Making it a rule would
+mean a rule class, a rule test and a catalogue entry — enlarging the harness the budget
 exists to shrink. Do not "fix" this by promoting it.
 
 Nothing here states how many `CE` rules exist. `tests/lint/rules/` owns that count, and a
@@ -59,8 +79,8 @@ number written down anywhere else is a second declaration that will be wrong.
 
 ## Where a CE rule's rationale lives
 
-Each rule's authoritative rationale is its own module docstring under
-`tests/lint/rules/` — or, for the doc-surface and whole-tree rules, the corresponding
-`@pytest.mark.lint` class in `tests/test_custom_lint.py`. Read that before editing,
-suppressing or widening a rule. No prose summary is kept here: a second copy is a second
-declaration, and it is the one that goes stale.
+A rule's invariant, scope and blind spots live in its rule file under `tests/lint/rules/`
+(or its `@pytest.mark.lint` class in `tests/test_custom_lint.py`). Read that before you
+edit, suppress or widen a rule. The defect that motivated it lives in
+[lint-rules.md](lint-rules.md), under the rule's id. That file does not repeat the
+invariant, scope or blind spots.

{coder_eval-0.12.2 → coder_eval-0.12.3}/.claude/notes/agents.md

@@ -231,6 +231,27 @@ overlaps, so a call closing in the gap before a step start needs nobody to remem
 The rule that used to live there was wrong once on OpenCode — clearing at `step_start`
 wiped the span before `step_finish` could subtract it, a 100% overstatement of that window.
 
+The OpenCode gap case is reachable only through the NON-TERMINAL tool path. The CLI
+normally emits one already-`completed` event per call, and that call closes inside the
+step that opened it. That is why the measured corpus shows 0.00% for this case, and why
+`TestToolSpansSurviveTheStepBoundary` drives `_OpenCodeTurnState` directly instead of a
+recorded stream.
+
+The Pi inter-turn gap is small in practice: measured across 25 real window pairs, the
+median was 0.25 ms and the maximum 0.75 ms. The fix is worth
+keeping mainly because the tool spans must keep working once the gap closes, which
+`tests/test_pi_agent.py::TestToolSpansSurviveTheTurnBoundary` pins.
+
+An Antigravity call that is still running when a flush cuts the window must have its
+already-elapsed part taken out of that window. Subtracting only CLOSED intervals published
+that part as model time while the call's own `duration_ms` counted it again, and because
+the windows cover the turn end to end, there is no slack to absorb it. Measured on `tasks/hello_date`
+with a live gemini-3.1-pro-preview: a `Bash` opening 1.7 ms before the flush drove
+Σ generation + Σ command 0.26 ms PAST the turn's own `duration_seconds`, on a turn whose
+entire headroom was 1.4 ms. Four sibling runs passed by 1.2-8.7 ms out of ~12 s, so the
+defect was a coin flip per run. `tests/test_antigravity_agent.py::test_a_tool_still_open_at_the_flush_is_not_generation_time`
+pins it.
+
 ## Why a clean exit can still be a crash
 
 An exit code of 0 with no telemetry is indistinguishable from a real pass in every
@@ -267,6 +288,14 @@ reports no usage at all — where crashing every turn makes the harness unusable
 merely imprecise. It deliberately does NOT cover vocabulary drift: that arm has silently
 zeroed a whole run before, and no provider quirk explains it.
 
+The OpenCode golden fixtures (`tests/_fixtures/golden_streams/opencode_fixtures.py`) mirror
+events captured from a live `opencode run --format json`, in the CLI's own compact
+vocabulary (`step_start` / `step_finish` / `text` / `tool_use`, payload under `part`). The
+`session.next.*` names in the server's OpenAPI schema describe `opencode serve`'s SSE
+surface, not the CLI stream, so "correcting" a fixture toward them re-creates the
+zero-telemetry vocabulary drift above. The fixtures need no import guard because
+`pyproject.toml` declares `opencode = []`: there is no Python package to skip on.
+
 ## Token accounting, per harness
 
 Keep the buckets straight: `uncached_input_tokens` is the FRESH prompt slice only, because

{coder_eval-0.12.2 → coder_eval-0.12.3}/.claude/notes/contracts.md

@@ -158,6 +158,23 @@ interpreter and its directory goes FIRST on a PATH the orchestrator also reuses
 `run_command`, so `tool: python3` made the shim re-resolve its own interpreter to itself —
 an exec loop that spins to the task timeout.
 
+### Why the record_cli probe reads a real shim log
+
+The `tasks/record_cli_responses.yaml` probe needs a detector that proves per-invocation
+response dispatch, and most of its signals cannot. `cli_called` matches argv only, so it
+passes whether a response rule answered or the entry fallback did. The agent can
+transcribe `captured.txt`, because the task YAML is serialised to `/work/input` and mounted
+at `/work/task_dir`, and both are readable. A codegen regression that renders `RULES = []`
+raises nothing, so neither `rule_error` nor `sidecar_error` is booked. Only the
+`"rule": N` key in the log catches it.
+
+`tests/test_tags.py::TestRecordCliProbeIntegrity` therefore does not compare the YAML
+against itself. It runs each stubbed command through a real shim and matches the task's
+regexes against the lines the shim wrote. The needle's spelling (`"rule": 0`, with the
+space) comes from the default `json.dumps` separators in `invocation_log.record`. A test
+that copied the needle would stay green after a switch to compact separators, while the
+blocking CI probe failed.
+
 ## The checker base class
 
 ### Exactly one of _check_impl or _check_impl_async