coder-eval 0.8.8__tar.gz → 0.8.9__tar.gz

{coder_eval-0.8.8 → coder_eval-0.8.9}/.claude/commands/coder-eval-code-review.md

@@ -12,7 +12,7 @@ description: Run a multi-model code review on uncommitted changes or a described
 
 Run a thorough code review using multiple AI models in parallel, then fix all high-confidence medium-severity and above findings.
 
-**Cost note**: This command runs `make verify` + a multi-model review (`gemini-3` + `codex` via MCP, plus an Opus sub-agent) + automatic fixes. Expensive in time and tokens. Use this for changes you're about to ship; for broad codebase audits use `/coder-eval-code-review-full`; for quick local checks, run targeted tools (`ruff`, `pyright`, `pytest`) directly.
+**Cost note**: This command runs `make verify` + a multi-model review (`gemini-3` + `gpt-5` via MCP, plus an Opus sub-agent) + automatic fixes. Expensive in time and tokens. Use this for changes you're about to ship; for broad codebase audits use `/coder-eval-code-review-full`; for quick local checks, run targeted tools (`ruff`, `pyright`, `pytest`) directly.
 
 Input: $ARGUMENTS
 
@@ -65,7 +65,7 @@ If it fails, report what's failing and stop — code review on broken code is pr
 
 Launch two reviews **in parallel**:
 
-**Review A — Multi-model (via `mcp__multi__codereview`)**: run it per the shared procedure in `.claude/shared/multi-model-review.md` — `models: ["gemini-3", "codex"]`, `relevant_files` = absolute paths of all changed files, `content` = a review request citing the Severity Standard, Review Principles, and Checklist. **Heed the multi-step protocol described there** (the step-1 response is usually an `in_progress` checklist, not findings — you must make the `step_number: 2` follow-up call with the same `thread_id`). Falls back to two Opus sub-agents if the tool is unavailable.
+**Review A — Multi-model (via `mcp__multi__codereview`)**: run it per the shared procedure in `.claude/shared/multi-model-review.md` — `models: ["gemini-3", "gpt-5"]`, `relevant_files` = absolute paths of all changed files, `content` = a review request citing the Severity Standard, Review Principles, and Checklist. **Heed the multi-step protocol described there** (the step-1 response is usually an `in_progress` checklist, not findings — you must make the `step_number: 2` follow-up call with the same `thread_id`). Falls back to two Opus sub-agents if the tool is unavailable.
 
 **Review B — Opus sub-agent**: Use the `Agent` tool with `model: "opus"`. Give it the list of changed files and the following specialized tasks to run in parallel internally:
 
@@ -120,7 +120,7 @@ across sessions and provides forensic context if a fix later proves wrong:
 - Git SHA: <`git rev-parse HEAD`>
 - Branch: <`git rev-parse --abbrev-ref HEAD`>
 - Scope: <uncommitted | described "<input>" | last commit>
-- Reviewers: <e.g. gemini-3, codex, opus-fallback-1, opus-fallback-2 — list actual reviewers used>
+- Reviewers: <e.g. gemini-3, gpt-5, opus-fallback-1, opus-fallback-2 — list actual reviewers used>
 
 ### Scope
 - Files reviewed: (list)

{coder_eval-0.8.8 → coder_eval-0.8.9}/.claude/shared/multi-model-review.md

@@ -14,7 +14,7 @@ If `mcp__multi__codereview` is available (check the deferred-tools list; if list
 but not loaded, load it with `ToolSearch` → `select:mcp__multi__codereview`), call
 it with:
 
-- `models`: the strongest available (e.g. `["gemini-3", "codex"]`)
+- `models`: the strongest available (e.g. `["gemini-3", "gpt-5"]` — as of July 2026 these resolve to `gemini-3.1-pro-preview` and `gpt-5.6-sol`; check `mcp__multi__models` if unsure)
 - `relevant_files`: absolute paths of every changed / in-scope file
 - `content`: the review request, citing the consumer's rubric (the shared **Review Criteria**, the **Severity Standard**, etc.)
 - `base_path`: project root

coder_eval-0.8.9/.github/ISSUE_TEMPLATE/adopter.yml

@@ -0,0 +1,84 @@
+name: Adopter — we're using coder_eval
+description: Tell us your team uses coder_eval so we can add you to ADOPTERS.md
+title: "[adopter] <organization>"
+labels: ["adopter"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for using coder_eval! Adopter entries help other teams see the project is
+        real and in production, and they tell us which use cases to keep supporting.
+
+        Everything below is optional except the organization and how you use it. We only
+        publish what you explicitly allow here.
+  - type: input
+    id: organization
+    attributes:
+      label: Organization
+      description: The name you'd like listed.
+      placeholder: Acme Corp
+    validations:
+      required: true
+  - type: input
+    id: website
+    attributes:
+      label: Website
+      placeholder: https://example.com
+    validations:
+      required: false
+  - type: textarea
+    id: usage
+    attributes:
+      label: How do you use coder_eval?
+      description: |
+        A sentence or two. What do you evaluate — Claude Code skills, agent A/B tests,
+        CI quality gates, your own repos — and roughly at what scale?
+      placeholder: |
+        We gate CI on a suite of ~40 YAML tasks that check our internal Claude Code
+        skills still trigger, and A/B Claude Code against Codex each release.
+    validations:
+      required: true
+  - type: dropdown
+    id: listing
+    attributes:
+      label: How may we list you?
+      description: |
+        Pick the most you're comfortable with. **Logo use needs someone who can approve
+        trademark use for your organization** — if that isn't you, choose a name-only or
+        private option and we'll follow up.
+      options:
+        - Name only, in ADOPTERS.md
+        - Name in ADOPTERS.md and on the project website
+        - Name and logo in ADOPTERS.md and on the project website
+        - Please don't list us publicly — this is just for the maintainers
+      default: 0
+    validations:
+      required: true
+  - type: checkboxes
+    id: logo-permission
+    attributes:
+      label: Trademark permission
+      description: Only required if you selected a logo option above.
+      options:
+        - label: >-
+            I am authorized to grant this permission on behalf of my organization, and I
+            grant the coder_eval maintainers a non-exclusive, revocable license to display
+            our name and logo as an adopter in ADOPTERS.md and on the project website. We
+            can withdraw it at any time by commenting on this issue.
+          required: false
+  - type: input
+    id: contact
+    attributes:
+      label: Contact
+      description: GitHub handle or email, so we can reach you before publishing anything.
+    validations:
+      required: false
+  - type: checkboxes
+    id: case-study
+    attributes:
+      label: Anything else?
+      options:
+        - label: We'd be open to a short case study or a talk about our setup.
+          required: false
+        - label: We'd like to hear about breaking changes before they ship.
+          required: false

coder_eval-0.8.9/ADOPTERS.md

@@ -0,0 +1,13 @@
+# Adopters
+
+Teams and organizations using `coder_eval` in their development or CI workflows.
+
+Using it? [Open an adopter issue](https://github.com/UiPath/coder_eval/issues/new?template=adopter.yml)
+and we'll add you here. Name-only listings are welcome — a logo is never required.
+
+Every entry below is published with the organization's permission, and any of them can be
+removed on request by opening an issue.
+
+| Organization | How they use it |
+|---|---|
+| [UiPath](https://www.uipath.com) | Nightly evaluation of internal Claude Code skills and agent A/B experiments; maintains `coder_eval`. |

{coder_eval-0.8.8 → coder_eval-0.8.9}/CHANGELOG.md

@@ -2,6 +2,103 @@
 
 <!-- version list -->
 
+## v0.8.9 (2026-07-23)
+
+### Bug Fixes
+
+- Code review fixes for welch-t-test-exact ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- Render weight:0 criteria as informational on every display surface
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- Weight:0 un-gates criteria (informational criteria)
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- Weight:0 un-gates criteria and renders as informational
+  ([#34](https://github.com/UiPath/coder_eval/pull/34),
+  [`9a34e90`](https://github.com/UiPath/coder_eval/commit/9a34e90bba2854316149cad61613a75dd3bd91e4))
+
+- **early-stop**: Decide skill activation on the tool call, not its result
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+- **early-stop**: Latch skill activation on any engagement, not first
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+- **evalboard**: Match watchlist skeleton header to avoid layout shift
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **reports**: 1/2 — exact Student-t p-values in welch_t_test
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Exact Student-t p-values and a paired comparison section
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Fail loud on t* overflow; surface excluded paired tasks
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: One source of truth for variant series and paired stats
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+- **reports**: Validate confidence and n_resamples in bootstrap_mean_ci
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Chores
+
+- **harness**: Defer two guards from the welch-t-test-exact run
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Documentation
+
+- Add adopter issue template and ADOPTERS.md ([#40](https://github.com/UiPath/coder_eval/pull/40),
+  [`bfbed4e`](https://github.com/UiPath/coder_eval/commit/bfbed4e4a2ca857167fac2cb15462b7159e97684))
+
+- Switch multi-model review from codex to gpt-5 alias
+  ([#36](https://github.com/UiPath/coder_eval/pull/36),
+  [`0a3f2a7`](https://github.com/UiPath/coder_eval/commit/0a3f2a71a0a1e1e56f42a668fff57bac06eb99ec))
+
+### Features
+
+- **evalboard**: Make all pages harness-aware and stream tables
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **evalboard**: Make analytics surfaces harness-aware and stream tables
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **evalboard**: Scope task trends to one harness
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+- **reports**: 2/2 — add a Paired Comparison section to experiment reports
+  ([#38](https://github.com/UiPath/coder_eval/pull/38),
+  [`6df6e9b`](https://github.com/UiPath/coder_eval/commit/6df6e9bb1f3bd009592185ec3473e8f2e67d3816))
+
+### Refactoring
+
+- **evalboard**: Address review nits on harness plumbing
+  ([#45](https://github.com/UiPath/coder_eval/pull/45),
+  [`acc1c86`](https://github.com/UiPath/coder_eval/commit/acc1c86bc97b80383ce1ef402356853447088e4a))
+
+### Testing
+
+- **early-stop**: Cover second-review items (two-AgentStart, golden corpus, parity)
+  ([#43](https://github.com/UiPath/coder_eval/pull/43),
+  [`d34aa97`](https://github.com/UiPath/coder_eval/commit/d34aa97416ae3af09d09cdf78cb6607b4e2e6f7c))
+
+
 ## v0.8.8 (2026-07-22)
 
 ### Bug Fixes

{coder_eval-0.8.8 → coder_eval-0.8.9}/CLAUDE.md

@@ -35,7 +35,7 @@ coder_eval/
 │   ├── criteria.py                # 14 success criterion types + base + union
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
-│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
+│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template)
 │   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
@@ -294,7 +294,7 @@ Tasks are YAML files. See [docs/TASK_DEFINITION_GUIDE.md](docs/TASK_DEFINITION_G
 
 **Runtime (always)**: pydantic, pydantic-settings, pyyaml, typer, rich, python-dotenv, anthropic, claude-agent-sdk, anyio, radon, tqdm, jmespath, jsonschema
 
-**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge and the `rephrase` mutation no longer use the LLM Gateway client — they route through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
+**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge no longer uses the LLM Gateway client — it routes through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
 
 **Dev**: pytest, pytest-asyncio, pytest-mock, pytest-cov, ruff, pyright, pip-audit, bandit, pre-commit, mcp
 

{coder_eval-0.8.8 → coder_eval-0.8.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coder-eval
-Version: 0.8.8
+Version: 0.8.9
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.8.8 → coder_eval-0.8.9}/docs/AB_EXPERIMENTS.md

@@ -227,8 +227,8 @@ variants:
         text: "\n\nThink step by step and validate your work before finishing."
 ```
 
-The full mutation catalog (prefix / suffix / replace / template / rephrase) is
-defined in `coder_eval/models/mutations.py`.
+The full mutation catalog (prefix / suffix / replace / template) is defined in
+`coder_eval/models/mutations.py`.
 
 ## Recipe: Smoke vs. e2e Flavors (Early Stop)
 

{coder_eval-0.8.8 → coder_eval-0.8.9}/docs/TASK_DEFINITION_GUIDE.md

@@ -275,11 +275,17 @@ sandbox:
     - "!node_modules"
     - "*.bak"                         #   bare entry adds an extra pattern
   limits:                             # Optional: resource limits
-    timeout: 300                       # Enforced via subprocess timeout
-    max_memory_mb: 512                 # NOT enforced (reserved for future use)
-    max_disk_mb: 1024                  # NOT enforced (reserved for future use)
+    timeout: 300                       # Enforced via subprocess timeout (both drivers)
+    max_memory_mb: 512                 # driver:docker -> `--memory`; ignored under tempdir
+    max_cpus: 2                        # driver:docker -> `--cpus`; ignored under tempdir
+    max_pids: 512                      # driver:docker -> `--pids-limit`; ignored under tempdir
+    max_disk_mb: 1024                  # NOT enforced (reserved: no portable docker knob)
 ```
 
+Under `driver: tempdir` only `timeout` is enforced — the agent can consume
+arbitrary host memory, CPU, and PIDs. Use `driver: docker` when you need the
+container limits above to actually bind.
+
 ## Template Sources
 
 Tasks can start with preset files instead of an empty sandbox. Multiple sources are applied sequentially (last wins for conflicts).
@@ -383,7 +389,7 @@ All criteria share these fields:
 | Field | Default | Description |
 |-------|---------|-------------|
 | `description` | — | Human-readable description (required) |
-| `weight` | 1.0 | Relative importance for weighted score |
+| `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
 | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
 
@@ -392,7 +398,20 @@ All criteria share these fields:
 - **Fractional** (0.0–1.0): `file_contains`, `file_check`, `json_check`, `command_executed`, `uipath_eval`
 - **Continuous** (0.0–1.0): `reference_comparison`, `llm_judge`, `agent_judge`
 
-**Task success:** ALL criteria must score >= their `pass_threshold`.
+**Task success:** all *gating* criteria must score >= their `pass_threshold`. A
+criterion with `weight: 0` is informational — it is still checked, stored, and
+rendered in reports, but it neither contributes to the score nor fails the task.
+(A `weight: 0` criterion may not set `stop_when` or `suite_thresholds`: arming a
+non-gating criterion for the early-stop or suite gate would let an
+"informational" check flip a run to failure.)
+
+Every surface labels it as such rather than as a failure: the terminal shows `○`
+instead of `✓`/`✗`, the HTML report tags the row "informational — not gated" and
+excludes it from the *n*/*m* passed header, the evalboard renders an `INFO` pill,
+and a below-threshold informational criterion is never sampled as the reason a
+suite row failed. The persisted `CriterionResult.gating` field carries this to
+every consumer, so no reader needs the original criterion to know whether a low
+score mattered.
 
 **Weighted score:** `weighted_score = sum(score * weight) / sum(weight)` — calculated regardless for quality assessment.
 

{coder_eval-0.8.8 → coder_eval-0.8.9}/evalboard/app/_components/harness-badge.tsx

@@ -1,22 +1,35 @@
 import Image from "next/image";
 
-// Vendor logo for a run's harness (RunConfig). Renders the recognizable
+// Canonical harness constants live in the leaf lib/harness module (shared by
+// the server data layer); re-exported here so existing badge importers are
+// unaffected.
+export { KNOWN_HARNESSES, type KnownHarness } from "@/lib/harness";
+
+// Vendor logo + labels for a run's harness (RunConfig). Renders the recognizable
 // vendor mark instead of raw "claude-code"/"codex"/"antigravity" text,
 // mirroring the Slack rollup's vendor emoji. A missing harness defaults to
 // claude-code (the nightly). This is an internal-only column — the caller
 // gates it behind isInternal (see lib/edition.ts).
-const HARNESS_LOGO: Record<string, { src: string; label: string }> = {
+const HARNESS_LOGO: Record<string, { src: string; label: string; short: string }> = {
     "claude-code": {
         src: "/harness/claude-code.png",
         label: "Claude Code · Anthropic",
+        short: "Claude Code",
     },
-    codex: { src: "/harness/codex.png", label: "Codex · OpenAI" },
+    codex: { src: "/harness/codex.png", label: "Codex · OpenAI", short: "Codex" },
     antigravity: {
         src: "/harness/antigravity.png",
         label: "Antigravity · Google Gemini",
+        short: "Antigravity",
     },
 };
 
+// Short human label for a harness id ("Claude Code"), for selectors and prose.
+// Unknown ids fall through to the raw id.
+export function harnessShortLabel(harness: string): string {
+    return HARNESS_LOGO[harness]?.short ?? harness;
+}
+
 export function HarnessBadge({ harness }: { harness?: string | null }) {
     const key = harness ?? "claude-code";
     const logo = HARNESS_LOGO[key];

coder_eval-0.8.9/evalboard/app/_components/harness-selector.tsx

@@ -0,0 +1,49 @@
+"use client";
+
+import { usePathname, useRouter, useSearchParams } from "next/navigation";
+import { HarnessBadge, harnessShortLabel } from "./harness-badge";
+
+// Segmented control for the trends page's harness scope. Sets `?h=<harness>`
+// while preserving the active q/tag params, mirroring WindowSelector. Each
+// segment shows the vendor logo + short label so it reads like the per-run
+// harness badge on the runs tables.
+export function HarnessSelector({
+    current,
+    harnesses,
+}: {
+    current: string;
+    harnesses: readonly string[];
+}) {
+    const router = useRouter();
+    const pathname = usePathname();
+    const searchParams = useSearchParams();
+    const set = (h: string) => {
+        const p = new URLSearchParams(searchParams.toString());
+        p.set("h", h);
+        router.replace(`${pathname}?${p.toString()}`, { scroll: false });
+    };
+    // Always show the active harness, even if it has aged out of the recent
+    // window (so a deep-linked `?h=` still reads as selected rather than absent).
+    const opts = harnesses.includes(current)
+        ? harnesses
+        : [current, ...harnesses];
+    return (
+        <div className="inline-flex border border-gray-200 rounded-md overflow-hidden text-sm">
+            {opts.map((h) => {
+                const active = h === current;
+                return (
+                    <button
+                        key={h}
+                        type="button"
+                        onClick={() => set(h)}
+                        aria-pressed={active}
+                        className={`flex items-center gap-1.5 px-3 py-1 ${active ? "bg-studio-blue text-white" : "bg-white text-gray-700 hover:bg-gray-50"}`}
+                    >
+                        <HarnessBadge harness={h} />
+                        {harnessShortLabel(h)}
+                    </button>
+                );
+            })}
+        </div>
+    );
+}

{coder_eval-0.8.8 → coder_eval-0.8.9}/evalboard/app/_overview/tag-rail.tsx

@@ -1,5 +1,6 @@
 import Link from "next/link";
 import type { TagCount } from "@/lib/overview";
+import { DEFAULT_HARNESS } from "@/lib/harness";
 import type { Window } from "@/lib/reviews-types";
 
 type Variant = "neutral" | "rose" | "indigo";
@@ -35,11 +36,16 @@ function hrefForTag(
     tag: string | null,
     window: Window | null,
     q: string | null,
+    harness: string | null,
 ): string {
     const params = new URLSearchParams();
     if (window) params.set("window", window);
     if (tag) params.set("tag", tag);
     if (q) params.set("q", q);
+    // Preserve the active harness scope across tag clicks (omit the default to
+    // keep URLs clean). Without this, filtering by a tag would silently reset a
+    // codex/antigravity view back to claude-code.
+    if (harness && harness !== DEFAULT_HARNESS) params.set("h", harness);
     const qs = params.toString();
     return qs ? `${basePath}?${qs}` : basePath;
 }
@@ -52,6 +58,7 @@ function TagChip({
     basePath,
     window,
     q,
+    harness,
 }: {
     tag: string;
     count: number;
@@ -60,11 +67,12 @@ function TagChip({
     basePath: string;
     window: Window | null;
     q: string | null;
+    harness: string | null;
 }) {
     const s = STYLES[variant];
     return (
         <Link
-            href={hrefForTag(basePath, active ? null : tag, window, q)}
+            href={hrefForTag(basePath, active ? null : tag, window, q, harness)}
             scroll={false}
             className={`inline-flex items-center gap-1 text-[11px] leading-none px-2 py-1 rounded border transition-colors ${active ? s.chipActive : s.chip}`}
         >
@@ -111,6 +119,7 @@ export function MergedTagRail({
     basePath = "/",
     window = null,
     q = null,
+    harness = null,
     limit = 24,
 }: {
     skills: TagCount[];
@@ -123,6 +132,8 @@ export function MergedTagRail({
     // Null on pages that don't expose a window selector (e.g. /trends).
     window?: Window | null;
     q?: string | null;
+    // Active harness scope to preserve in chip links (null = not harness-scoped).
+    harness?: string | null;
     limit?: number;
 }) {
     const s = pickShown(skills, limit, activeTag);
@@ -146,6 +157,7 @@ export function MergedTagRail({
                     basePath={basePath}
                     window={window}
                     q={q}
+                    harness={harness}
                 />
             ))}
             {r.shown.map((tc) => (
@@ -158,6 +170,7 @@ export function MergedTagRail({
                     basePath={basePath}
                     window={window}
                     q={q}
+                    harness={harness}
                 />
             ))}
             {t.shown.map((tc) => (
@@ -170,6 +183,7 @@ export function MergedTagRail({
                     basePath={basePath}
                     window={window}
                     q={q}
+                    harness={harness}
                 />
             ))}
             {totalRemaining > 0 && (

{coder_eval-0.8.8 → coder_eval-0.8.9}/evalboard/app/page.tsx

@@ -3,8 +3,10 @@ import {
     getAdhocRunListing,
     getOverview,
     getRunListing,
+    listRecentHarnesses,
     type TagCount,
 } from "@/lib/overview";
+import { parseHarnessParam, DEFAULT_HARNESS } from "@/lib/harness";
 import { fmtDuration, fmtRunTime, fmtTimestamp, passClass } from "@/lib/format";
 import { WindowSelector } from "./_components/window-selector";
 import { WINDOWS, type Window } from "@/lib/reviews-types";
@@ -15,7 +17,8 @@ import { ChipLegend, MergedTagRail } from "./_overview/tag-rail";
 import { TableScroll } from "./_components/scroll-table";
 import { CollapsibleRail } from "./_components/collapsible-rail";
 import { isInternal } from "@/lib/edition";
-import { HarnessBadge } from "@/app/_components/harness-badge";
+import { HarnessBadge, harnessShortLabel } from "@/app/_components/harness-badge";
+import { HarnessSelector } from "@/app/_components/harness-selector";
 
 export const dynamic = "force-dynamic";
 
@@ -98,6 +101,7 @@ export default async function Page({
         window?: string;
         tag?: string;
         q?: string;
+        h?: string;
         limit?: string;
         alimit?: string;
     }>;
@@ -106,14 +110,20 @@ export default async function Page({
     const window = parseWindow(params.window);
     const activeTag = parseTag(params.tag);
     const q = parseQ(params.q);
+    const harness = parseHarnessParam(params.h);
     const limit = parseLimit(params.limit);
     const adhocLimit = parseAdhocLimit(params.alimit);
     const isFiltered = activeTag != null || q != null;
 
-    const [overview, listing, adhoc] = await Promise.all([
-        getOverview(window, activeTag, q),
+    // The analytics block (chart + rails) is scoped to one harness so the
+    // success line stops zigzagging across incomparable harnesses. The run
+    // LIST stays all-harness — seeing every recent run is the page's job, and
+    // the Harness column already disambiguates each row.
+    const [overview, listing, adhoc, harnesses] = await Promise.all([
+        getOverview(window, activeTag, q, harness),
         getRunListing(window, activeTag, q, limit),
         getAdhocRunListing(adhocLimit),
+        listRecentHarnesses(),
     ]);
 
     const skills = filterTagsByQuery(overview.skills, q);
@@ -133,10 +143,14 @@ export default async function Page({
     const rawAlimit = Array.isArray(params.alimit)
         ? params.alimit[0]
         : params.alimit;
+    // Omit the default harness from URLs to keep them clean; carry a non-default
+    // scope through every self-link so it isn't reset by pagination/clear.
+    const hParam = harness === DEFAULT_HARNESS ? undefined : harness;
     const base = {
         window,
         tag: activeTag,
         q,
+        h: hParam,
         limit: rawLimit,
         alimit: rawAlimit,
     };
@@ -146,7 +160,7 @@ export default async function Page({
         limit: Math.min(tableTotalLabel, shownCount + DEFAULT_LIMIT),
     });
     const showAllHref = buildHref({ ...base, limit: "all" });
-    const clearAllHref = buildHref({ window });
+    const clearAllHref = buildHref({ window, h: hParam });
 
     // Ad-hoc section disclosure: rows are filtered (by `q`) then capped to
     // adhocLimit; offer "Show all" while more match than are shown, and a
@@ -212,7 +226,7 @@ export default async function Page({
                                     {overview.runs.length === 1 ? "" : "s"}
                                     {" · "}
                                     <Link
-                                        href={buildHref({ window })}
+                                        href={buildHref({ window, h: hParam })}
                                         scroll={false}
                                         className="text-studio-blue hover:underline"
                                     >
@@ -221,14 +235,21 @@ export default async function Page({
                                 </>
                             ) : (
                                 <>
-                                    Success rate per run across the last{" "}
-                                    {window} · {overview.runs.length} run
+                                    Success rate per{" "}
+                                    {harnessShortLabel(harness)} run across the
+                                    last {window} · {overview.runs.length} run
                                     {overview.runs.length === 1 ? "" : "s"}
                                 </>
                             )}
                         </p>
                     </div>
-                    <WindowSelector current={window} />
+                    <div className="flex items-center gap-3">
+                        <HarnessSelector
+                            current={harness}
+                            harnesses={harnesses}
+                        />
+                        <WindowSelector current={window} />
+                    </div>
                 </div>
                 <DailySuccessChart
                     data={overview.runs}
@@ -261,6 +282,7 @@ export default async function Page({
                             activeTag={activeTag}
                             window={window}
                             q={q}
+                            harness={harness}
                             limit={24}
                         />
                     </CollapsibleRail>