lastlight-evals 0.4.0 → 0.6.0

package/README.md

@@ -9,9 +9,10 @@
 `lastlight-evals` takes [**Last Light**](https://lastlight.dev)'s *real*
 production workflows — the actual prompts, skills, and agent loop that ship — and
 runs them end to end against a fully mocked GitHub, for whatever models you throw
-at it. No toy benchmarks, no LLM-as-judge: every run is graded **deterministically**
-(did the agent apply the right labels? did the held-out tests turn green?), then
-ranked side by side on **pass rate, cost, and latency**.
+at it. No toy benchmarks: grading is **deterministic** by default (did the agent
+apply the right labels? did the held-out tests turn green?) — with one scoped
+LLM-judge for the `pr-review` tier's precision/recall/F-beta (F1 by default) — then ranked side by
+side on **pass/score rate, cost, and latency**.
 
 The payoff is one scorecard that tells you, for *your* workflows, exactly what
 each model delivers — and what it costs you per run. Swap a model, re-run, see the
@@ -144,19 +145,51 @@ tests) is taken through the **real** production workflow end to end:
 2. For `code-fix`, the **workspace is seeded** with the fixture repo at its base
    commit plus a local bare `origin`, so `git push` works fully offline.
 3. The **real workflow YAML** (`issue-triage`, `build`, …) is loaded from
-   `lastlight` and run with `sandbox:"none"`, the agent's `github_*` tools
-   pointed at the fake, and approval gates disabled (so it never pauses).
-4. The result is **graded deterministically** — no LLM judge:
+   `lastlight` and run with `sandbox:"none"` by default (in-process), the agent's
+   `github_*` tools pointed at the fake, and approval gates disabled (so it never
+   pauses). Pass `--sandbox gondolin` to isolate the agent's bash/file tools in a
+   QEMU micro-VM (see [Isolation](#isolation---sandbox) below).
+4. The result is **graded deterministically** (with one scoped judge for
+   pr-review):
    - **behavioral** — the recorded GitHub calls (labels, comments, PRs) vs the
      instance's `expect_github` / `triage_gold`.
    - **execution** (code-fix) — the held-out tests are applied and run; the case
      is *resolved* only if every `FAIL_TO_PASS` passes and every `PASS_TO_PASS`
      stays green (SWE-bench's criterion).
+   - **review** (pr-review) — the posted review is matched to a human-verified
+     gold set by an **LLM judge** → precision / recall / **F-beta** (F1 by default,
+     Martian's leaderboard metric; `EVAL_F_BETA` reweights). The one,
+     deliberately-scoped exception; triage/code-fix stay judge-free.
 5. Token usage, cost, and latency are collected per run.
 
 Run multiple models and you get a side-by-side **scorecard** (HTML + JSON)
 ranking them on pass rate, cost, and latency.
 
+### Isolation (`--sandbox`)
+
+By default the agent runs **in-process** (`sandbox:"none"`) — fast and CI-friendly,
+but with **no filesystem restriction**: the agent process can read any absolute
+path, including this repo's held-out gold data (`datasets/<tier>/tests/`,
+`instances.json`, `.eval-cache/`). A capable model that explores the disk could
+find and spoil the answer key.
+
+`--sandbox gondolin` (or `EVAL_SANDBOX=gondolin`) closes that gap: the agent's
+bash/file tools execute inside a **QEMU micro-VM** that only sees its own
+workspace, so host gold paths are invisible. The agent runtime and `github_*`
+tools stay in-process, so the fake-GitHub mock still works unchanged — this is
+why gondolin, and not `docker`, is the supported isolation backend (`docker`/`smol`
+run the *whole* agent in the container/VM, where the in-process fake GitHub isn't
+reachable).
+
+Gondolin needs QEMU with hardware acceleration and runs **natively** (macOS via
+Apple's Hypervisor.framework, Linux via KVM) — install it with `brew install qemu`
+(macOS) or your distro's `qemu-system` package. It does **not** work inside a
+container on macOS (no `/dev/kvm`, and the failure is a silent hang), so the
+harness runs a fail-fast preflight and aborts with guidance rather than wedging.
+Expect a one-time ~13s VM cold start plus per-tool-call overhead, so keep
+`--sandbox gondolin` for trustworthy/anti-spoil runs and leave the default `none`
+for quick iteration.
+
 ## Run it
 
 ```bash
@@ -180,6 +213,10 @@ lastlight-evals run triage --model glm,deepseek   # a comma-list also works
 # repeat each case N times; verdicts WORST-case, cost/tokens/latency MEAN
 lastlight-evals run triage --runs 3
 
+# isolate the agent's tools in a QEMU micro-VM so it can't read host gold data
+# (anti-spoil; needs QEMU natively — see "Isolation" above). Default is none.
+lastlight-evals run pr-review --sandbox gondolin
+
 # run against an overlay repo's OWN workflows + datasets (see below)
 lastlight-evals run --overlay ~/work/lastlight-instance
 
@@ -303,7 +340,10 @@ A **tier** is a directory containing `instances.json` (+ an optional `tier.json`
 declaring its `defaultWorkflow`). Tiers are discovered from three roots, merged
 by name with **overlay > user (`--datasets`) > built-in** precedence:
 
-- **built-in** (shipped here): `triage` → `issue-triage`, `code-fix` → `build`.
+- **built-in** (shipped here): `triage` → `issue-triage`, `code-fix` → `build`,
+  `pr-review` → `pr-review` (ships empty — populate with
+  `scripts/import-martian.ts`; see [PR-review tier](#pr-review-tier-code-review-bench)
+  below and `datasets/pr-review/README.md`).
 - **user**: `--datasets <dir>` / `LASTLIGHT_EVALS_DATASETS`.
 - **overlay**: `<overlay>/evals/datasets/*`.
 
@@ -366,6 +406,73 @@ A new tier just needs a directory with an `instances.json` and a `tier.json`
 (`{ "name", "defaultWorkflow", "description" }`); per-instance `workflow` wins
 when present.
 
+## PR-review tier (Code Review Bench)
+
+The **`pr-review`** tier measures review *quality* against
+[Martian's Code Review Bench](https://github.com/withmartian/code-review-benchmark):
+the review the real `pr-review` workflow posts is scored against a human-verified
+**gold set** of the issues a reviewer should have caught. It's the **one** tier
+graded by an LLM judge — matching free-text findings to semantic gold comments
+can't be done deterministically — so triage and code-fix stay judge-free.
+
+**Cases** come from Martian's *offline* set — 50 real merged PRs across Sentry,
+Grafana, Cal.com, Discourse, and Keycloak, each carrying inlined `golden_comments`.
+They ship **empty** (`datasets/pr-review/instances.json` is `[]`) because they're
+large real-repo PRs — *generated*, not vendored:
+
+```bash
+npx tsx scripts/import-martian.ts            # resolve all 50 via gh (pins base/head SHAs)
+npx tsx scripts/import-martian.ts --limit 3  # a quick subset first
+```
+
+**Seeding** clones the real repo into the gitignored `./.eval-cache/` and checks
+out the PR **head** (mirroring production's pre-clone contract), so the skill's
+`git diff origin/<base>...HEAD` works fully offline — no fixture is vendored.
+
+**Grading** (`gradeReview`, `src/grade.ts`) is a two-step LLM judge:
+
+1. **Extract** the review's distinct, concrete findings (drop praise/summaries).
+2. **Match** each finding to a gold comment ("same underlying issue?").
+
+From the matches: **precision** = matched ÷ posted, **recall** = matched ÷ gold,
+combined as **F-beta**. The headline is **F1** (β=1 — precision and recall weighted
+equally, Martian's leaderboard metric). Pass **`--f-beta 0.5`** (or `EVAL_F_BETA=0.5`)
+to weight precision 2× (F0.5), mirroring Martian's adjustable F-beta; the dashboard
+relabels itself `F{β}` to match.
+
+> **Gold-set caveat.** Martian's own methodology documents the gold set as
+> *incomplete* — it caps at human performance, so a real issue the annotators
+> missed is scored as a false positive. That understates precision, which is why
+> the default is F1, not the precision-weighted F0.5. Treat the score as a
+> **relative** signal and inspect each match with the dashboard's **judge** button.
+
+**The judge model is independent** of the models under test — a strong default per
+your provider key (`EVAL_JUDGE_MODEL` overrides). A judge failure marks the case
+*errored* (ungraded), never a silent zero. Alongside the judge score, a cheap
+deterministic `review_submitted` proxy checks a review was actually posted.
+
+**Diff-blind by default.** The judge sees only the posted review (body + inline
+comments) matched against the gold set — *not* the PR diff — mirroring Martian's
+offline judge. This can penalize terse, location-anchored comments (`off-by-one
+here` on a line the judge can't see). Pass **`--judge-with-diff`** to feed the PR
+diff into the judge for higher-fidelity matching (the judge is instructed never to
+invent findings from the diff); this trades away leaderboard parity, and the
+dashboard marks such grades **`diff-aware`**.
+
+**Run it** (heavy — clones real repos + calls the judge):
+
+```bash
+lastlight-evals run pr-review --model <model>            # full tier
+lastlight-evals run pr-review --model <model> --limit 3  # first 3 cases (controlled)
+lastlight-evals run pr-review --model <model> --f-beta 0.5        # weight precision 2×
+lastlight-evals run pr-review --model <model> --judge-with-diff   # give the judge the diff
+```
+
+In the dashboard, each row's **judge** button opens the judge's working — the
+findings it extracted, the gold set, the finding↔gold pairing (matched / false
+positive / missed), and its raw replies — so the F1 score is inspectable, not a
+black box.
+
 ## Models (`models.json`)
 
 - `default` — the single model `run` uses.
@@ -378,5 +485,7 @@ when present.
 
 - **`lastlight-evals extract <owner>/<repo>#<n>`** — generate eval cases from
   GitHub historical issues/PRs (issue → fixture, merged PR → held-out tests).
-- Docker-backed runs; real SWE-bench Lite ingestion; per-fixture test runners.
+- Docker-backed sandboxed runs (needs the fake GitHub reachable from inside the
+  container — `--sandbox gondolin` already gives native isolation today); real
+  SWE-bench Lite ingestion; per-fixture test runners.
 - LLM-as-judge stays out by design — grading is deterministic.