lastlight 0.7.8 → 0.9.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lastlight",
-  "version": "0.7.8",
+  "version": "0.9.0",
   "description": "GitHub repository maintenance agent — Agent SDK harness",
   "type": "module",
   "main": "dist/index.js",

package/plugins/lastlight/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin.json",
   "name": "lastlight",
   "displayName": "Last Light",
-  "version": "0.7.8",
+  "version": "0.9.0",
   "description": "Install, configure and operate Last Light (GitHub maintenance agent) — its server, CLI client, deployment overlay, and the Last Light Evals harness.",
   "author": {
     "name": "Clifton Cunningham"

package/plugins/lastlight/README.md

@@ -13,10 +13,12 @@ Light's *internal* sandbox skills under the repo's top-level `skills/` dir
 
 | Skill | Use it when you want to… |
 |-------|--------------------------|
+| `lastlight-guide` | **Not sure which of the below you need** — orientation & router across all the Last Light skills. Start here (or `/lastlight-guide`). |
 | `lastlight-server` | Install & configure a Last Light **server** (the agent + docker stack) on a host. |
 | `lastlight-client` | Point the `lastlight` **CLI client** at an existing server and log in. |
 | `lastlight-overlay` | Create a deployment **overlay** instance and fork/customize workflows, prompts, skills, or the agent persona. |
 | `lastlight-evals` | Scaffold and run a **Last Light Evals** workspace (datasets, models, model comparisons). |
+| `lastlight-evals-loop` | **Drive an eval toward a target score** with a disciplined, anti-gaming improvement loop — diagnose on a train split, validate on a blind held-out split, propose one generic fix at a time. |
 
 ## Install
 
@@ -41,4 +43,6 @@ claude plugin install lastlight@lastlight-skills
 auto-discovers them on the next session.
 
 After installing, start a new Claude Code session and say e.g. *"set up a Last
-Light server"* or *"scaffold a Last Light evals workspace"*.
+Light server"* or *"scaffold a Last Light evals workspace"* — or, if you're not
+sure where to start, *"help me with Last Light"* (or `/lastlight-guide`) and the
+guide skill will route you.

package/plugins/lastlight/skills/lastlight-evals/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: lastlight-evals
-description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases — including "create an eval dataset/case from this PR/issue <github url>" (triage / code-fix instances). GitHub is mocked when running, so no real GitHub token is needed — only a model provider API key (the one exception is `add-case`, which reads real PRs/issues via `gh`).
-version: 1.2.0
+description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases — including "create an eval dataset/case from this PR/issue <github url>" and "create a PR-review eval dataset from these gold PRs" (triage / code-fix / pr-review instances, via `add-case --pr <url> --review|--code-fix` / `--issue`). GitHub is mocked when running, so no real GitHub token is needed — only a model provider API key (the one exception is `add-case`, which reads real PRs/issues via `gh`).
+version: 1.3.0
 tags: [lastlight, evals, benchmark, models, swe-bench]
 ---
 
@@ -14,6 +14,25 @@ thin CLI on top of the `lastlight` core package (via the `lastlight/evals`
 barrel), so it exercises the same workflows/skills production does. SWE-bench
 compatible. **Node 24+.**
 
+## Start here — what do you want to do?
+
+If the user's evals goal is clear, jump to the section. If it's vague ("help me
+with the evals", "what can this do?"), **ask** (`AskUserQuestion`) which of these
+they want, then go:
+
+| Goal | Section |
+|---|---|
+| Set up an evals workspace (first time) | **§2 Scaffold** (+ §1 prereqs, §3 providers) |
+| Run evals / compare models | **§4 Run** |
+| Look at past runs (no models run) | **§4** — `lastlight-evals serve` |
+| Author one case from a GitHub PR or issue | **§6** |
+| **Build a PR-review dataset from my own gold PRs** | **§6 → "Build a PR-review dataset"** |
+| Add cases by hand / understand the schema | **§5** + `references/instance-schema.md` |
+| **Iteratively improve the score toward a target** | the **`lastlight-evals-loop`** skill |
+
+New to the whole plugin (server / overlay / client, not just evals)? That's the
+**`lastlight-guide`** skill — this one is evals-only.
+
 ## 1. Check prerequisites
 
 ```bash
@@ -102,9 +121,10 @@ lastlight-evals run triage --model openai/gpt-5.5,anthropic/claude-opus-4-8
 lastlight-evals run --compare                   # cross-vendor set (only models whose envKey is present)
 lastlight-evals run triage --runs 3             # repeat each case 3× (worst-case verdict, mean metrics)
 lastlight-evals run pr-review --limit 3         # only the first 3 cases of the tier (controlled/cheap run)
-lastlight-evals run triage --instance <id>      # only this exact instance_id (or set EVAL_INSTANCE)
+lastlight-evals run triage --instance <id[,id2]> # only these exact instance_id(s), comma-separated (or set EVAL_INSTANCE)
 lastlight-evals run pr-review --f-beta 0.5      # pr-review F-beta β (default 1=F1; 0.5=precision 2×). Or EVAL_F_BETA
 lastlight-evals run pr-review --judge-with-diff # feed the PR diff to the judge (higher fidelity; off by default)
+lastlight-evals run pr-review --no-inject-context # DON'T inject synthetic repo-context into the checkout (clean A/B control)
 lastlight-evals run pr-review --sandbox gondolin # isolate the agent's tools in a QEMU micro-VM (anti-spoil). Or EVAL_SANDBOX
 lastlight-evals run triage --no-open            # don't open the report
 # Plain layout: add --overlay .   (e.g. lastlight-evals run triage --overlay .)
@@ -144,26 +164,49 @@ Quick shape (paths are relative to the workspace's `evals/` dir):
   `evals/datasets/code-fix/tests/<instance_id>/` (held-out tests applied at grade time).
 - **PR-review case:** a `SweBenchInstance` with a `pr` fixture (base/head refs +
   commits, checked out at the PR head) and a `review_gold` set (severity +
-  description). The `pr-review` tier ships empty — populate the full Martian
+  description). Populate the full Martian
   [Code Review Bench](https://codereview.withmartian.com/) 50 with
-  `npx tsx scripts/import-martian.ts`. Grading needs a judge model
+  `npx tsx scripts/import-martian.ts`, **or author your *own* gold-PR dataset**
+  with `add-case --pr <url> --review` (see §6 and
+  **`references/authoring-pr-review.md`**). Grading needs a judge model
   (`EVAL_JUDGE_MODEL`, else a strong default per provider key).
+  - **Repo-context injection (pr-review).** The harness can drop a synthetic
+    `AGENTS.md`/`CLAUDE.md` into the checked-out repo so the reviewing agent reads
+    it — a **generic** block from `<overlay>/repo-context/AGENTS.md` (every repo)
+    plus a **per-repo** block from `datasets/pr-review/context/<instance_id>/AGENTS.md`.
+    Presence-based (just create the file), on by default; `--no-inject-context`
+    (or `EVAL_INJECT_CONTEXT=0`) forces a clean control for an A/B. It appends to
+    a real `AGENTS.md`/`CLAUDE.md` if the repo ships one (never shadowing it). This
+    is how you prove *"adding this to your repo improves review quality"* — and the
+    lever the **`lastlight-evals-loop`** skill drives.
 - **Custom tier:** a new `evals/datasets/<tier>/` with `tier.json` +
   `instances.json` (+ `repos/` & `tests/` for code-fix-style tiers). Discovery is
   automatic — no code change.
 
 ## 6. Author a case from a real GitHub PR or issue
 
-When the user says **"create an eval dataset/case from this PR/issue <url>"**, use
-the `add-case` subcommand — it does the mechanical extraction; you refine the
-fuzzy parts.
+When the user says **"create an eval dataset/case from this PR/issue <url>"** or
+**"create a PR-review eval dataset from these gold PRs"**, use the `add-case`
+subcommand — it does the mechanical extraction; you refine the fuzzy parts.
+
+A PR seeds two different evals, so `--pr` requires a kind — `--review` (the PR's
+human review is the gold set) or `--code-fix` (hide the fix, the agent must
+reproduce it):
 
 ```bash
-lastlight-evals add-case --pr <github-pr-url> --dry-run        # propose a code-fix case; don't write
-lastlight-evals add-case --pr <github-pr-url>                  # write into ./datasets (or --datasets/--overlay)
-lastlight-evals add-case --issue <github-issue-url> --dry-run  # propose a triage case
+lastlight-evals add-case --pr <github-pr-url> --review --dry-run    # propose a pr-review case; don't write
+lastlight-evals add-case --pr <github-pr-url> --review              # write into ./datasets/pr-review (or --overlay)
+lastlight-evals add-case --pr <github-pr-url> --code-fix --dry-run  # propose a code-fix case
+lastlight-evals add-case --issue <github-issue-url> --dry-run       # propose a triage case
 ```
 
+- **From a PR (pr-review):** derives the `pr` fixture (base/head refs + commits,
+  from the PR's `baseRefOid`/`headRefOid`) and a **candidate `review_gold`** from
+  the PR's *human* review — inline comments + substantive review bodies, with
+  bot/nit/reply/outdated noise filtered out. Prints an evidence block; **you**
+  assign real severities and prune non-actionable comments. Byte-compatible with
+  `import-martian.ts`, so authored cases sit alongside imported ones. Leak-safe:
+  the review lands only in `review_gold`, never in the seeded PR the agent sees.
 - **From a PR (code-fix):** derives `repo`, `base_commit` (merge-base of base &
   head) + `head_commit`, the PR's **test** diff as the held-out `test_patch`, and
   auto-detects `FAIL_TO_PASS` / `PASS_TO_PASS` by running the tests at base (red)
@@ -176,9 +219,35 @@ lastlight-evals add-case --issue <github-issue-url> --dry-run  # propose a triag
   **reviewer comments** — the raw triage signal. You then assign
   `triage_gold` (category/state) from those labels per the deployment's taxonomy.
 
+### Build a PR-review dataset from gold PRs (interactive)
+
+When the user says **"I want to create a PR-review eval dataset"** (from *their*
+PRs, to track review quality as they evolve models/prompts):
+
+1. **Get the PRs.** If they didn't paste URLs, **ask** for them — "give me the
+   gold PR URL(s) — one or many, whose human review is the standard you want a
+   model to reproduce." Also confirm *where* to write: the current workspace's
+   `./datasets` (or `--overlay <dir>` for a deployment's own dataset).
+2. **One PR at a time.** For each URL run
+   `add-case --pr <url> --review --dry-run`, read the evidence block, then
+   **curate** the candidate `review_gold` — set real severities and drop
+   comments that aren't concrete findings — and write it
+   (`add-case --pr <url> --review [--datasets/--overlay …]`). Re-running the same
+   PR **replaces by id**, so curation is idempotent.
+   - For **many** PRs, spawn a **curation sub-agent per PR** (each handed one URL
+     + the target dir, returning the curated instance) so the judgement stays
+     isolated and they run in parallel.
+3. **Smoke-run** the new tier with the cheapest model to confirm the seed + judge
+   plumbing (needs a judge-model key): `lastlight-evals run pr-review --limit 1 --model haiku`.
+
+See **`references/authoring-pr-review.md`** for the full extract→curate→write
+flow, the `--severity` / `--include-bots` knobs, and the anti-spoil guarantee.
+
 The full flow, what you refine for each, the `test_cmd` / `setup_cmd` options for
 non-`node --test` runners, and the trust/offline caveats are in
-**`references/authoring-from-pr.md`** — read it before authoring.
+**`references/authoring-from-pr.md`** (PR code-fix + issue triage) and
+**`references/authoring-pr-review.md`** (PR review) — read the relevant one
+before authoring.
 
 ## Done when
 

package/plugins/lastlight/skills/lastlight-evals/references/authoring-from-pr.md

@@ -1,11 +1,17 @@
-# Author an eval case from a real GitHub PR or issue
+# Author a code-fix or triage eval case from a real GitHub PR or issue
 
-`lastlight-evals add-case` turns a real GitHub **PR** into a code-fix (build) case
-or an **issue** into a triage case. It does the mechanical, reproducible
-extraction with `gh` + `git`; **you** refine the judgement parts. The result is a
-**git-source** case — no fixture repo is vendored; at run time the harness clones
-the repo into the gitignored `./.eval-cache/` and checks out `base_commit` (see
-the `instance-schema.md` "Git-source" flavor).
+`lastlight-evals add-case --pr <url> --code-fix` turns a real GitHub **PR** into a
+code-fix (build) case; `--issue <url>` turns an **issue** into a triage case. It
+does the mechanical, reproducible extraction with `gh` + `git`; **you** refine the
+judgement parts. The result is a **git-source** case — no fixture repo is
+vendored; at run time the harness clones the repo into the gitignored
+`./.eval-cache/` and checks out `base_commit` (see the `instance-schema.md`
+"Git-source" flavor).
+
+> A PR seeds two different evals. Use `--code-fix` (this doc) to hide the fix and
+> have the agent reproduce it; use `--review` (see **`authoring-pr-review.md`**)
+> to grade a review of the PR against its human review. `--pr` **requires** one of
+> the two — it no longer defaults to code-fix.
 
 ## Two grading modes (default: suite)
 
@@ -34,7 +40,7 @@ the `instance-schema.md` "Git-source" flavor).
 ## Command
 
 ```bash
-lastlight-evals add-case --pr <github-pr-url> [options]
+lastlight-evals add-case --pr <github-pr-url> --code-fix [options]
 lastlight-evals add-case --issue <github-issue-url> [options]
 ```
 
@@ -42,9 +48,10 @@ Options:
 
 | flag | meaning |
 |---|---|
-| `--pr <url>` | a GitHub PR url → a **code-fix** (build) case |
+| `--pr <url>` | a GitHub PR url (pair with `--code-fix` here, or `--review` for a pr-review case) |
+| `--code-fix` | with `--pr` → a **code-fix** (build) case |
 | `--issue <url>` | a GitHub issue url → a **triage** case |
-| `--tier <name>` | target tier dir (default `code-fix` for `--pr`, `triage` for `--issue`) |
+| `--tier <name>` | target tier dir (default `code-fix` for `--pr --code-fix`, `triage` for `--issue`) |
 | `--id <slug>` | `instance_id` (default derived from repo + number) |
 | `--datasets <dir>` | datasets root to write into (a `<tier>/` subdir). Default `./datasets`, else `./evals/datasets` |
 | `--overlay <dir>` | write into `<dir>/evals/datasets` instead |
@@ -57,8 +64,8 @@ Options:
 
 ## The recommended flow (CLI extracts → you refine)
 
-1. **Dry-run first.** `add-case --pr <url> --dry-run` prints the proposed instance.
-   The CLI derives:
+1. **Dry-run first.** `add-case --pr <url> --code-fix --dry-run` prints the proposed
+   instance. The CLI derives:
    - `repo`, `base_commit` (the **merge-base** of the base branch and the PR head —
      the true fork point, not the base-branch tip), and `head_commit`;
    - gold `patch` — the diff of the non-test files (reference only, never graded);

package/plugins/lastlight/skills/lastlight-evals/references/authoring-pr-review.md

@@ -0,0 +1,110 @@
+# Author a pr-review eval case from a real GitHub PR
+
+`lastlight-evals add-case --pr <url> --review` turns a real GitHub **PR** into a
+`pr-review` case: the PR's **human review is the gold set** the model's review is
+scored against (precision / recall / F-beta, via an LLM judge). It does the
+mechanical, reproducible extraction with `gh`; **you** curate the gold. The output
+is **byte-compatible** with `scripts/import-martian.ts`, so cases you author sit
+alongside the Martian Code Review Bench cases in the same `pr-review/instances.json`.
+
+This is the path for *"I want to create a PR-review eval dataset from these gold
+PRs"* — PRs whose human reviews are the standard you want a model (or an evolving
+prompt/skill) to reproduce.
+
+## What it extracts
+
+- **The `pr` fixture** — `base_ref`/`head_ref` + `base_commit`/`head_commit` (from
+  the PR's `baseRefOid`/`headRefOid`), `title`, `body` (truncated), `user`. At run
+  time the harness clones the repo and checks out the PR **head**; the review
+  workflow diffs `base..head`. No repo clone happens at author time.
+- **A candidate `review_gold`** from the PR's *human* review: inline review
+  comments (`path` + `line` + body) and substantive top-level review bodies. The
+  CLI filters the obvious noise — reply threads, `LGTM`/approve-only comments, and
+  bot comments (unless `--include-bots`) — and defaults every entry's `severity`
+  to `medium` (GitHub comments carry none). Outdated comments (whose hunk moved)
+  are kept via `original_line` and flagged.
+
+> **Anti-spoil.** The human review is written **only** into `review_gold` (the
+> held-out judge gold). The CLI never puts it in `pr.reviews` / `pr.review_comments`
+> — those seed the mocked GitHub the agent can read, which would hand it the
+> answer. This mirrors `import-martian.ts`, and the run-time seeding
+> (`seedWorkspacePrReview`) only checks out git, never the prior discussion.
+
+## Prerequisites
+
+- `gh` on PATH and authenticated (`gh auth login`) — used to read PR metadata + review comments.
+- Node 24+ (same as the rest of the harness). A **judge model** at run time
+  (`EVAL_JUDGE_MODEL`, else a strong default per provider key).
+
+## Command
+
+```bash
+lastlight-evals add-case --pr <github-pr-url> --review [options]
+```
+
+Options:
+
+| flag | meaning |
+|---|---|
+| `--pr <url>` | a GitHub PR url (pair with `--review` here, or `--code-fix` for a build case) |
+| `--review` | with `--pr` → a **pr-review** case (gold = the PR's human review) |
+| `--severity <level>` | default severity for candidate gold comments (`low`\|`medium`\|`high`\|`critical`; default `medium`) |
+| `--include-bots` | keep review comments from bots (default: skipped) |
+| `--tier <name>` | target tier dir (default `pr-review`) |
+| `--id <slug>` | `instance_id` (default `prreview__<repo>-<n>`) |
+| `--datasets <dir>` | datasets root to write into (a `<tier>/` subdir). Default `./datasets`, else `./evals/datasets` |
+| `--overlay <dir>` | write into `<dir>/evals/datasets` instead |
+| `--dry-run` | print the proposed instance JSON + evidence block; don't write |
+
+## The recommended flow (CLI extracts → you refine)
+
+1. **Dry-run first.** `add-case --pr <url> --review --dry-run` prints the proposed
+   instance and an **evidence block** listing every candidate gold comment
+   (`author · file:line [severity] first-line`, tagged `(outdated)` / `(bot)`).
+2. **Read the evidence and curate `review_gold`** — this is the whole point of a
+   *gold* dataset:
+   - **Severity.** Every candidate defaults to `medium`. Set each to its real
+     impact (`critical`/`high`/`medium`/`low`) — the judge is severity-aware.
+   - **Prune noise.** Drop comments that aren't concrete findings — questions,
+     style nits, "why not X?", praise, anything `(outdated)` that no longer applies
+     to the head diff.
+   - **Add what the humans missed** (optional). If you know of a real issue the
+     human reviewer didn't post, add a `review_gold` entry by hand (`severity` +
+     `description`; `file`/`line` optional — the judge matches on substance).
+   - **Empty gold?** If the PR had no usable review comments, the CLI warns and
+     leaves `review_gold: []` — author it by hand from the PR diff, or the case
+     can't be judged.
+3. **Write it.** Re-run without `--dry-run` (add `--datasets <dir>` / `--overlay
+   <dir>` to target a workspace). The instance is appended to
+   `<root>/pr-review/instances.json` (creating `tier.json` if the tier is new);
+   re-running with the same PR **replaces by `instance_id`**, so curation is
+   idempotent.
+4. **Verify end-to-end** with the cheapest model — confirms the seed + judge
+   plumbing (needs a judge-model key):
+
+   ```bash
+   EVAL_INSTANCE=<instance_id> lastlight-evals run pr-review --model haiku
+   ```
+
+   Inspect the judge trace in the dashboard (the **judge** button) to see the
+   finding↔gold pairing behind the precision/recall/F score.
+
+## Authoring from a list of gold PRs
+
+Run `add-case --pr <url> --review` **once per URL** — each appends into the same
+`pr-review/instances.json`. Because `review_gold` curation is per-PR judgement,
+do one curation pass per PR rather than a bulk import: dry-run → refine → write.
+Spawning a **curation sub-agent per PR** (each handed one URL, returning the
+curated instance) keeps the judgement isolated and parallelizes cleanly, while the
+CLI's replace-by-id write makes accumulation safe.
+
+## Tuning knobs
+
+- **`--severity <level>`** sets the default for *all* candidates in one PR — handy
+  when a PR's review is uniformly high- or low-stakes; you still refine per entry.
+- **`--include-bots`** keeps bot review comments (CodeRabbit, Greptile, etc.).
+  Off by default because bot comments are noisy and rarely the human gold; include
+  them only if the bot's review *is* the standard you're grading against.
+
+See `instance-schema.md` for the full `pr` / `review_gold` field reference and the
+pr-review grading model.

package/plugins/lastlight/skills/lastlight-evals/references/instance-schema.md

@@ -44,12 +44,23 @@ Datasets are discovered from (overlay > user > built-in): `<overlay>/evals/datas
     "state": "open"
   },
   "triage_gold": { "category": "bug", "state": "ready-for-agent" },  // triage grading
+  "pr": {                                  // pr-review: the PR fixture (base/head refs + SHAs)
+    "number": 37429, "title": "...", "body": "...",
+    "base_ref": "main", "head_ref": "feature-x",
+    "base_commit": "f3c8e8f...", "head_commit": "02f48f7...",
+    "user": "author"                       // do NOT put the human review here — see below
+  },
+  "review_gold": [                          // pr-review: the held-out judge gold
+    { "file": "src/x.ts", "line": 42, "severity": "high", "description": "..." },
+    { "severity": "medium", "description": "..." }   // file/line optional (judge matches on substance)
+  ],
   "expect_github": {                       // behavioral assertions on recorded GitHub calls
     "labels_added": ["bug"],
     "labels_absent": ["wontfix"],
     "issue_closed": false,
     "comment_matches": "(?i)thanks",
-    "pr_opened": { "base": "main", "head_is_branch": true, "title_matches": "(?i)fix" }
+    "pr_opened": { "base": "main", "head_is_branch": true, "title_matches": "(?i)fix" },
+    "review_submitted": {}                 // pr-review: proxy check that a review was posted
   }
 }
 ```
@@ -83,6 +94,26 @@ hand-build these — use `lastlight-evals add-case --pr <url>` (see
 **`authoring-from-pr.md`**), which fills `base_commit`, `head_commit`,
 `test_patch`, and the verdicts for you.
 
+## Add a pr-review case
+
+A pr-review instance has a **`pr`** fixture + a **`review_gold`** set and
+`workflow: "pr-review"`. The `pr` fixture drives both the mocked PR endpoints and
+the workspace checkout: at run time the harness checks out the PR **head**, and
+the review workflow diffs `base..head`. `review_gold` is the held-out gold the
+posted review is scored against by an LLM judge (each entry: `severity` ∈
+`low`|`medium`|`high`|`critical` + `description`; `file`/`line` optional — the
+judge matches on substance). Set `expect_github.review_submitted: {}` (a cheap
+proxy that a review was posted).
+
+> **Never** put the human/gold review in `pr.reviews` / `pr.review_comments` —
+> those seed the mocked GitHub the agent can read, spoiling the case. The gold
+> lives only in `review_gold`.
+
+Don't hand-build these — use `lastlight-evals add-case --pr <url> --review` (see
+**`authoring-pr-review.md`**), which pins the `pr` fixture and seeds a candidate
+`review_gold` from the PR's human review for you to curate. Or bulk-import the
+Martian Code Review Bench with `npx tsx scripts/import-martian.ts`.
+
 ## Add a custom tier
 
 Create `datasets/<tier-name>/` with `tier.json` (`name`, `defaultWorkflow`,
@@ -95,6 +126,11 @@ and `tests/<id>/`. Discovery auto-finds it — no code change. Run it with
 - **Behavioral:** did the workflow take the expected GitHub actions
   (`expect_github`)?
 - **Triage:** did the decision match `triage_gold`?
+- **Review (pr-review):** an LLM judge matches the posted review's findings against
+  `review_gold` → **precision / recall / F-beta** (β via `EVAL_F_BETA` or
+  `--f-beta`; default F1). Needs a judge model (`EVAL_JUDGE_MODEL`, else a strong
+  default per provider key); independent of the model under test. The judge trace
+  is inspectable in the dashboard.
 - **Execution (code-fix)** — two modes:
   - **Suite (default).** Nothing held out: run the repo's own `test_cmd` against the
     agent's final tree, **resolved iff it exits 0**. Grades "did the agent leave the

package/plugins/lastlight/skills/lastlight-evals-loop/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: lastlight-evals-loop
+description: Drive a Last Light EVAL toward a target score with a disciplined, anti-gaming improvement loop — run → diagnose → propose ONE generic fix → re-measure → keep or revert → repeat. Use when the user wants to "improve / raise the pr-review F1", "make the reviewer better against the eval", "close the loop on evals", "iterate on prompts/skills to pass more cases", or "tune the workflow to hit a score target". The loop diagnoses on a TRAIN split and validates on a BLIND held-out split so fixes must generalize, not overfit; it prefers generic overlay prompt/skill edits, and stops for human sign-off before editing any gold answer. For a one-off run/compare use lastlight-evals; to hand-fork a workflow/prompt use lastlight-overlay. Needs an already-scaffolded evals workspace (lastlight-evals) with a pr-review dataset.
+version: 1.0.0
+tags: [lastlight, evals, loop, improvement, overfitting, pr-review]
+---
+
+# Improve an eval, honestly
+
+A loop that raises an eval score **without gaming it**. The trap: the loop can see
+both the gold answer and the agent's answer for every case, so it could overfit —
+hardcode a repo's finding into a prompt, or edit a gold answer to force a pass.
+That lifts the number and ruins the reviewer. Every guardrail here exists to force
+**generic, cross-cutting** changes that would help on repos the loop has never seen.
+
+Scope: the **pr-review** tier (its judge trace gives you the agent-vs-gold detail
+diagnosis needs). The pattern extends to triage/code-fix, but start here.
+
+## Prerequisites
+
+- A scaffolded evals workspace with a **pr-review** dataset and a working provider
+  key — set up via **`lastlight-evals`** first if you don't have one. Confirm a
+  bare `lastlight-evals run pr-review --limit 1 --no-open` grades a case.
+- An **`instance/` overlay** (the deployment overlay) to receive generic edits —
+  see **`lastlight-overlay`**. The loop edits copies here, never core.
+- The `diff-runs.ts` helper ships with the evals package
+  (`scripts/diff-runs.ts`) — it compares two runs and calls keep/revert.
+
+## The one rule
+
+**One change per iteration.** Propose a single hypothesis, measure its effect in
+isolation, keep or revert, then move on. No batching — batching hides which edit
+moved the number and invites scattershot overfitting.
+
+## Start here
+
+| You have… | Go to |
+|---|---|
+| Never run this loop on this dataset | **§1 Set up the split** then §2 |
+| A split + baseline already recorded | **§3 Diagnose** |
+| A proposed change ready to test | **§5 Audit → §6 Apply → §7 Measure → §8 Decide** |
+
+## 1. Set up the split (once per dataset)
+
+Split the tier's `instance_id`s into **TRAIN** (you diagnose on these — traces
+visible) and **HELD-OUT** (blind — you NEVER read their traces; only their
+aggregate F1 gates a keep). ~70/30, deterministic (sort ids, take the first 70%
+as train). Record both lists at the top of the journal (§9) — they must stay
+fixed across the whole loop, or the held-out gate means nothing.
+
+```bash
+# list the ids (jq over instances.json), then split deterministically
+lastlight-evals run pr-review --instance <all-ids> --no-open   # or omit --instance for the whole tier
+```
+
+Set a **target**: an F1 goal (e.g. 0.55) or "improve until plateau" (N=3
+consecutive no-keep iterations).
+
+## 2. Baseline
+
+Run both splits and record `avgFbeta` for each in the journal. Use the cheapest
+adequate model while iterating on prompts; switch to the real target model only to
+confirm a kept change.
+
+```bash
+lastlight-evals run pr-review --overlay instance --instance <train-ids>   --no-open
+lastlight-evals run pr-review --overlay instance --instance <heldout-ids> --no-open
+```
+
+Each writes `eval-results/pr-review/<runId>/scorecard.json`. Note the two runIds.
+
+## 3. Diagnose (TRAIN only)
+
+Read the TRAIN scorecard's `results[].review` and `review.trace`. **Do not open
+held-out traces.** For each failing case:
+- `falseNegatives` (with `severity`) = **recall** loss — real issues the agent missed.
+- `falsePositives` = **precision** loss — noise the agent posted that matched no gold.
+
+Cluster across cases and name the **systematic pattern** — not a one-off. Good
+patterns generalize: *"misses security-relevant findings"*, *"posts style nits the
+rubric says to suppress"*, *"confidence bar too low → noise"*, *"no awareness of the
+repo's conventions → wrong-fit findings"*. See **references/levers.md** for how
+each pattern maps to an edit.
+
+## 4. Hypothesis — ONE change, lowest lever first
+
+Pick the **lowest** lever that could move the whole cluster (full detail in
+**references/levers.md**):
+
+- **(a) generic — auto.** Edit an overlay prompt/skill/persona, or add a generic
+  `instance/repo-context/AGENTS.md` injected into every repo. Must be general.
+- **(b) per-repo context — signed-off.** A `datasets/pr-review/context/<id>/AGENTS.md`
+  the harness injects only for that repo — the portable *"add this to your repo"*
+  finding. Human approves before writing.
+- **(c) the eval itself — rare, signed-off.** Edit `review_gold[]` ONLY when the
+  gold is demonstrably wrong/incomplete; name the evidence. Human approves.
+
+Escalate only when a lower lever genuinely can't address the pattern, and write
+down why.
+
+## 5. Audit the change (adversarial sub-agent)
+
+Before applying, spawn a **generality + leak auditor** sub-agent (prompt template
+in **references/guardrails.md**). It **REJECTS** if the change:
+- names a specific repo / `instance_id` / file path (lever a must be generic), or
+- **encodes the gold answer** — injected repo context that describes the specific
+  findings is cheating. It must read as plausible maintainer guidance written
+  *without* knowledge of this PR's bug (conventions, architecture, review
+  priorities), never "look for bug X".
+
+Rejected → back to §4.
+
+## 6. Apply (hybrid autonomy)
+
+- **Lever (a)** (incl. generic `repo-context/AGENTS.md`): apply automatically to
+  `instance/…` once the auditor passes.
+- **Lever (b)/(c)**: **STOP.** Present the diff + the written justification and ask
+  the human to approve before writing to the dataset.
+
+## 7. Re-measure
+
+Re-run both splits (§2) with the change, then diff against the baseline:
+
+```bash
+npx tsx scripts/diff-runs.ts <baseline-train.json> <candidate-train.json> \
+    --train <train-ids> --heldout <heldout-ids>
+```
+
+Point it at the run pair that covers both splits (or run the two splits in one
+`--instance <all-ids>` run so a single scorecard holds both). It prints per-case
+F1 deltas, the arm-summary delta, and a **KEEP / REVERT verdict**.
+
+## 8. Decide
+
+- **KEEP** iff train F1 improved **and** held-out did **not** regress (the helper's
+  verdict). Train ↑ with held-out ↓ = **overfit → revert**.
+- **Lever (b)** per-repo edits are case-scoped: keep iff that case improved, the
+  auditor passed, and the human approved — recorded as a per-repo recommendation,
+  not a workflow change (no held-out claim is made for it).
+- Else **REVERT**: `git checkout` the overlay file, or remove the sidecar.
+
+## 9. Journal, then repeat
+
+Append one entry per iteration (schema in **references/journal-format.md**):
+pattern, lever, the diff/context, auditor verdict, train Δ, held-out Δ, decision,
+running best. Then return to §3. Stop at the target or the plateau.
+
+Run the loop on a cadence with the built-in `/loop` if you want it to keep going
+unattended between your check-ins — but the lever-(b)/(c) sign-off stops still
+apply.
+
+## Done when
+
+The target F1 is reached or the loop plateaus, the journal records every
+hypothesis and its train/held-out deltas, kept changes live in `instance/` (and,
+with sign-off, per-repo `context/` sidecars), and **no** change was kept that
+regressed the blind held-out set. Report: baseline → final F1 (train and
+held-out), the kept changes, any per-repo recommendations, and anything that
+needed human sign-off.

package/plugins/lastlight/skills/lastlight-evals-loop/references/guardrails.md

@@ -0,0 +1,86 @@
+# Guardrails — keeping the loop honest
+
+The loop sees the gold answer and the agent's answer, so it *could* game the score.
+These guardrails make that hard and, where it isn't automatable, put a human in the
+way. None is optional.
+
+## 1. Held-out split (empirical anti-overfit gate)
+
+Diagnose on TRAIN; validate on a BLIND HELD-OUT split whose traces you never read.
+A generic (lever a) change is kept only if **train F1 improves AND held-out F1 does
+not regress** beyond a small epsilon. A train gain with a held-out regression is
+overfitting — revert it. `scripts/diff-runs.ts` computes this verdict for you; do
+not override it by eyeballing train alone.
+
+The split ids are fixed for the whole loop. If you re-split (e.g. to add cases),
+say so in the journal and treat prior held-out results as no longer comparable.
+
+## 2. One change per iteration
+
+Isolates causal effect. If two edits ship together and the number moves, you can't
+attribute it — and you can't tell an overfit edit riding along with a good one.
+
+## 3. The generality + leak auditor (adversarial sub-agent)
+
+Before applying ANY change, spawn a fresh sub-agent to try to reject it. It is
+adversarial on purpose — its job is to find the reason this is gaming, not to
+approve. Give it the diff (or the injected-context text), the diagnosed pattern,
+and this instruction:
+
+> You are auditing a proposed change to a code-review agent that is being tuned
+> against an eval. Your job is to REJECT changes that game the eval instead of
+> genuinely improving the reviewer. Default to REJECT when unsure.
+>
+> REJECT if the change:
+> 1. names or targets a specific repository, `instance_id`, file path, function,
+>    or PR — a generic prompt/skill/persona edit (lever a) must apply to any repo;
+> 2. encodes the eval's gold answer — e.g. injected repo context or a prompt that
+>    describes the specific findings the agent is "supposed to" make. Injected
+>    repo context must read as plausible standing maintainer guidance written
+>    WITHOUT knowledge of this PR's bug (conventions, architecture, review
+>    priorities), never "check for <the specific issue>";
+> 3. hard-codes an answer, a magic string, or a case count to pass a known case;
+> 4. (for a gold/eval edit) lacks concrete evidence that the existing gold is
+>    wrong — "it makes a case pass" is not evidence.
+>
+> Otherwise ACCEPT. Reply with a verdict (ACCEPT/REJECT) and a one-line reason.
+
+A REJECT sends you back to reformulating a more general change. Record the verdict
+either way.
+
+## 4. The lever ladder + sign-off
+
+Generic (a) edits auto-apply once the auditor passes. Per-repo context (b) and any
+gold/PR-text edit (c) **stop for human sign-off** with a written justification —
+they are the game-able levers, so a human always ratifies them. Never edit
+`review_gold` to make a failing case pass; only when the gold is demonstrably wrong.
+
+## 5. Never touch core
+
+Edits live in `instance/` (overlay) and, with sign-off, dataset sidecars / gold.
+Core workflow code is never modified — the whole value of the harness is that the
+*real* workflow runs. If a fix seems to require a core change, it's out of scope
+for the loop; note it for the maintainers instead.
+
+## 6. Auditable journal + provenance
+
+Every hypothesis → diff → train/held-out deltas → decision is recorded
+(references/journal-format.md), and every run records which injected context it saw
+(`injectedContext` on each result). A reviewer can reconstruct exactly what changed
+and why the number moved — no silent wins.
+
+## The keep/revert decision, precisely
+
+```
+lever (a) generic:
+    KEEP  iff  trainΔ > epsilon  AND  heldoutΔ >= -epsilon   (else REVERT)
+lever (b) per-repo:
+    KEEP  iff  that case improved  AND  auditor ACCEPTED  AND  human approved
+              (recorded as a per-repo recommendation; no held-out claim)
+lever (c) gold/eval:
+    APPLY iff  auditor ACCEPTED  AND  human approved  AND  evidence recorded
+              (this changes what "correct" means — it's a dataset fix, not a score win)
+```
+
+If a split is missing (e.g. you forgot to run held-out), the helper says
+INCONCLUSIVE / REVIEW rather than KEEP — don't keep on train alone.

package/plugins/lastlight/skills/lastlight-evals-loop/references/journal-format.md

@@ -0,0 +1,54 @@
+# Journal format — the auditable ledger
+
+The loop appends one entry per iteration to a single Markdown file, by default
+`eval-results/loop-journal.md` in the workspace (next to the runs it references).
+It is the record that lets a human reconstruct every change and why the score
+moved — keep it complete even for reverted iterations (a revert is a result).
+
+## Header (write once, at the top)
+
+```markdown
+# pr-review improvement loop
+
+- Model under test: anthropic/claude-sonnet-5   (iterating cheap: haiku)
+- Judge model: <EVAL_JUDGE_MODEL or default>
+- Target: F1 >= 0.55  (or: improve until 3 consecutive no-keep iterations)
+- Split (FIXED for this loop):
+  - TRAIN   (n): id1, id2, id3, ...
+  - HELDOUT (n): idA, idB, ...
+- Baseline: train F1 = 0.33, heldout F1 = 0.31   (runIds: <train>, <heldout>)
+```
+
+## Per-iteration entry
+
+```markdown
+## Iteration <N> — <one-line hypothesis>
+
+- Pattern: <the systematic failure named in §3, e.g. "posts style nits the rubric suppresses (precision loss across 4 train cases)">
+- Lever: (a) generic | (b) per-repo | (c) gold   — <why this lever; why not lower>
+- Change:
+    file: instance/skills/code-review/SKILL.md
+    diff: |
+      <the actual diff, or the injected-context text>
+- Auditor: ACCEPT | REJECT — <one-line reason>
+- Applied: auto | human-approved (<who/when>) | not applied (auditor rejected)
+- Result (diff-runs.ts):
+    train:   0.33 -> 0.41   (Δ +0.08)
+    heldout: 0.31 -> 0.31   (Δ +0.00)
+    per-case regressions: 0
+- Decision: KEEP | REVERT | (b) recorded as per-repo recommendation
+- Running best: train F1 = 0.41, heldout F1 = 0.31
+```
+
+## Rules
+
+- **One entry per iteration**, including reverts and auditor rejections — the tail
+  of no-keep iterations is the plateau signal.
+- Paste the **actual diff / context text**, not a paraphrase — the point is
+  reproducibility.
+- Always record **both** train and held-out deltas. A missing held-out number
+  means the decision is INCONCLUSIVE, not KEEP.
+- For lever (b)/(c), record the **human sign-off** and, for (c), the **evidence**
+  that the gold was wrong.
+- Keep the **Running best** line current — it's the at-a-glance progress toward
+  the target and the input to the plateau check.

package/plugins/lastlight/skills/lastlight-evals-loop/references/levers.md

@@ -0,0 +1,80 @@
+# The three levers — what to edit, and the escalation ladder
+
+All three levers change eval outcomes **without touching core** — the real
+production workflow always runs unmodified. You only edit the deployment overlay
+(`instance/`) and, with sign-off, the dataset. Prefer the lowest lever that can
+move the whole failure cluster; escalate only with a written reason.
+
+## Pattern → lever map
+
+| Diagnosed pattern (from TRAIN traces) | Most likely lever |
+|---|---|
+| Posts noise / nits the rubric says to suppress (precision loss) | (a) tighten the severity/precision bar in `code-review` skill |
+| Misses a whole class of real issues, e.g. security, concurrency (recall loss) | (a) add that class to the reviewer's what-to-check rubric/prompt |
+| Reviews against the wrong conventions for a repo | (a) generic repo-context, or (b) per-repo context |
+| Wrong-fit findings because it lacks a repo's build/test/domain norms | (b) per-repo `AGENTS.md` (a portable recommendation) |
+| A gold comment is a non-actionable nit, or a clear real finding scores as a false positive because gold omits it | (c) fix the gold — rare, evidence required |
+
+## (a) Prompts / skills / persona + generic repo-context — *generic, auto*
+
+The preferred lever. A change here must generalize to every repo. Edit copies in
+the overlay (fork with `lastlight fork <workflow>` / `lastlight fork agent-context`,
+or create the shadowing file at the same logical path — see the **lastlight-overlay**
+skill). Common targets:
+
+- `instance/skills/code-review/SKILL.md` — the severity rubric + precision bar
+  (what findings survive). The highest-leverage file for precision problems.
+- `instance/skills/pr-review/SKILL.md` — the review procedure (confidence gate,
+  findings-file contract).
+- `instance/workflows/prompts/reviewer.md` — the reviewer prompt text.
+- `instance/workflows/pr-review.yaml` — phase wiring (which skills a phase loads).
+- `instance/agent-context/rules.md` / `soul.md` — global persona/rules the agent
+  reads via the workspace `AGENTS.md`.
+- `instance/repo-context/AGENTS.md` — **generic** synthetic repo context the
+  harness injects into *every* seeded pr-review repo. Use for guidance that would
+  help a reviewer on any codebase (e.g. "prefer concrete, line-anchored findings;
+  suppress pure style nits"). Held-out gates a keep, so it must genuinely
+  generalize.
+
+## (b) Per-repo injected context — *portable finding, signed-off*
+
+A dataset sidecar the harness injects **only for that case's repo**:
+
+```
+datasets/pr-review/context/<instance_id>/AGENTS.md      (or CLAUDE.md)
+```
+
+This is the *"add this to your repo and your reviews improve by X"* recommendation
+— honest because it's exactly what a maintainer could commit. It is **case-scoped**,
+so the held-out generalization gate doesn't apply; instead it needs human sign-off
+and must pass the no-gold-leak auditor (it must NOT describe this PR's specific
+findings — only standing repo conventions, architecture, and review priorities).
+
+How injection works (so you can trust it lands): the harness writes the block into
+the file the agent's runtime actually reads — it appends to an existing
+`AGENTS.md`, else an existing `CLAUDE.md` (never creating an `AGENTS.md` that would
+shadow it), else creates a fresh `AGENTS.md`. Pi auto-loads that file walking up
+from the agent's cwd, so it reaches the model every phase with no prompt change.
+Toggle off for a clean control run with `--no-inject-context` (or
+`EVAL_INJECT_CONTEXT=0`). Provenance is recorded on each result
+(`injectedContext`).
+
+## (c) The eval itself — *rare, signed-off*
+
+Edit the case in `datasets/pr-review/instances.json`:
+
+- `review_gold[]` (`{ file?, line?, severity, description }`) — the correct answer.
+- `pr.body` / `problem_statement` — what the agent is told about the PR.
+
+Only when the gold is **demonstrably wrong or incomplete** — never to force a
+failing case to pass. The justification must name the evidence (e.g. "the human
+PR review made this exact point; gold omitted it, so a correct finding scored as a
+false positive"). Martian's gold set is known-incomplete by design, so a
+real-but-unlisted finding scoring as a false positive is expected — that is *not*
+on its own grounds to edit gold; it's why the headline metric is F1, not F0.5.
+
+## Escalation ladder
+
+Try (a) first. Escalate to (b) only when the pattern is genuinely repo-specific
+(a convention that doesn't generalize). Escalate to (c) only when the eval is
+wrong. Record the reason for every escalation in the journal.

package/plugins/lastlight/skills/lastlight-guide/SKILL.md

@@ -0,0 +1,97 @@
+---
+name: lastlight-guide
+description: Orientation & router for the Last Light skills. Use ONLY when the user has NOT named a concrete Last Light task — i.e. they're unsure which skill/flow they need, ask "what can Last Light do / where do I start", say "help me with Last Light" with no specific goal, or want a guided tour across server / client / overlay / evals. Do NOT use when the ask already names a task — "set up/deploy a server" → lastlight-server, "connect/log in my CLI" → lastlight-client, "customize a workflow/prompt/persona/config" → lastlight-overlay, "run evals / compare models / author a case / build a PR-review dataset" → lastlight-evals, "iteratively improve an eval toward a target score" → lastlight-evals-loop: invoke that skill directly. This skill only routes and, when the goal is ambiguous, asks. Also invocable as /lastlight-guide.
+version: 1.0.0
+tags: [lastlight, guide, router, orientation, help]
+---
+
+# Last Light — where do you want to go?
+
+You don't need to know which skill you need — describe what you want and this
+maps you onto it.
+
+> **If the user already named a concrete task, don't linger here** — hand off to
+> the matching skill immediately (see the menu below). This guide is for
+> *"which one do I need?"*, not for doing the work.
+
+There are two big things people do with Last Light:
+
+- **Operate a deployment** — stand up the agent, connect to it, customize it.
+- **Measure its quality** — run its real workflows as evals, compare models,
+  build datasets.
+
+**If the user's goal is ambiguous, ask** (use `AskUserQuestion`) rather than
+guessing which flow they mean — "are you setting up a server, connecting a CLI to
+one, customizing workflows, or running evals?" Then hand off to the skill below;
+each is a full runbook, so **invoke it and follow it** rather than re-deriving its
+steps here.
+
+## Flow 1 — operate a Last Light deployment
+
+The route from nothing to a running, customized agent. Do these in order the
+first time; each is also usable on its own later.
+
+1. **`lastlight-server`** — install & configure the **server**: Docker stack, a
+   GitHub App, model provider key, and the repos it may manage. Start here to
+   stand one up for the first time. Config lives in a private **overlay** at
+   `instance/`.
+2. **`lastlight-client`** — point the `lastlight` **CLI** at an existing server
+   and log in (saves URL + token to `~/.lastlight/config.json`). For people who
+   *use* a server someone else runs — no Docker, no GitHub App.
+3. **`lastlight-overlay`** — create & customize the `instance/` **overlay**: fork
+   built-in **workflows**, **prompts**, **skills**, or the agent **persona**
+   (`agent-context`), and tune `config.yaml` — overriding only what you name,
+   layered over the packaged defaults. Reach for it whenever you want to change
+   *what the agent does or how it behaves* on an existing deployment.
+
+## Flow 2 — measure quality with Last Light Evals
+
+Not required to run a deployment — this is how you **tell if a model or a
+workflow/prompt change is any good** before shipping it.
+
+- **`lastlight-evals`** — run Last Light's **real** workflows against a mocked
+  GitHub and grade them deterministically; compare models on pass rate / cost /
+  latency; author eval cases; **build a PR-review dataset from your own gold
+  PRs**. Its own front door lists the sub-flows (scaffold · run/compare · browse
+  past runs · author a case from a PR/issue · build a PR-review dataset). If the
+  user just hands you PR URLs and says "make a review eval set", that skill's
+  interactive flow prompts for the URLs and does the rest.
+
+## The bridge — overlay ↔ evals
+
+The **overlay is shared** by both flows. `lastlight-evals --overlay <dir>` points
+the harness at a *deployment's own* workflows **and** datasets, so you evaluate
+exactly what you ship. The loop that ties it together:
+
+> customize in **`lastlight-overlay`** → measure the change with
+> **`lastlight-evals --overlay .`** → keep it or revert.
+
+To run that loop *toward a target score* — automatically, and without overfitting
+to specific cases — use **`lastlight-evals-loop`**: it diagnoses on a train split,
+validates on a blind held-out split, and proposes one generic overlay fix at a
+time (stopping for sign-off before it touches a gold answer).
+
+## Quick menu — "I want to…"
+
+| …do this | go to |
+|---|---|
+| Stand up / install / deploy a server for the first time | **`lastlight-server`** |
+| Connect my CLI to a server / log in / run commands against it | **`lastlight-client`** |
+| Customize a workflow, prompt, skill, or the agent's persona; tune config | **`lastlight-overlay`** |
+| Run evals, compare models, browse past runs | **`lastlight-evals`** |
+| Author an eval case from a GitHub PR or issue | **`lastlight-evals`** (§6) |
+| Build a PR-review eval dataset from a list of gold PRs | **`lastlight-evals`** (§6) |
+| Iteratively improve a workflow/prompt to raise an eval score | **`lastlight-evals-loop`** |
+
+## Preconditions (what every flow assumes)
+
+- **Node 24+** and the CLI: `lastlight` (server/client/overlay) or
+  `lastlight-evals` (evals) — `npm i -g` either.
+- **Docker + a GitHub App** — server only.
+- **`gh` authenticated** — only for `lastlight-evals add-case` (reads real
+  PRs/issues); running evals mocks GitHub and needs no token.
+- **A model provider API key** — server and evals (`OPENAI_API_KEY` /
+  `ANTHROPIC_API_KEY` / …).
+
+Not sure which you have? Ask the user what they're trying to accomplish, confirm
+the precondition for that flow, then hand off to the skill.