lastlight 0.7.8 → 0.8.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lastlight",
-  "version": "0.7.8",
+  "version": "0.8.0",
   "description": "GitHub repository maintenance agent — Agent SDK harness",
   "type": "module",
   "main": "dist/index.js",

package/plugins/lastlight/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin.json",
   "name": "lastlight",
   "displayName": "Last Light",
-  "version": "0.7.8",
+  "version": "0.8.0",
   "description": "Install, configure and operate Last Light (GitHub maintenance agent) — its server, CLI client, deployment overlay, and the Last Light Evals harness.",
   "author": {
     "name": "Clifton Cunningham"

package/plugins/lastlight/README.md

@@ -13,6 +13,7 @@ Light's *internal* sandbox skills under the repo's top-level `skills/` dir
 
 | Skill | Use it when you want to… |
 |-------|--------------------------|
+| `lastlight-guide` | **Not sure which of the below you need** — orientation & router across all the Last Light skills. Start here (or `/lastlight-guide`). |
 | `lastlight-server` | Install & configure a Last Light **server** (the agent + docker stack) on a host. |
 | `lastlight-client` | Point the `lastlight` **CLI client** at an existing server and log in. |
 | `lastlight-overlay` | Create a deployment **overlay** instance and fork/customize workflows, prompts, skills, or the agent persona. |
@@ -41,4 +42,6 @@ claude plugin install lastlight@lastlight-skills
 auto-discovers them on the next session.
 
 After installing, start a new Claude Code session and say e.g. *"set up a Last
-Light server"* or *"scaffold a Last Light evals workspace"*.
+Light server"* or *"scaffold a Last Light evals workspace"* — or, if you're not
+sure where to start, *"help me with Last Light"* (or `/lastlight-guide`) and the
+guide skill will route you.

package/plugins/lastlight/skills/lastlight-evals/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: lastlight-evals
-description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases — including "create an eval dataset/case from this PR/issue <github url>" (triage / code-fix instances). GitHub is mocked when running, so no real GitHub token is needed — only a model provider API key (the one exception is `add-case`, which reads real PRs/issues via `gh`).
-version: 1.2.0
+description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases — including "create an eval dataset/case from this PR/issue <github url>" and "create a PR-review eval dataset from these gold PRs" (triage / code-fix / pr-review instances, via `add-case --pr <url> --review|--code-fix` / `--issue`). GitHub is mocked when running, so no real GitHub token is needed — only a model provider API key (the one exception is `add-case`, which reads real PRs/issues via `gh`).
+version: 1.3.0
 tags: [lastlight, evals, benchmark, models, swe-bench]
 ---
 
@@ -14,6 +14,24 @@ thin CLI on top of the `lastlight` core package (via the `lastlight/evals`
 barrel), so it exercises the same workflows/skills production does. SWE-bench
 compatible. **Node 24+.**
 
+## Start here — what do you want to do?
+
+If the user's evals goal is clear, jump to the section. If it's vague ("help me
+with the evals", "what can this do?"), **ask** (`AskUserQuestion`) which of these
+they want, then go:
+
+| Goal | Section |
+|---|---|
+| Set up an evals workspace (first time) | **§2 Scaffold** (+ §1 prereqs, §3 providers) |
+| Run evals / compare models | **§4 Run** |
+| Look at past runs (no models run) | **§4** — `lastlight-evals serve` |
+| Author one case from a GitHub PR or issue | **§6** |
+| **Build a PR-review dataset from my own gold PRs** | **§6 → "Build a PR-review dataset"** |
+| Add cases by hand / understand the schema | **§5** + `references/instance-schema.md` |
+
+New to the whole plugin (server / overlay / client, not just evals)? That's the
+**`lastlight-guide`** skill — this one is evals-only.
+
 ## 1. Check prerequisites
 
 ```bash
@@ -102,7 +120,7 @@ lastlight-evals run triage --model openai/gpt-5.5,anthropic/claude-opus-4-8
 lastlight-evals run --compare                   # cross-vendor set (only models whose envKey is present)
 lastlight-evals run triage --runs 3             # repeat each case 3× (worst-case verdict, mean metrics)
 lastlight-evals run pr-review --limit 3         # only the first 3 cases of the tier (controlled/cheap run)
-lastlight-evals run triage --instance <id>      # only this exact instance_id (or set EVAL_INSTANCE)
+lastlight-evals run triage --instance <id[,id2]> # only these exact instance_id(s), comma-separated (or set EVAL_INSTANCE)
 lastlight-evals run pr-review --f-beta 0.5      # pr-review F-beta β (default 1=F1; 0.5=precision 2×). Or EVAL_F_BETA
 lastlight-evals run pr-review --judge-with-diff # feed the PR diff to the judge (higher fidelity; off by default)
 lastlight-evals run pr-review --sandbox gondolin # isolate the agent's tools in a QEMU micro-VM (anti-spoil). Or EVAL_SANDBOX
@@ -144,9 +162,11 @@ Quick shape (paths are relative to the workspace's `evals/` dir):
   `evals/datasets/code-fix/tests/<instance_id>/` (held-out tests applied at grade time).
 - **PR-review case:** a `SweBenchInstance` with a `pr` fixture (base/head refs +
   commits, checked out at the PR head) and a `review_gold` set (severity +
-  description). The `pr-review` tier ships empty — populate the full Martian
+  description). Populate the full Martian
   [Code Review Bench](https://codereview.withmartian.com/) 50 with
-  `npx tsx scripts/import-martian.ts`. Grading needs a judge model
+  `npx tsx scripts/import-martian.ts`, **or author your *own* gold-PR dataset**
+  with `add-case --pr <url> --review` (see §6 and
+  **`references/authoring-pr-review.md`**). Grading needs a judge model
   (`EVAL_JUDGE_MODEL`, else a strong default per provider key).
 - **Custom tier:** a new `evals/datasets/<tier>/` with `tier.json` +
   `instances.json` (+ `repos/` & `tests/` for code-fix-style tiers). Discovery is
@@ -154,16 +174,28 @@ Quick shape (paths are relative to the workspace's `evals/` dir):
 
 ## 6. Author a case from a real GitHub PR or issue
 
-When the user says **"create an eval dataset/case from this PR/issue <url>"**, use
-the `add-case` subcommand — it does the mechanical extraction; you refine the
-fuzzy parts.
+When the user says **"create an eval dataset/case from this PR/issue <url>"** or
+**"create a PR-review eval dataset from these gold PRs"**, use the `add-case`
+subcommand — it does the mechanical extraction; you refine the fuzzy parts.
+
+A PR seeds two different evals, so `--pr` requires a kind — `--review` (the PR's
+human review is the gold set) or `--code-fix` (hide the fix, the agent must
+reproduce it):
 
 ```bash
-lastlight-evals add-case --pr <github-pr-url> --dry-run        # propose a code-fix case; don't write
-lastlight-evals add-case --pr <github-pr-url>                  # write into ./datasets (or --datasets/--overlay)
-lastlight-evals add-case --issue <github-issue-url> --dry-run  # propose a triage case
+lastlight-evals add-case --pr <github-pr-url> --review --dry-run    # propose a pr-review case; don't write
+lastlight-evals add-case --pr <github-pr-url> --review              # write into ./datasets/pr-review (or --overlay)
+lastlight-evals add-case --pr <github-pr-url> --code-fix --dry-run  # propose a code-fix case
+lastlight-evals add-case --issue <github-issue-url> --dry-run       # propose a triage case
 ```
 
+- **From a PR (pr-review):** derives the `pr` fixture (base/head refs + commits,
+  from the PR's `baseRefOid`/`headRefOid`) and a **candidate `review_gold`** from
+  the PR's *human* review — inline comments + substantive review bodies, with
+  bot/nit/reply/outdated noise filtered out. Prints an evidence block; **you**
+  assign real severities and prune non-actionable comments. Byte-compatible with
+  `import-martian.ts`, so authored cases sit alongside imported ones. Leak-safe:
+  the review lands only in `review_gold`, never in the seeded PR the agent sees.
 - **From a PR (code-fix):** derives `repo`, `base_commit` (merge-base of base &
   head) + `head_commit`, the PR's **test** diff as the held-out `test_patch`, and
   auto-detects `FAIL_TO_PASS` / `PASS_TO_PASS` by running the tests at base (red)
@@ -176,9 +208,35 @@ lastlight-evals add-case --issue <github-issue-url> --dry-run  # propose a triag
   **reviewer comments** — the raw triage signal. You then assign
   `triage_gold` (category/state) from those labels per the deployment's taxonomy.
 
+### Build a PR-review dataset from gold PRs (interactive)
+
+When the user says **"I want to create a PR-review eval dataset"** (from *their*
+PRs, to track review quality as they evolve models/prompts):
+
+1. **Get the PRs.** If they didn't paste URLs, **ask** for them — "give me the
+   gold PR URL(s) — one or many, whose human review is the standard you want a
+   model to reproduce." Also confirm *where* to write: the current workspace's
+   `./datasets` (or `--overlay <dir>` for a deployment's own dataset).
+2. **One PR at a time.** For each URL run
+   `add-case --pr <url> --review --dry-run`, read the evidence block, then
+   **curate** the candidate `review_gold` — set real severities and drop
+   comments that aren't concrete findings — and write it
+   (`add-case --pr <url> --review [--datasets/--overlay …]`). Re-running the same
+   PR **replaces by id**, so curation is idempotent.
+   - For **many** PRs, spawn a **curation sub-agent per PR** (each handed one URL
+     + the target dir, returning the curated instance) so the judgement stays
+     isolated and they run in parallel.
+3. **Smoke-run** the new tier with the cheapest model to confirm the seed + judge
+   plumbing (needs a judge-model key): `lastlight-evals run pr-review --limit 1 --model haiku`.
+
+See **`references/authoring-pr-review.md`** for the full extract→curate→write
+flow, the `--severity` / `--include-bots` knobs, and the anti-spoil guarantee.
+
 The full flow, what you refine for each, the `test_cmd` / `setup_cmd` options for
 non-`node --test` runners, and the trust/offline caveats are in
-**`references/authoring-from-pr.md`** — read it before authoring.
+**`references/authoring-from-pr.md`** (PR code-fix + issue triage) and
+**`references/authoring-pr-review.md`** (PR review) — read the relevant one
+before authoring.
 
 ## Done when
 

package/plugins/lastlight/skills/lastlight-evals/references/authoring-from-pr.md

@@ -1,11 +1,17 @@
-# Author an eval case from a real GitHub PR or issue
+# Author a code-fix or triage eval case from a real GitHub PR or issue
 
-`lastlight-evals add-case` turns a real GitHub **PR** into a code-fix (build) case
-or an **issue** into a triage case. It does the mechanical, reproducible
-extraction with `gh` + `git`; **you** refine the judgement parts. The result is a
-**git-source** case — no fixture repo is vendored; at run time the harness clones
-the repo into the gitignored `./.eval-cache/` and checks out `base_commit` (see
-the `instance-schema.md` "Git-source" flavor).
+`lastlight-evals add-case --pr <url> --code-fix` turns a real GitHub **PR** into a
+code-fix (build) case; `--issue <url>` turns an **issue** into a triage case. It
+does the mechanical, reproducible extraction with `gh` + `git`; **you** refine the
+judgement parts. The result is a **git-source** case — no fixture repo is
+vendored; at run time the harness clones the repo into the gitignored
+`./.eval-cache/` and checks out `base_commit` (see the `instance-schema.md`
+"Git-source" flavor).
+
+> A PR seeds two different evals. Use `--code-fix` (this doc) to hide the fix and
+> have the agent reproduce it; use `--review` (see **`authoring-pr-review.md`**)
+> to grade a review of the PR against its human review. `--pr` **requires** one of
+> the two — it no longer defaults to code-fix.
 
 ## Two grading modes (default: suite)
 
@@ -34,7 +40,7 @@ the `instance-schema.md` "Git-source" flavor).
 ## Command
 
 ```bash
-lastlight-evals add-case --pr <github-pr-url> [options]
+lastlight-evals add-case --pr <github-pr-url> --code-fix [options]
 lastlight-evals add-case --issue <github-issue-url> [options]
 ```
 
@@ -42,9 +48,10 @@ Options:
 
 | flag | meaning |
 |---|---|
-| `--pr <url>` | a GitHub PR url → a **code-fix** (build) case |
+| `--pr <url>` | a GitHub PR url (pair with `--code-fix` here, or `--review` for a pr-review case) |
+| `--code-fix` | with `--pr` → a **code-fix** (build) case |
 | `--issue <url>` | a GitHub issue url → a **triage** case |
-| `--tier <name>` | target tier dir (default `code-fix` for `--pr`, `triage` for `--issue`) |
+| `--tier <name>` | target tier dir (default `code-fix` for `--pr --code-fix`, `triage` for `--issue`) |
 | `--id <slug>` | `instance_id` (default derived from repo + number) |
 | `--datasets <dir>` | datasets root to write into (a `<tier>/` subdir). Default `./datasets`, else `./evals/datasets` |
 | `--overlay <dir>` | write into `<dir>/evals/datasets` instead |
@@ -57,8 +64,8 @@ Options:
 
 ## The recommended flow (CLI extracts → you refine)
 
-1. **Dry-run first.** `add-case --pr <url> --dry-run` prints the proposed instance.
-   The CLI derives:
+1. **Dry-run first.** `add-case --pr <url> --code-fix --dry-run` prints the proposed
+   instance. The CLI derives:
    - `repo`, `base_commit` (the **merge-base** of the base branch and the PR head —
      the true fork point, not the base-branch tip), and `head_commit`;
    - gold `patch` — the diff of the non-test files (reference only, never graded);

package/plugins/lastlight/skills/lastlight-evals/references/authoring-pr-review.md

@@ -0,0 +1,110 @@
+# Author a pr-review eval case from a real GitHub PR
+
+`lastlight-evals add-case --pr <url> --review` turns a real GitHub **PR** into a
+`pr-review` case: the PR's **human review is the gold set** the model's review is
+scored against (precision / recall / F-beta, via an LLM judge). It does the
+mechanical, reproducible extraction with `gh`; **you** curate the gold. The output
+is **byte-compatible** with `scripts/import-martian.ts`, so cases you author sit
+alongside the Martian Code Review Bench cases in the same `pr-review/instances.json`.
+
+This is the path for *"I want to create a PR-review eval dataset from these gold
+PRs"* — PRs whose human reviews are the standard you want a model (or an evolving
+prompt/skill) to reproduce.
+
+## What it extracts
+
+- **The `pr` fixture** — `base_ref`/`head_ref` + `base_commit`/`head_commit` (from
+  the PR's `baseRefOid`/`headRefOid`), `title`, `body` (truncated), `user`. At run
+  time the harness clones the repo and checks out the PR **head**; the review
+  workflow diffs `base..head`. No repo clone happens at author time.
+- **A candidate `review_gold`** from the PR's *human* review: inline review
+  comments (`path` + `line` + body) and substantive top-level review bodies. The
+  CLI filters the obvious noise — reply threads, `LGTM`/approve-only comments, and
+  bot comments (unless `--include-bots`) — and defaults every entry's `severity`
+  to `medium` (GitHub comments carry none). Outdated comments (whose hunk moved)
+  are kept via `original_line` and flagged.
+
+> **Anti-spoil.** The human review is written **only** into `review_gold` (the
+> held-out judge gold). The CLI never puts it in `pr.reviews` / `pr.review_comments`
+> — those seed the mocked GitHub the agent can read, which would hand it the
+> answer. This mirrors `import-martian.ts`, and the run-time seeding
+> (`seedWorkspacePrReview`) only checks out git, never the prior discussion.
+
+## Prerequisites
+
+- `gh` on PATH and authenticated (`gh auth login`) — used to read PR metadata + review comments.
+- Node 24+ (same as the rest of the harness). A **judge model** at run time
+  (`EVAL_JUDGE_MODEL`, else a strong default per provider key).
+
+## Command
+
+```bash
+lastlight-evals add-case --pr <github-pr-url> --review [options]
+```
+
+Options:
+
+| flag | meaning |
+|---|---|
+| `--pr <url>` | a GitHub PR url (pair with `--review` here, or `--code-fix` for a build case) |
+| `--review` | with `--pr` → a **pr-review** case (gold = the PR's human review) |
+| `--severity <level>` | default severity for candidate gold comments (`low`\|`medium`\|`high`\|`critical`; default `medium`) |
+| `--include-bots` | keep review comments from bots (default: skipped) |
+| `--tier <name>` | target tier dir (default `pr-review`) |
+| `--id <slug>` | `instance_id` (default `prreview__<repo>-<n>`) |
+| `--datasets <dir>` | datasets root to write into (a `<tier>/` subdir). Default `./datasets`, else `./evals/datasets` |
+| `--overlay <dir>` | write into `<dir>/evals/datasets` instead |
+| `--dry-run` | print the proposed instance JSON + evidence block; don't write |
+
+## The recommended flow (CLI extracts → you refine)
+
+1. **Dry-run first.** `add-case --pr <url> --review --dry-run` prints the proposed
+   instance and an **evidence block** listing every candidate gold comment
+   (`author · file:line [severity] first-line`, tagged `(outdated)` / `(bot)`).
+2. **Read the evidence and curate `review_gold`** — this is the whole point of a
+   *gold* dataset:
+   - **Severity.** Every candidate defaults to `medium`. Set each to its real
+     impact (`critical`/`high`/`medium`/`low`) — the judge is severity-aware.
+   - **Prune noise.** Drop comments that aren't concrete findings — questions,
+     style nits, "why not X?", praise, anything `(outdated)` that no longer applies
+     to the head diff.
+   - **Add what the humans missed** (optional). If you know of a real issue the
+     human reviewer didn't post, add a `review_gold` entry by hand (`severity` +
+     `description`; `file`/`line` optional — the judge matches on substance).
+   - **Empty gold?** If the PR had no usable review comments, the CLI warns and
+     leaves `review_gold: []` — author it by hand from the PR diff, or the case
+     can't be judged.
+3. **Write it.** Re-run without `--dry-run` (add `--datasets <dir>` / `--overlay
+   <dir>` to target a workspace). The instance is appended to
+   `<root>/pr-review/instances.json` (creating `tier.json` if the tier is new);
+   re-running with the same PR **replaces by `instance_id`**, so curation is
+   idempotent.
+4. **Verify end-to-end** with the cheapest model — confirms the seed + judge
+   plumbing (needs a judge-model key):
+
+   ```bash
+   EVAL_INSTANCE=<instance_id> lastlight-evals run pr-review --model haiku
+   ```
+
+   Inspect the judge trace in the dashboard (the **judge** button) to see the
+   finding↔gold pairing behind the precision/recall/F score.
+
+## Authoring from a list of gold PRs
+
+Run `add-case --pr <url> --review` **once per URL** — each appends into the same
+`pr-review/instances.json`. Because `review_gold` curation is per-PR judgement,
+do one curation pass per PR rather than a bulk import: dry-run → refine → write.
+Spawning a **curation sub-agent per PR** (each handed one URL, returning the
+curated instance) keeps the judgement isolated and parallelizes cleanly, while the
+CLI's replace-by-id write makes accumulation safe.
+
+## Tuning knobs
+
+- **`--severity <level>`** sets the default for *all* candidates in one PR — handy
+  when a PR's review is uniformly high- or low-stakes; you still refine per entry.
+- **`--include-bots`** keeps bot review comments (CodeRabbit, Greptile, etc.).
+  Off by default because bot comments are noisy and rarely the human gold; include
+  them only if the bot's review *is* the standard you're grading against.
+
+See `instance-schema.md` for the full `pr` / `review_gold` field reference and the
+pr-review grading model.

package/plugins/lastlight/skills/lastlight-evals/references/instance-schema.md

@@ -44,12 +44,23 @@ Datasets are discovered from (overlay > user > built-in): `<overlay>/evals/datas
     "state": "open"
   },
   "triage_gold": { "category": "bug", "state": "ready-for-agent" },  // triage grading
+  "pr": {                                  // pr-review: the PR fixture (base/head refs + SHAs)
+    "number": 37429, "title": "...", "body": "...",
+    "base_ref": "main", "head_ref": "feature-x",
+    "base_commit": "f3c8e8f...", "head_commit": "02f48f7...",
+    "user": "author"                       // do NOT put the human review here — see below
+  },
+  "review_gold": [                          // pr-review: the held-out judge gold
+    { "file": "src/x.ts", "line": 42, "severity": "high", "description": "..." },
+    { "severity": "medium", "description": "..." }   // file/line optional (judge matches on substance)
+  ],
   "expect_github": {                       // behavioral assertions on recorded GitHub calls
     "labels_added": ["bug"],
     "labels_absent": ["wontfix"],
     "issue_closed": false,
     "comment_matches": "(?i)thanks",
-    "pr_opened": { "base": "main", "head_is_branch": true, "title_matches": "(?i)fix" }
+    "pr_opened": { "base": "main", "head_is_branch": true, "title_matches": "(?i)fix" },
+    "review_submitted": {}                 // pr-review: proxy check that a review was posted
   }
 }
 ```
@@ -83,6 +94,26 @@ hand-build these — use `lastlight-evals add-case --pr <url>` (see
 **`authoring-from-pr.md`**), which fills `base_commit`, `head_commit`,
 `test_patch`, and the verdicts for you.
 
+## Add a pr-review case
+
+A pr-review instance has a **`pr`** fixture + a **`review_gold`** set and
+`workflow: "pr-review"`. The `pr` fixture drives both the mocked PR endpoints and
+the workspace checkout: at run time the harness checks out the PR **head**, and
+the review workflow diffs `base..head`. `review_gold` is the held-out gold the
+posted review is scored against by an LLM judge (each entry: `severity` ∈
+`low`|`medium`|`high`|`critical` + `description`; `file`/`line` optional — the
+judge matches on substance). Set `expect_github.review_submitted: {}` (a cheap
+proxy that a review was posted).
+
+> **Never** put the human/gold review in `pr.reviews` / `pr.review_comments` —
+> those seed the mocked GitHub the agent can read, spoiling the case. The gold
+> lives only in `review_gold`.
+
+Don't hand-build these — use `lastlight-evals add-case --pr <url> --review` (see
+**`authoring-pr-review.md`**), which pins the `pr` fixture and seeds a candidate
+`review_gold` from the PR's human review for you to curate. Or bulk-import the
+Martian Code Review Bench with `npx tsx scripts/import-martian.ts`.
+
 ## Add a custom tier
 
 Create `datasets/<tier-name>/` with `tier.json` (`name`, `defaultWorkflow`,
@@ -95,6 +126,11 @@ and `tests/<id>/`. Discovery auto-finds it — no code change. Run it with
 - **Behavioral:** did the workflow take the expected GitHub actions
   (`expect_github`)?
 - **Triage:** did the decision match `triage_gold`?
+- **Review (pr-review):** an LLM judge matches the posted review's findings against
+  `review_gold` → **precision / recall / F-beta** (β via `EVAL_F_BETA` or
+  `--f-beta`; default F1). Needs a judge model (`EVAL_JUDGE_MODEL`, else a strong
+  default per provider key); independent of the model under test. The judge trace
+  is inspectable in the dashboard.
 - **Execution (code-fix)** — two modes:
   - **Suite (default).** Nothing held out: run the repo's own `test_cmd` against the
     agent's final tree, **resolved iff it exits 0**. Grades "did the agent leave the

package/plugins/lastlight/skills/lastlight-guide/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: lastlight-guide
+description: Orientation & router for the Last Light skills. Use ONLY when the user has NOT named a concrete Last Light task — i.e. they're unsure which skill/flow they need, ask "what can Last Light do / where do I start", say "help me with Last Light" with no specific goal, or want a guided tour across server / client / overlay / evals. Do NOT use when the ask already names a task — "set up/deploy a server" → lastlight-server, "connect/log in my CLI" → lastlight-client, "customize a workflow/prompt/persona/config" → lastlight-overlay, "run evals / compare models / author a case / build a PR-review dataset" → lastlight-evals: invoke that skill directly. This skill only routes and, when the goal is ambiguous, asks. Also invocable as /lastlight-guide.
+version: 1.0.0
+tags: [lastlight, guide, router, orientation, help]
+---
+
+# Last Light — where do you want to go?
+
+You don't need to know which skill you need — describe what you want and this
+maps you onto it.
+
+> **If the user already named a concrete task, don't linger here** — hand off to
+> the matching skill immediately (see the menu below). This guide is for
+> *"which one do I need?"*, not for doing the work.
+
+There are two big things people do with Last Light:
+
+- **Operate a deployment** — stand up the agent, connect to it, customize it.
+- **Measure its quality** — run its real workflows as evals, compare models,
+  build datasets.
+
+**If the user's goal is ambiguous, ask** (use `AskUserQuestion`) rather than
+guessing which flow they mean — "are you setting up a server, connecting a CLI to
+one, customizing workflows, or running evals?" Then hand off to the skill below;
+each is a full runbook, so **invoke it and follow it** rather than re-deriving its
+steps here.
+
+## Flow 1 — operate a Last Light deployment
+
+The route from nothing to a running, customized agent. Do these in order the
+first time; each is also usable on its own later.
+
+1. **`lastlight-server`** — install & configure the **server**: Docker stack, a
+   GitHub App, model provider key, and the repos it may manage. Start here to
+   stand one up for the first time. Config lives in a private **overlay** at
+   `instance/`.
+2. **`lastlight-client`** — point the `lastlight` **CLI** at an existing server
+   and log in (saves URL + token to `~/.lastlight/config.json`). For people who
+   *use* a server someone else runs — no Docker, no GitHub App.
+3. **`lastlight-overlay`** — create & customize the `instance/` **overlay**: fork
+   built-in **workflows**, **prompts**, **skills**, or the agent **persona**
+   (`agent-context`), and tune `config.yaml` — overriding only what you name,
+   layered over the packaged defaults. Reach for it whenever you want to change
+   *what the agent does or how it behaves* on an existing deployment.
+
+## Flow 2 — measure quality with Last Light Evals
+
+Not required to run a deployment — this is how you **tell if a model or a
+workflow/prompt change is any good** before shipping it.
+
+- **`lastlight-evals`** — run Last Light's **real** workflows against a mocked
+  GitHub and grade them deterministically; compare models on pass rate / cost /
+  latency; author eval cases; **build a PR-review dataset from your own gold
+  PRs**. Its own front door lists the sub-flows (scaffold · run/compare · browse
+  past runs · author a case from a PR/issue · build a PR-review dataset). If the
+  user just hands you PR URLs and says "make a review eval set", that skill's
+  interactive flow prompts for the URLs and does the rest.
+
+## The bridge — overlay ↔ evals
+
+The **overlay is shared** by both flows. `lastlight-evals --overlay <dir>` points
+the harness at a *deployment's own* workflows **and** datasets, so you evaluate
+exactly what you ship. The loop that ties it together:
+
+> customize in **`lastlight-overlay`** → measure the change with
+> **`lastlight-evals --overlay .`** → keep it or revert.
+
+## Quick menu — "I want to…"
+
+| …do this | go to |
+|---|---|
+| Stand up / install / deploy a server for the first time | **`lastlight-server`** |
+| Connect my CLI to a server / log in / run commands against it | **`lastlight-client`** |
+| Customize a workflow, prompt, skill, or the agent's persona; tune config | **`lastlight-overlay`** |
+| Run evals, compare models, browse past runs | **`lastlight-evals`** |
+| Author an eval case from a GitHub PR or issue | **`lastlight-evals`** (§6) |
+| Build a PR-review eval dataset from a list of gold PRs | **`lastlight-evals`** (§6) |
+
+## Preconditions (what every flow assumes)
+
+- **Node 24+** and the CLI: `lastlight` (server/client/overlay) or
+  `lastlight-evals` (evals) — `npm i -g` either.
+- **Docker + a GitHub App** — server only.
+- **`gh` authenticated** — only for `lastlight-evals add-case` (reads real
+  PRs/issues); running evals mocks GitHub and needs no token.
+- **A model provider API key** — server and evals (`OPENAI_API_KEY` /
+  `ANTHROPIC_API_KEY` / …).
+
+Not sure which you have? Ask the user what they're trying to accomplish, confirm
+the precondition for that flow, then hand off to the skill.