lastlight 0.8.0 → 0.9.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lastlight",
-  "version": "0.8.0",
+  "version": "0.9.0",
   "description": "GitHub repository maintenance agent — Agent SDK harness",
   "type": "module",
   "main": "dist/index.js",

package/plugins/lastlight/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin.json",
   "name": "lastlight",
   "displayName": "Last Light",
-  "version": "0.8.0",
+  "version": "0.9.0",
   "description": "Install, configure and operate Last Light (GitHub maintenance agent) — its server, CLI client, deployment overlay, and the Last Light Evals harness.",
   "author": {
     "name": "Clifton Cunningham"

package/plugins/lastlight/README.md

@@ -18,6 +18,7 @@ Light's *internal* sandbox skills under the repo's top-level `skills/` dir
 | `lastlight-client` | Point the `lastlight` **CLI client** at an existing server and log in. |
 | `lastlight-overlay` | Create a deployment **overlay** instance and fork/customize workflows, prompts, skills, or the agent persona. |
 | `lastlight-evals` | Scaffold and run a **Last Light Evals** workspace (datasets, models, model comparisons). |
+| `lastlight-evals-loop` | **Drive an eval toward a target score** with a disciplined, anti-gaming improvement loop — diagnose on a train split, validate on a blind held-out split, propose one generic fix at a time. |
 
 ## Install
 

package/plugins/lastlight/skills/lastlight-evals/SKILL.md

@@ -28,6 +28,7 @@ they want, then go:
 | Author one case from a GitHub PR or issue | **§6** |
 | **Build a PR-review dataset from my own gold PRs** | **§6 → "Build a PR-review dataset"** |
 | Add cases by hand / understand the schema | **§5** + `references/instance-schema.md` |
+| **Iteratively improve the score toward a target** | the **`lastlight-evals-loop`** skill |
 
 New to the whole plugin (server / overlay / client, not just evals)? That's the
 **`lastlight-guide`** skill — this one is evals-only.
@@ -123,6 +124,7 @@ lastlight-evals run pr-review --limit 3         # only the first 3 cases of the
 lastlight-evals run triage --instance <id[,id2]> # only these exact instance_id(s), comma-separated (or set EVAL_INSTANCE)
 lastlight-evals run pr-review --f-beta 0.5      # pr-review F-beta β (default 1=F1; 0.5=precision 2×). Or EVAL_F_BETA
 lastlight-evals run pr-review --judge-with-diff # feed the PR diff to the judge (higher fidelity; off by default)
+lastlight-evals run pr-review --no-inject-context # DON'T inject synthetic repo-context into the checkout (clean A/B control)
 lastlight-evals run pr-review --sandbox gondolin # isolate the agent's tools in a QEMU micro-VM (anti-spoil). Or EVAL_SANDBOX
 lastlight-evals run triage --no-open            # don't open the report
 # Plain layout: add --overlay .   (e.g. lastlight-evals run triage --overlay .)
@@ -168,6 +170,15 @@ Quick shape (paths are relative to the workspace's `evals/` dir):
   with `add-case --pr <url> --review` (see §6 and
   **`references/authoring-pr-review.md`**). Grading needs a judge model
   (`EVAL_JUDGE_MODEL`, else a strong default per provider key).
+  - **Repo-context injection (pr-review).** The harness can drop a synthetic
+    `AGENTS.md`/`CLAUDE.md` into the checked-out repo so the reviewing agent reads
+    it — a **generic** block from `<overlay>/repo-context/AGENTS.md` (every repo)
+    plus a **per-repo** block from `datasets/pr-review/context/<instance_id>/AGENTS.md`.
+    Presence-based (just create the file), on by default; `--no-inject-context`
+    (or `EVAL_INJECT_CONTEXT=0`) forces a clean control for an A/B. It appends to
+    a real `AGENTS.md`/`CLAUDE.md` if the repo ships one (never shadowing it). This
+    is how you prove *"adding this to your repo improves review quality"* — and the
+    lever the **`lastlight-evals-loop`** skill drives.
 - **Custom tier:** a new `evals/datasets/<tier>/` with `tier.json` +
   `instances.json` (+ `repos/` & `tests/` for code-fix-style tiers). Discovery is
   automatic — no code change.

package/plugins/lastlight/skills/lastlight-evals-loop/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: lastlight-evals-loop
+description: Drive a Last Light EVAL toward a target score with a disciplined, anti-gaming improvement loop — run → diagnose → propose ONE generic fix → re-measure → keep or revert → repeat. Use when the user wants to "improve / raise the pr-review F1", "make the reviewer better against the eval", "close the loop on evals", "iterate on prompts/skills to pass more cases", or "tune the workflow to hit a score target". The loop diagnoses on a TRAIN split and validates on a BLIND held-out split so fixes must generalize, not overfit; it prefers generic overlay prompt/skill edits, and stops for human sign-off before editing any gold answer. For a one-off run/compare use lastlight-evals; to hand-fork a workflow/prompt use lastlight-overlay. Needs an already-scaffolded evals workspace (lastlight-evals) with a pr-review dataset.
+version: 1.0.0
+tags: [lastlight, evals, loop, improvement, overfitting, pr-review]
+---
+
+# Improve an eval, honestly
+
+A loop that raises an eval score **without gaming it**. The trap: the loop can see
+both the gold answer and the agent's answer for every case, so it could overfit —
+hardcode a repo's finding into a prompt, or edit a gold answer to force a pass.
+That lifts the number and ruins the reviewer. Every guardrail here exists to force
+**generic, cross-cutting** changes that would help on repos the loop has never seen.
+
+Scope: the **pr-review** tier (its judge trace gives you the agent-vs-gold detail
+diagnosis needs). The pattern extends to triage/code-fix, but start here.
+
+## Prerequisites
+
+- A scaffolded evals workspace with a **pr-review** dataset and a working provider
+  key — set up via **`lastlight-evals`** first if you don't have one. Confirm a
+  bare `lastlight-evals run pr-review --limit 1 --no-open` grades a case.
+- An **`instance/` overlay** (the deployment overlay) to receive generic edits —
+  see **`lastlight-overlay`**. The loop edits copies here, never core.
+- The `diff-runs.ts` helper ships with the evals package
+  (`scripts/diff-runs.ts`) — it compares two runs and calls keep/revert.
+
+## The one rule
+
+**One change per iteration.** Propose a single hypothesis, measure its effect in
+isolation, keep or revert, then move on. No batching — batching hides which edit
+moved the number and invites scattershot overfitting.
+
+## Start here
+
+| You have… | Go to |
+|---|---|
+| Never run this loop on this dataset | **§1 Set up the split** then §2 |
+| A split + baseline already recorded | **§3 Diagnose** |
+| A proposed change ready to test | **§5 Audit → §6 Apply → §7 Measure → §8 Decide** |
+
+## 1. Set up the split (once per dataset)
+
+Split the tier's `instance_id`s into **TRAIN** (you diagnose on these — traces
+visible) and **HELD-OUT** (blind — you NEVER read their traces; only their
+aggregate F1 gates a keep). ~70/30, deterministic (sort ids, take the first 70%
+as train). Record both lists at the top of the journal (§9) — they must stay
+fixed across the whole loop, or the held-out gate means nothing.
+
+```bash
+# list the ids (jq over instances.json), then split deterministically
+lastlight-evals run pr-review --instance <all-ids> --no-open   # or omit --instance for the whole tier
+```
+
+Set a **target**: an F1 goal (e.g. 0.55) or "improve until plateau" (N=3
+consecutive no-keep iterations).
+
+## 2. Baseline
+
+Run both splits and record `avgFbeta` for each in the journal. Use the cheapest
+adequate model while iterating on prompts; switch to the real target model only to
+confirm a kept change.
+
+```bash
+lastlight-evals run pr-review --overlay instance --instance <train-ids>   --no-open
+lastlight-evals run pr-review --overlay instance --instance <heldout-ids> --no-open
+```
+
+Each writes `eval-results/pr-review/<runId>/scorecard.json`. Note the two runIds.
+
+## 3. Diagnose (TRAIN only)
+
+Read the TRAIN scorecard's `results[].review` and `review.trace`. **Do not open
+held-out traces.** For each failing case:
+- `falseNegatives` (with `severity`) = **recall** loss — real issues the agent missed.
+- `falsePositives` = **precision** loss — noise the agent posted that matched no gold.
+
+Cluster across cases and name the **systematic pattern** — not a one-off. Good
+patterns generalize: *"misses security-relevant findings"*, *"posts style nits the
+rubric says to suppress"*, *"confidence bar too low → noise"*, *"no awareness of the
+repo's conventions → wrong-fit findings"*. See **references/levers.md** for how
+each pattern maps to an edit.
+
+## 4. Hypothesis — ONE change, lowest lever first
+
+Pick the **lowest** lever that could move the whole cluster (full detail in
+**references/levers.md**):
+
+- **(a) generic — auto.** Edit an overlay prompt/skill/persona, or add a generic
+  `instance/repo-context/AGENTS.md` injected into every repo. Must be general.
+- **(b) per-repo context — signed-off.** A `datasets/pr-review/context/<id>/AGENTS.md`
+  the harness injects only for that repo — the portable *"add this to your repo"*
+  finding. Human approves before writing.
+- **(c) the eval itself — rare, signed-off.** Edit `review_gold[]` ONLY when the
+  gold is demonstrably wrong/incomplete; name the evidence. Human approves.
+
+Escalate only when a lower lever genuinely can't address the pattern, and write
+down why.
+
+## 5. Audit the change (adversarial sub-agent)
+
+Before applying, spawn a **generality + leak auditor** sub-agent (prompt template
+in **references/guardrails.md**). It **REJECTS** if the change:
+- names a specific repo / `instance_id` / file path (lever a must be generic), or
+- **encodes the gold answer** — injected repo context that describes the specific
+  findings is cheating. It must read as plausible maintainer guidance written
+  *without* knowledge of this PR's bug (conventions, architecture, review
+  priorities), never "look for bug X".
+
+Rejected → back to §4.
+
+## 6. Apply (hybrid autonomy)
+
+- **Lever (a)** (incl. generic `repo-context/AGENTS.md`): apply automatically to
+  `instance/…` once the auditor passes.
+- **Lever (b)/(c)**: **STOP.** Present the diff + the written justification and ask
+  the human to approve before writing to the dataset.
+
+## 7. Re-measure
+
+Re-run both splits (§2) with the change, then diff against the baseline:
+
+```bash
+npx tsx scripts/diff-runs.ts <baseline-train.json> <candidate-train.json> \
+    --train <train-ids> --heldout <heldout-ids>
+```
+
+Point it at the run pair that covers both splits (or run the two splits in one
+`--instance <all-ids>` run so a single scorecard holds both). It prints per-case
+F1 deltas, the arm-summary delta, and a **KEEP / REVERT verdict**.
+
+## 8. Decide
+
+- **KEEP** iff train F1 improved **and** held-out did **not** regress (the helper's
+  verdict). Train ↑ with held-out ↓ = **overfit → revert**.
+- **Lever (b)** per-repo edits are case-scoped: keep iff that case improved, the
+  auditor passed, and the human approved — recorded as a per-repo recommendation,
+  not a workflow change (no held-out claim is made for it).
+- Else **REVERT**: `git checkout` the overlay file, or remove the sidecar.
+
+## 9. Journal, then repeat
+
+Append one entry per iteration (schema in **references/journal-format.md**):
+pattern, lever, the diff/context, auditor verdict, train Δ, held-out Δ, decision,
+running best. Then return to §3. Stop at the target or the plateau.
+
+Run the loop on a cadence with the built-in `/loop` if you want it to keep going
+unattended between your check-ins — but the lever-(b)/(c) sign-off stops still
+apply.
+
+## Done when
+
+The target F1 is reached or the loop plateaus, the journal records every
+hypothesis and its train/held-out deltas, kept changes live in `instance/` (and,
+with sign-off, per-repo `context/` sidecars), and **no** change was kept that
+regressed the blind held-out set. Report: baseline → final F1 (train and
+held-out), the kept changes, any per-repo recommendations, and anything that
+needed human sign-off.

package/plugins/lastlight/skills/lastlight-evals-loop/references/guardrails.md

@@ -0,0 +1,86 @@
+# Guardrails — keeping the loop honest
+
+The loop sees the gold answer and the agent's answer, so it *could* game the score.
+These guardrails make that hard and, where it isn't automatable, put a human in the
+way. None is optional.
+
+## 1. Held-out split (empirical anti-overfit gate)
+
+Diagnose on TRAIN; validate on a BLIND HELD-OUT split whose traces you never read.
+A generic (lever a) change is kept only if **train F1 improves AND held-out F1 does
+not regress** beyond a small epsilon. A train gain with a held-out regression is
+overfitting — revert it. `scripts/diff-runs.ts` computes this verdict for you; do
+not override it by eyeballing train alone.
+
+The split ids are fixed for the whole loop. If you re-split (e.g. to add cases),
+say so in the journal and treat prior held-out results as no longer comparable.
+
+## 2. One change per iteration
+
+Isolates causal effect. If two edits ship together and the number moves, you can't
+attribute it — and you can't tell an overfit edit riding along with a good one.
+
+## 3. The generality + leak auditor (adversarial sub-agent)
+
+Before applying ANY change, spawn a fresh sub-agent to try to reject it. It is
+adversarial on purpose — its job is to find the reason this is gaming, not to
+approve. Give it the diff (or the injected-context text), the diagnosed pattern,
+and this instruction:
+
+> You are auditing a proposed change to a code-review agent that is being tuned
+> against an eval. Your job is to REJECT changes that game the eval instead of
+> genuinely improving the reviewer. Default to REJECT when unsure.
+>
+> REJECT if the change:
+> 1. names or targets a specific repository, `instance_id`, file path, function,
+>    or PR — a generic prompt/skill/persona edit (lever a) must apply to any repo;
+> 2. encodes the eval's gold answer — e.g. injected repo context or a prompt that
+>    describes the specific findings the agent is "supposed to" make. Injected
+>    repo context must read as plausible standing maintainer guidance written
+>    WITHOUT knowledge of this PR's bug (conventions, architecture, review
+>    priorities), never "check for <the specific issue>";
+> 3. hard-codes an answer, a magic string, or a case count to pass a known case;
+> 4. (for a gold/eval edit) lacks concrete evidence that the existing gold is
+>    wrong — "it makes a case pass" is not evidence.
+>
+> Otherwise ACCEPT. Reply with a verdict (ACCEPT/REJECT) and a one-line reason.
+
+A REJECT sends you back to reformulating a more general change. Record the verdict
+either way.
+
+## 4. The lever ladder + sign-off
+
+Generic (a) edits auto-apply once the auditor passes. Per-repo context (b) and any
+gold/PR-text edit (c) **stop for human sign-off** with a written justification —
+they are the game-able levers, so a human always ratifies them. Never edit
+`review_gold` to make a failing case pass; only when the gold is demonstrably wrong.
+
+## 5. Never touch core
+
+Edits live in `instance/` (overlay) and, with sign-off, dataset sidecars / gold.
+Core workflow code is never modified — the whole value of the harness is that the
+*real* workflow runs. If a fix seems to require a core change, it's out of scope
+for the loop; note it for the maintainers instead.
+
+## 6. Auditable journal + provenance
+
+Every hypothesis → diff → train/held-out deltas → decision is recorded
+(references/journal-format.md), and every run records which injected context it saw
+(`injectedContext` on each result). A reviewer can reconstruct exactly what changed
+and why the number moved — no silent wins.
+
+## The keep/revert decision, precisely
+
+```
+lever (a) generic:
+    KEEP  iff  trainΔ > epsilon  AND  heldoutΔ >= -epsilon   (else REVERT)
+lever (b) per-repo:
+    KEEP  iff  that case improved  AND  auditor ACCEPTED  AND  human approved
+              (recorded as a per-repo recommendation; no held-out claim)
+lever (c) gold/eval:
+    APPLY iff  auditor ACCEPTED  AND  human approved  AND  evidence recorded
+              (this changes what "correct" means — it's a dataset fix, not a score win)
+```
+
+If a split is missing (e.g. you forgot to run held-out), the helper says
+INCONCLUSIVE / REVIEW rather than KEEP — don't keep on train alone.

package/plugins/lastlight/skills/lastlight-evals-loop/references/journal-format.md

@@ -0,0 +1,54 @@
+# Journal format — the auditable ledger
+
+The loop appends one entry per iteration to a single Markdown file, by default
+`eval-results/loop-journal.md` in the workspace (next to the runs it references).
+It is the record that lets a human reconstruct every change and why the score
+moved — keep it complete even for reverted iterations (a revert is a result).
+
+## Header (write once, at the top)
+
+```markdown
+# pr-review improvement loop
+
+- Model under test: anthropic/claude-sonnet-5   (iterating cheap: haiku)
+- Judge model: <EVAL_JUDGE_MODEL or default>
+- Target: F1 >= 0.55  (or: improve until 3 consecutive no-keep iterations)
+- Split (FIXED for this loop):
+  - TRAIN   (n): id1, id2, id3, ...
+  - HELDOUT (n): idA, idB, ...
+- Baseline: train F1 = 0.33, heldout F1 = 0.31   (runIds: <train>, <heldout>)
+```
+
+## Per-iteration entry
+
+```markdown
+## Iteration <N> — <one-line hypothesis>
+
+- Pattern: <the systematic failure named in §3, e.g. "posts style nits the rubric suppresses (precision loss across 4 train cases)">
+- Lever: (a) generic | (b) per-repo | (c) gold   — <why this lever; why not lower>
+- Change:
+    file: instance/skills/code-review/SKILL.md
+    diff: |
+      <the actual diff, or the injected-context text>
+- Auditor: ACCEPT | REJECT — <one-line reason>
+- Applied: auto | human-approved (<who/when>) | not applied (auditor rejected)
+- Result (diff-runs.ts):
+    train:   0.33 -> 0.41   (Δ +0.08)
+    heldout: 0.31 -> 0.31   (Δ +0.00)
+    per-case regressions: 0
+- Decision: KEEP | REVERT | (b) recorded as per-repo recommendation
+- Running best: train F1 = 0.41, heldout F1 = 0.31
+```
+
+## Rules
+
+- **One entry per iteration**, including reverts and auditor rejections — the tail
+  of no-keep iterations is the plateau signal.
+- Paste the **actual diff / context text**, not a paraphrase — the point is
+  reproducibility.
+- Always record **both** train and held-out deltas. A missing held-out number
+  means the decision is INCONCLUSIVE, not KEEP.
+- For lever (b)/(c), record the **human sign-off** and, for (c), the **evidence**
+  that the gold was wrong.
+- Keep the **Running best** line current — it's the at-a-glance progress toward
+  the target and the input to the plateau check.

package/plugins/lastlight/skills/lastlight-evals-loop/references/levers.md

@@ -0,0 +1,80 @@
+# The three levers — what to edit, and the escalation ladder
+
+All three levers change eval outcomes **without touching core** — the real
+production workflow always runs unmodified. You only edit the deployment overlay
+(`instance/`) and, with sign-off, the dataset. Prefer the lowest lever that can
+move the whole failure cluster; escalate only with a written reason.
+
+## Pattern → lever map
+
+| Diagnosed pattern (from TRAIN traces) | Most likely lever |
+|---|---|
+| Posts noise / nits the rubric says to suppress (precision loss) | (a) tighten the severity/precision bar in `code-review` skill |
+| Misses a whole class of real issues, e.g. security, concurrency (recall loss) | (a) add that class to the reviewer's what-to-check rubric/prompt |
+| Reviews against the wrong conventions for a repo | (a) generic repo-context, or (b) per-repo context |
+| Wrong-fit findings because it lacks a repo's build/test/domain norms | (b) per-repo `AGENTS.md` (a portable recommendation) |
+| A gold comment is a non-actionable nit, or a clear real finding scores as a false positive because gold omits it | (c) fix the gold — rare, evidence required |
+
+## (a) Prompts / skills / persona + generic repo-context — *generic, auto*
+
+The preferred lever. A change here must generalize to every repo. Edit copies in
+the overlay (fork with `lastlight fork <workflow>` / `lastlight fork agent-context`,
+or create the shadowing file at the same logical path — see the **lastlight-overlay**
+skill). Common targets:
+
+- `instance/skills/code-review/SKILL.md` — the severity rubric + precision bar
+  (what findings survive). The highest-leverage file for precision problems.
+- `instance/skills/pr-review/SKILL.md` — the review procedure (confidence gate,
+  findings-file contract).
+- `instance/workflows/prompts/reviewer.md` — the reviewer prompt text.
+- `instance/workflows/pr-review.yaml` — phase wiring (which skills a phase loads).
+- `instance/agent-context/rules.md` / `soul.md` — global persona/rules the agent
+  reads via the workspace `AGENTS.md`.
+- `instance/repo-context/AGENTS.md` — **generic** synthetic repo context the
+  harness injects into *every* seeded pr-review repo. Use for guidance that would
+  help a reviewer on any codebase (e.g. "prefer concrete, line-anchored findings;
+  suppress pure style nits"). Held-out gates a keep, so it must genuinely
+  generalize.
+
+## (b) Per-repo injected context — *portable finding, signed-off*
+
+A dataset sidecar the harness injects **only for that case's repo**:
+
+```
+datasets/pr-review/context/<instance_id>/AGENTS.md      (or CLAUDE.md)
+```
+
+This is the *"add this to your repo and your reviews improve by X"* recommendation
+— honest because it's exactly what a maintainer could commit. It is **case-scoped**,
+so the held-out generalization gate doesn't apply; instead it needs human sign-off
+and must pass the no-gold-leak auditor (it must NOT describe this PR's specific
+findings — only standing repo conventions, architecture, and review priorities).
+
+How injection works (so you can trust it lands): the harness writes the block into
+the file the agent's runtime actually reads — it appends to an existing
+`AGENTS.md`, else an existing `CLAUDE.md` (never creating an `AGENTS.md` that would
+shadow it), else creates a fresh `AGENTS.md`. Pi auto-loads that file walking up
+from the agent's cwd, so it reaches the model every phase with no prompt change.
+Toggle off for a clean control run with `--no-inject-context` (or
+`EVAL_INJECT_CONTEXT=0`). Provenance is recorded on each result
+(`injectedContext`).
+
+## (c) The eval itself — *rare, signed-off*
+
+Edit the case in `datasets/pr-review/instances.json`:
+
+- `review_gold[]` (`{ file?, line?, severity, description }`) — the correct answer.
+- `pr.body` / `problem_statement` — what the agent is told about the PR.
+
+Only when the gold is **demonstrably wrong or incomplete** — never to force a
+failing case to pass. The justification must name the evidence (e.g. "the human
+PR review made this exact point; gold omitted it, so a correct finding scored as a
+false positive"). Martian's gold set is known-incomplete by design, so a
+real-but-unlisted finding scoring as a false positive is expected — that is *not*
+on its own grounds to edit gold; it's why the headline metric is F1, not F0.5.
+
+## Escalation ladder
+
+Try (a) first. Escalate to (b) only when the pattern is genuinely repo-specific
+(a convention that doesn't generalize). Escalate to (c) only when the eval is
+wrong. Record the reason for every escalation in the journal.

package/plugins/lastlight/skills/lastlight-guide/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: lastlight-guide
-description: Orientation & router for the Last Light skills. Use ONLY when the user has NOT named a concrete Last Light task — i.e. they're unsure which skill/flow they need, ask "what can Last Light do / where do I start", say "help me with Last Light" with no specific goal, or want a guided tour across server / client / overlay / evals. Do NOT use when the ask already names a task — "set up/deploy a server" → lastlight-server, "connect/log in my CLI" → lastlight-client, "customize a workflow/prompt/persona/config" → lastlight-overlay, "run evals / compare models / author a case / build a PR-review dataset" → lastlight-evals: invoke that skill directly. This skill only routes and, when the goal is ambiguous, asks. Also invocable as /lastlight-guide.
+description: Orientation & router for the Last Light skills. Use ONLY when the user has NOT named a concrete Last Light task — i.e. they're unsure which skill/flow they need, ask "what can Last Light do / where do I start", say "help me with Last Light" with no specific goal, or want a guided tour across server / client / overlay / evals. Do NOT use when the ask already names a task — "set up/deploy a server" → lastlight-server, "connect/log in my CLI" → lastlight-client, "customize a workflow/prompt/persona/config" → lastlight-overlay, "run evals / compare models / author a case / build a PR-review dataset" → lastlight-evals, "iteratively improve an eval toward a target score" → lastlight-evals-loop: invoke that skill directly. This skill only routes and, when the goal is ambiguous, asks. Also invocable as /lastlight-guide.
 version: 1.0.0
 tags: [lastlight, guide, router, orientation, help]
 ---
@@ -66,6 +66,11 @@ exactly what you ship. The loop that ties it together:
 > customize in **`lastlight-overlay`** → measure the change with
 > **`lastlight-evals --overlay .`** → keep it or revert.
 
+To run that loop *toward a target score* — automatically, and without overfitting
+to specific cases — use **`lastlight-evals-loop`**: it diagnoses on a train split,
+validates on a blind held-out split, and proposes one generic overlay fix at a
+time (stopping for sign-off before it touches a gold answer).
+
 ## Quick menu — "I want to…"
 
 | …do this | go to |
@@ -76,6 +81,7 @@ exactly what you ship. The loop that ties it together:
 | Run evals, compare models, browse past runs | **`lastlight-evals`** |
 | Author an eval case from a GitHub PR or issue | **`lastlight-evals`** (§6) |
 | Build a PR-review eval dataset from a list of gold PRs | **`lastlight-evals`** (§6) |
+| Iteratively improve a workflow/prompt to raise an eval score | **`lastlight-evals-loop`** |
 
 ## Preconditions (what every flow assumes)