lastlight 0.9.0 → 0.10.1

package/plugins/lastlight/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin.json",
   "name": "lastlight",
   "displayName": "Last Light",
-  "version": "0.9.0",
+  "version": "0.10.1",
   "description": "Install, configure and operate Last Light (GitHub maintenance agent) — its server, CLI client, deployment overlay, and the Last Light Evals harness.",
   "author": {
     "name": "Clifton Cunningham"

package/plugins/lastlight/skills/lastlight-evals-loop/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: lastlight-evals-loop
-description: Drive a Last Light EVAL toward a target score with a disciplined, anti-gaming improvement loop — run → diagnose → propose ONE generic fix → re-measure → keep or revert → repeat. Use when the user wants to "improve / raise the pr-review F1", "make the reviewer better against the eval", "close the loop on evals", "iterate on prompts/skills to pass more cases", or "tune the workflow to hit a score target". The loop diagnoses on a TRAIN split and validates on a BLIND held-out split so fixes must generalize, not overfit; it prefers generic overlay prompt/skill edits, and stops for human sign-off before editing any gold answer. For a one-off run/compare use lastlight-evals; to hand-fork a workflow/prompt use lastlight-overlay. Needs an already-scaffolded evals workspace (lastlight-evals) with a pr-review dataset.
-version: 1.0.0
+description: Drive a Last Light EVAL toward a target score with a disciplined, anti-gaming improvement loop — run → mine failures → propose candidate fix(es) → re-measure → keep the best or revert → repeat. Use when the user wants to "improve / raise the pr-review F1", "make the reviewer better against the eval", "close the loop on evals", "iterate on prompts/skills to pass more cases", or "tune the workflow to hit a score target". The loop diagnoses on a TRAIN split and validates on a BLIND held-out split so fixes must generalize, not overfit; it prefers generic overlay prompt/skill edits, and stops for human sign-off before editing any gold answer. For a one-off run/compare use lastlight-evals; to hand-fork a workflow/prompt use lastlight-overlay. Needs an already-scaffolded evals workspace (lastlight-evals) with a pr-review dataset.
+version: 1.1.0
 tags: [lastlight, evals, loop, improvement, overfitting, pr-review]
 ---
 
@@ -16,6 +16,11 @@ That lifts the number and ruins the reviewer. Every guardrail here exists to for
 Scope: the **pr-review** tier (its judge trace gives you the agent-vs-gold detail
 diagnosis needs). The pattern extends to triage/code-fix, but start here.
 
+The method — **weakness-mine → propose a few minimal candidates → keep the one
+that survives a blind held-out gate** — follows *Self-Harness: Harnesses That
+Improve Themselves* ([arXiv:2606.09498](https://arxiv.org/abs/2606.09498)), adapted
+to keep our anti-gaming discipline. See **references/approach.md** for the mapping.
+
 ## Prerequisites
 
 - A scaffolded evals workspace with a **pr-review** dataset and a working provider
@@ -23,22 +28,25 @@ diagnosis needs). The pattern extends to triage/code-fix, but start here.
   bare `lastlight-evals run pr-review --limit 1 --no-open` grades a case.
 - An **`instance/` overlay** (the deployment overlay) to receive generic edits —
   see **`lastlight-overlay`**. The loop edits copies here, never core.
-- The `diff-runs.ts` helper ships with the evals package
-  (`scripts/diff-runs.ts`) — it compares two runs and calls keep/revert.
+- Two helpers ship with the evals package: **`scripts/mine-failures.ts`** (ranks
+  the TRAIN failure signatures into an evidence bundle — the diagnosis input) and
+  **`scripts/diff-runs.ts`** (compares two runs and calls keep/revert).
 
 ## The one rule
 
-**One change per iteration.** Propose a single hypothesis, measure its effect in
-isolation, keep or revert, then move on. No batching — batching hides which edit
-moved the number and invites scattershot overfitting.
+**One change *kept* per round.** A round may *explore* a few minimal candidates,
+but at most one is ever kept and committed to `instance/` — the rest are reverted.
+So which edit moved the number is always attributable, and no overfit edit rides
+along with a good one. Never keep two edits from one round without re-measuring
+each in isolation.
 
 ## Start here
 
 | You have… | Go to |
 |---|---|
 | Never run this loop on this dataset | **§1 Set up the split** then §2 |
-| A split + baseline already recorded | **§3 Diagnose** |
-| A proposed change ready to test | **§5 Audit → §6 Apply → §7 Measure → §8 Decide** |
+| A split + baseline already recorded | **§3 Diagnose (mine failures)** |
+| Candidate change(s) ready to test | **§5 Audit → §6 Apply → §7 Measure → §8 Decide** |
 
 ## 1. Set up the split (once per dataset)
 
@@ -54,7 +62,7 @@ lastlight-evals run pr-review --instance <all-ids> --no-open   # or omit --insta
 ```
 
 Set a **target**: an F1 goal (e.g. 0.55) or "improve until plateau" (N=3
-consecutive no-keep iterations).
+consecutive no-keep rounds).
 
 ## 2. Baseline
 
@@ -69,23 +77,41 @@ lastlight-evals run pr-review --overlay instance --instance <heldout-ids> --no-o
 
 Each writes `eval-results/pr-review/<runId>/scorecard.json`. Note the two runIds.
 
-## 3. Diagnose (TRAIN only)
-
-Read the TRAIN scorecard's `results[].review` and `review.trace`. **Do not open
-held-out traces.** For each failing case:
-- `falseNegatives` (with `severity`) = **recall** loss — real issues the agent missed.
-- `falsePositives` = **precision** loss — noise the agent posted that matched no gold.
+## 3. Diagnose (TRAIN only) — mine the failure signatures
 
-Cluster across cases and name the **systematic pattern** — not a one-off. Good
-patterns generalize: *"misses security-relevant findings"*, *"posts style nits the
-rubric says to suppress"*, *"confidence bar too low → noise"*, *"no awareness of the
-repo's conventions → wrong-fit findings"*. See **references/levers.md** for how
-each pattern maps to an edit.
+Mine the TRAIN scorecard into a ranked **evidence bundle** — recall-loss and
+precision-loss signatures ordered by impact — instead of reading traces by hand:
 
-## 4. Hypothesis — ONE change, lowest lever first
+```bash
+npx tsx scripts/mine-failures.ts <train-scorecard.json> --train <train-ids> --keywords
+```
 
-Pick the **lowest** lever that could move the whole cluster (full detail in
-**references/levers.md**):
+It reads only the TRAIN split (never held-out) and prints two ranked blocks:
+- **RECALL LOSS** — missed gold (`falseNegatives`, weighted by `severity`): the
+  real issues the agent missed. Biggest F1 headroom is at the top.
+- **PRECISION LOSS** — noise (`falsePositives`): findings that matched no gold.
+
+Each signature is `axis·severity·area` with a frequency, case count, example ids,
+and (with `--keywords`) a heuristic category guess. Take the **top systematic
+pattern** — not a one-off — and name it: *"misses security-relevant findings"*,
+*"posts style nits the rubric says to suppress"*, *"confidence bar too low →
+noise"*, *"no awareness of the repo's conventions → wrong-fit findings"*. See
+**references/levers.md** for how each pattern maps to an edit. (You can still open
+individual TRAIN `review.trace`s to read a signature's underlying findings — just
+never a held-out trace.)
+
+## 4. Propose candidates — a few minimal, diverse edits
+
+For the top signature, draft **K = 2–4 candidate edits**, each **minimal** (touches
+only what the pattern needs) and each targeting a *different* mechanism or a
+different formulation of the same one — e.g. one tightens the severity bar in the
+`code-review` skill, another adds a confidence gate in the reviewer prompt. K stays
+small on purpose: selecting the best of many candidates optimistically inflates the
+train number, so keep the field narrow and let the held-out gate (§7b) be the check.
+Drafting one candidate is fine — best-of-K is an option, not an obligation.
+
+Every candidate uses the **lowest** lever that could move the cluster (full detail
+in **references/levers.md**):
 
 - **(a) generic — auto.** Edit an overlay prompt/skill/persona, or add a generic
   `instance/repo-context/AGENTS.md` injected into every repo. Must be general.
@@ -95,55 +121,80 @@ Pick the **lowest** lever that could move the whole cluster (full detail in
 - **(c) the eval itself — rare, signed-off.** Edit `review_gold[]` ONLY when the
   gold is demonstrably wrong/incomplete; name the evidence. Human approves.
 
-Escalate only when a lower lever genuinely can't address the pattern, and write
-down why.
+Escalate a candidate only when a lower lever genuinely can't address the pattern,
+and write down why.
 
-## 5. Audit the change (adversarial sub-agent)
+## 5. Audit each candidate (adversarial sub-agent)
 
-Before applying, spawn a **generality + leak auditor** sub-agent (prompt template
-in **references/guardrails.md**). It **REJECTS** if the change:
+Before measuring, run the **generality + leak auditor** sub-agent (prompt template
+in **references/guardrails.md**) on **every** candidate. It **REJECTS** if the change:
 - names a specific repo / `instance_id` / file path (lever a must be generic), or
 - **encodes the gold answer** — injected repo context that describes the specific
   findings is cheating. It must read as plausible maintainer guidance written
   *without* knowledge of this PR's bug (conventions, architecture, review
   priorities), never "look for bug X".
 
-Rejected → back to §4.
+Rejected candidates are dropped (not counted); survivors go to §6. If none survive,
+back to §4.
 
 ## 6. Apply (hybrid autonomy)
 
+Apply each surviving candidate on its **own throwaway branch** (or stash between
+candidates) so their §7a measurements don't cross-contaminate — only the winner is
+committed in §8.
+
 - **Lever (a)** (incl. generic `repo-context/AGENTS.md`): apply automatically to
   `instance/…` once the auditor passes.
 - **Lever (b)/(c)**: **STOP.** Present the diff + the written justification and ask
   the human to approve before writing to the dataset.
 
-## 7. Re-measure
+## 7. Re-measure — train selects, held-out confirms **once**
+
+The blind held-out set must be consumed **once per round**, not once per candidate —
+gating every candidate on held-out and keeping the best would inflate it (max-of-K
+selection bias). So split measurement in two:
+
+**7a. Rank candidates on TRAIN only.** Run each survivor on the TRAIN split and diff
+*without* `--heldout` (the verdict prints `REVIEW — no held-out`, expected here):
+
+```bash
+npx tsx scripts/diff-runs.ts <baseline-train.json> <candidate-train.json> --train <train-ids>
+```
+
+Pick **one winner**: highest train Δ → tie-break lowest lever → smallest diff →
+fewest per-case train regressions. **Do not touch held-out in 7a.**
 
-Re-run both splits (§2) with the change, then diff against the baseline:
+**7b. Confirm the winner on HELD-OUT, once.** Run the winner on both splits and diff
+with the full gate:
 
 ```bash
-npx tsx scripts/diff-runs.ts <baseline-train.json> <candidate-train.json> \
+npx tsx scripts/diff-runs.ts <baseline.json> <winner.json> \
     --train <train-ids> --heldout <heldout-ids>
 ```
 
-Point it at the run pair that covers both splits (or run the two splits in one
-`--instance <all-ids>` run so a single scorecard holds both). It prints per-case
-F1 deltas, the arm-summary delta, and a **KEEP / REVERT verdict**.
+(Run the two splits in one `--instance <all-ids>` run so a single scorecard holds
+both.) If a candidate may legitimately be held-out-driven with flat train, add
+`--symmetric` for the paper's non-regressive gate (KEEP iff neither split regresses
+and one improves); record which gate you used. With a single candidate, 7a and 7b
+collapse into one gated run.
 
 ## 8. Decide
 
-- **KEEP** iff train F1 improved **and** held-out did **not** regress (the helper's
-  verdict). Train ↑ with held-out ↓ = **overfit → revert**.
+- **KEEP** the winner iff its train F1 improved **and** held-out did **not** regress
+  (the helper's verdict). Train ↑ with held-out ↓ = **overfit → revert**.
 - **Lever (b)** per-repo edits are case-scoped: keep iff that case improved, the
   auditor passed, and the human approved — recorded as a per-repo recommendation,
   not a workflow change (no held-out claim is made for it).
-- Else **REVERT**: `git checkout` the overlay file, or remove the sidecar.
+- **Revert the K−1 losers** and, on a REVERT verdict, the winner too: `git checkout`
+  the overlay file, or remove the sidecar. Exactly one edit (or none) survives the
+  round — that is the "one change kept per round" rule.
 
 ## 9. Journal, then repeat
 
-Append one entry per iteration (schema in **references/journal-format.md**):
-pattern, lever, the diff/context, auditor verdict, train Δ, held-out Δ, decision,
-running best. Then return to §3. Stop at the target or the plateau.
+Append one entry per **round** (schema in **references/journal-format.md**): the
+mined pattern, the K candidates with each one's train Δ, which won, the winner's
+single held-out Δ, the decision, and the running best. Then return to §3. Stop at
+the target or the plateau (N=3 consecutive no-keep rounds).
 
 Run the loop on a cadence with the built-in `/loop` if you want it to keep going
 unattended between your check-ins — but the lever-(b)/(c) sign-off stops still

package/plugins/lastlight/skills/lastlight-evals-loop/references/approach.md

@@ -0,0 +1,47 @@
+# Approach & prior art — why the loop looks like this
+
+This loop is a **self-improving harness**: it edits the reviewing agent's harness
+(prompts, skills, persona, injected repo-context) — never its weights and never the
+core workflow — and keeps only edits that a blind held-out split confirms.
+
+## Prior art: *Self-Harness*
+
+The structure follows ***Self-Harness: Harnesses That Improve Themselves***
+([arXiv:2606.09498](https://arxiv.org/abs/2606.09498)), which runs a three-stage
+cycle on a fixed model: **mine weaknesses → propose K minimal candidate edits →
+validate on held-in *and* held-out under a non-regressive gate**, keeping only edits
+that don't degrade either split. Its notable result: the retained edits are
+concrete and evidence-grounded, "not generic instruction padding."
+
+We adopt the parts that make our loop better-grounded and better-explored, and
+deliberately *diverge* where our anti-gaming constraints are stricter than the
+paper's.
+
+| Self-Harness stage | Here | Where |
+|---|---|---|
+| Weakness mining (cluster failing traces by signature, ranked by impact) | `scripts/mine-failures.ts` → ranked recall/precision signature bundle from TRAIN traces | SKILL §3 |
+| K diverse minimal candidate edits | Draft K=2–4 minimal candidates, each a different mechanism/formulation | SKILL §4 |
+| Validate on both splits, non-regressive gate | **TRAIN selects, HELD-OUT confirms once**; default asymmetric gate, opt-in `--symmetric` for the paper's rule | SKILL §7, `diff-runs.ts` |
+| Document each transition | The per-round journal (all candidates + winner + held-out Δ) | `journal-format.md` |
+
+## Where we diverge from the paper (on purpose)
+
+- **Held-out is consumed once per round, not once per candidate.** The paper gates
+  each candidate on held-out. Because our loop can *see* the gold set, screening K
+  candidates on the blind split and keeping the best would inflate it (max-of-K
+  selection bias). So we rank on TRAIN only and give the single winner one held-out
+  confirmation. See guardrails §2.
+- **Generic-first, not model-specific.** The paper found value in *model-specific*
+  harness edits. Our default lever must generalize across **repos** (the held-out
+  split is a repo split), so we prefer generic overlay edits and gate them on
+  held-out. A model-specific-but-repo-generic lever is a possible future escalation,
+  but it is **not** part of this loop today — don't reach for it.
+- **A leak/generality auditor gates every candidate.** The paper relies on the
+  held-out gate alone; we add an adversarial auditor (guardrails §3) because a loop
+  that can read the gold answer can encode it — a failure mode the paper's setup
+  doesn't face.
+- **Human sign-off on the game-able levers.** Per-repo context (b) and gold edits
+  (c) are case-scoped with no held-out claim, so a human ratifies them.
+
+The net: same mine → propose → validate skeleton, hardened for a loop that can see
+the answers it is being graded against.

package/plugins/lastlight/skills/lastlight-evals-loop/references/guardrails.md

@@ -15,10 +15,18 @@ not override it by eyeballing train alone.
 The split ids are fixed for the whole loop. If you re-split (e.g. to add cases),
 say so in the journal and treat prior held-out results as no longer comparable.
 
-## 2. One change per iteration
+## 2. One change *kept* per round
 
 Isolates causal effect. If two edits ship together and the number moves, you can't
-attribute it — and you can't tell an overfit edit riding along with a good one.
+attribute it — and you can't tell an overfit edit riding along with a good one. A
+round may *explore* K=2–4 minimal candidates, but each is measured on its own
+branch and at most one is kept.
+
+**Best-of-K must not consume the blind split K times.** Selecting the best of many
+candidates on held-out inflates it (max-of-K selection bias). So candidates are
+ranked on **TRAIN only**; the single winner gets **one** held-out confirmation.
+Keep K small — the more candidates you screen on train, the more optimistic the
+winner's train number, and only the one held-out run keeps it honest.
 
 ## 3. The generality + leak auditor (adversarial sub-agent)
 
@@ -72,8 +80,14 @@ and why the number moved — no silent wins.
 ## The keep/revert decision, precisely
 
 ```
+best-of-K selection (per round, before the gate below):
+    rank surviving candidates on TRAIN ONLY  ->  winner = highest trainΔ
+    (tie-break: lowest lever, smallest diff, fewest train per-case regressions)
+    then apply the gate below to the WINNER, consuming held-out ONCE.
 lever (a) generic:
     KEEP  iff  trainΔ > epsilon  AND  heldoutΔ >= -epsilon   (else REVERT)
+    (--symmetric variant, for a held-out-driven candidate:
+        KEEP iff trainΔ >= -epsilon AND heldoutΔ >= -epsilon AND max(both) > epsilon)
 lever (b) per-repo:
     KEEP  iff  that case improved  AND  auditor ACCEPTED  AND  human approved
               (recorded as a per-repo recommendation; no held-out claim)

package/plugins/lastlight/skills/lastlight-evals-loop/references/journal-format.md

@@ -19,31 +19,42 @@ moved — keep it complete even for reverted iterations (a revert is a result).
 - Baseline: train F1 = 0.33, heldout F1 = 0.31   (runIds: <train>, <heldout>)
 ```
 
-## Per-iteration entry
+## Per-round entry
+
+A round mines one pattern, explores K candidates (ranked on TRAIN), and keeps at
+most one (confirmed on HELD-OUT once). A single-candidate round is just K=1.
 
 ```markdown
-## Iteration <N> — <one-line hypothesis>
+## Round <N> — <one-line hypothesis for the top pattern>
 
-- Pattern: <the systematic failure named in §3, e.g. "posts style nits the rubric suppresses (precision loss across 4 train cases)">
-- Lever: (a) generic | (b) per-repo | (c) gold   — <why this lever; why not lower>
-- Change:
+- Pattern: <the systematic failure named in §3, e.g. "misses security-relevant findings (recall·high, 4 train cases)"; from mine-failures.ts>
+- Candidates (ranked on TRAIN only):
+  | # | lever | change (file) | trainΔ | auditor |
+  |---|-------|---------------|--------|---------|
+  | 1 | (a)   | instance/skills/code-review/SKILL.md | +0.08 | ACCEPT |
+  | 2 | (a)   | instance/workflows/prompts/reviewer.md | +0.03 | ACCEPT |
+  | 3 | (a)   | instance/repo-context/AGENTS.md | -0.01 | REJECT (leak) |
+- Winner: #1 — <why (highest trainΔ; tie-break rule if used)>
+- Winner change:
     file: instance/skills/code-review/SKILL.md
     diff: |
       <the actual diff, or the injected-context text>
-- Auditor: ACCEPT | REJECT — <one-line reason>
 - Applied: auto | human-approved (<who/when>) | not applied (auditor rejected)
-- Result (diff-runs.ts):
+- Held-out confirmation (diff-runs.ts, winner only — held-out consumed ONCE):
+    gate:    default | --symmetric
     train:   0.33 -> 0.41   (Δ +0.08)
     heldout: 0.31 -> 0.31   (Δ +0.00)
-    per-case regressions: 0
+    per-case regressions: 0   (REGRESSED(train): none)
 - Decision: KEEP | REVERT | (b) recorded as per-repo recommendation
+- Reverted losers: #2, #3
 - Running best: train F1 = 0.41, heldout F1 = 0.31
 ```
 
 ## Rules
 
-- **One entry per iteration**, including reverts and auditor rejections — the tail
-  of no-keep iterations is the plateau signal.
+- **One entry per round**, including all-revert rounds and auditor rejections — the
+  tail of no-keep rounds is the plateau signal. Record every candidate's train Δ,
+  not just the winner's, so the best-of-K selection is auditable.
 - Paste the **actual diff / context text**, not a paraphrase — the point is
   reproducibility.
 - Always record **both** train and held-out deltas. A missing held-out number