lastlight 0.7.1 → 0.7.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lastlight",
-  "version": "0.7.1",
+  "version": "0.7.2",
   "description": "GitHub repository maintenance agent — Agent SDK harness",
   "type": "module",
   "main": "dist/index.js",

package/plugins/lastlight/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin.json",
   "name": "lastlight",
   "displayName": "Last Light",
-  "version": "0.7.1",
+  "version": "0.7.2",
   "description": "Install, configure and operate Last Light (GitHub maintenance agent) — its server, CLI client, deployment overlay, and the Last Light Evals harness.",
   "author": { "name": "Clifton Cunningham" },
   "homepage": "https://github.com/cliftonc/lastlight",

package/plugins/lastlight/skills/lastlight-evals/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: lastlight-evals
-description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases (triage / code-fix instances). GitHub is mocked, so no real GitHub token is needed — only a model provider API key.
-version: 1.1.0
+description: Scaffold, configure and run a Last Light EVALS workspace — the harness that runs Last Light's real workflows against a mocked GitHub and grades them deterministically. Use when the user wants to "set up / scaffold Last Light Evals", "create an evals workspace or instance", "run evals", "compare models", or author new eval cases — including "create an eval dataset/case from this PR/issue <github url>" (triage / code-fix instances). GitHub is mocked when running, so no real GitHub token is needed — only a model provider API key (the one exception is `add-case`, which reads real PRs/issues via `gh`).
+version: 1.2.0
 tags: [lastlight, evals, benchmark, models, swe-bench]
 ---
 
@@ -121,6 +121,34 @@ Quick shape (paths are relative to the workspace's `evals/` dir):
   `instances.json` (+ `repos/` & `tests/` for code-fix-style tiers). Discovery is
   automatic — no code change.
 
+## 6. Author a case from a real GitHub PR or issue
+
+When the user says **"create an eval dataset/case from this PR/issue <url>"**, use
+the `add-case` subcommand — it does the mechanical extraction; you refine the
+fuzzy parts.
+
+```bash
+lastlight-evals add-case --pr <github-pr-url> --dry-run        # propose a code-fix case; don't write
+lastlight-evals add-case --pr <github-pr-url>                  # write into ./datasets (or --datasets/--overlay)
+lastlight-evals add-case --issue <github-issue-url> --dry-run  # propose a triage case
+```
+
+- **From a PR (code-fix):** derives `repo`, `base_commit` (merge-base of base &
+  head) + `head_commit`, the PR's **test** diff as the held-out `test_patch`, and
+  auto-detects `FAIL_TO_PASS` / `PASS_TO_PASS` by running the tests at base (red)
+  vs head (green). Produces a **git-source** case (no `repos/<id>/` vendored): at
+  run time the harness clones the repo into the gitignored `./.eval-cache/` and
+  checks out `base_commit`.
+- **From an issue (triage):** derives the `issue` seed + `problem_statement`, the
+  **labels that were applied** (from the issue events API, with *who* applied each)
+  → `expect_github.labels_added`, `issue_closed` if it was closed, and prints the
+  **reviewer comments** — the raw triage signal. You then assign
+  `triage_gold` (category/state) from those labels per the deployment's taxonomy.
+
+The full flow, what you refine for each, the `test_cmd` / `setup_cmd` options for
+non-`node --test` runners, and the trust/offline caveats are in
+**`references/authoring-from-pr.md`** — read it before authoring.
+
 ## Done when
 
 The workspace is scaffolded (Separate: overlay in `instance/`, evals at root;

package/plugins/lastlight/skills/lastlight-evals/references/authoring-from-pr.md

@@ -0,0 +1,117 @@
+# Author an eval case from a real GitHub PR or issue
+
+`lastlight-evals add-case` turns a real GitHub **PR** into a code-fix (build) case
+or an **issue** into a triage case. It does the mechanical, reproducible
+extraction with `gh` + `git`; **you** refine the judgement parts. The result is a
+**git-source** case — no fixture repo is vendored; at run time the harness clones
+the repo into the gitignored `./.eval-cache/` and checks out `base_commit` (see
+the `instance-schema.md` "Git-source" flavor).
+
+> **Trust + network.** Validation and the run-time checkout execute the repo's own
+> code (`setup_cmd` / `test_cmd` / its tests). Only point this at repos you trust.
+> The first run per repo fetches over the network; after that the cache makes it
+> offline.
+
+## Prerequisites
+
+- `gh` on PATH and authenticated (`gh auth login`) — used to read PR/issue metadata.
+- `git`. Node 24+ (same as the rest of the harness).
+
+## Command
+
+```bash
+lastlight-evals add-case --pr <github-pr-url> [options]
+lastlight-evals add-case --issue <github-issue-url> [options]
+```
+
+Options:
+
+| flag | meaning |
+|---|---|
+| `--pr <url>` | a GitHub PR url → a **code-fix** (build) case |
+| `--issue <url>` | a GitHub issue url → a **triage** case |
+| `--tier <name>` | target tier dir (default `code-fix` for `--pr`, `triage` for `--issue`) |
+| `--id <slug>` | `instance_id` (default derived from repo + number) |
+| `--datasets <dir>` | datasets root to write into (a `<tier>/` subdir). Default `./datasets`, else `./evals/datasets` |
+| `--overlay <dir>` | write into `<dir>/evals/datasets` instead |
+| `--test-cmd "<cmd>"` | held-out test command (default `node --test`); stored as `test_cmd` |
+| `--setup-cmd "<cmd>"` | install/build run before tests (e.g. `"npm ci"`); stored as `setup_cmd` |
+| `--no-validate` | don't run the repo's tests to auto-detect `FAIL_TO_PASS` (just scaffold) |
+| `--dry-run` | print the proposed instance JSON; don't write |
+
+## The recommended flow (CLI extracts → you refine)
+
+1. **Dry-run first.** `add-case --pr <url> --dry-run` prints the proposed instance.
+   The CLI derives:
+   - `repo`, `base_commit` (the **merge-base** of the base branch and the PR head —
+     the true fork point, not the base-branch tip), and `head_commit`;
+   - `test_patch` — the diff of the PR's **test** files (path heuristic: `test/`,
+     `tests/`, `__tests__/`, `spec/`, or `*.test.*` / `*.spec.*` / `*_test.*`);
+   - gold `patch` — the diff of the non-test files (reference only, never graded);
+   - `FAIL_TO_PASS` / `PASS_TO_PASS` — auto-detected by running the tests at base
+     (with `test_patch` applied → expect red) then at head (→ expect green), unless
+     `--no-validate`;
+   - `issue` + `problem_statement` (the PR's linked issue if it closes one, else the
+     PR title/body) and `expect_github: { pr_opened: { base, head_is_branch } }`.
+
+2. **Review and repair** what the CLI can't get right on its own:
+   - **Held-out tests.** If the heuristic mislabeled files (warns "No test files
+     detected", or grabbed a non-test), fix `test_patch` by hand, or rely on the
+     repo's in-repo tests via `--test-cmd`.
+   - **Verdicts.** If validation couldn't run (custom runner, deps), set
+     `FAIL_TO_PASS` to the genuinely bug-revealing test name(s). Leave it **empty
+     for suite mode** (graded on the test command's exit code) when the runner
+     emits no TAP names.
+   - **Problem statement.** Tighten it to what the agent should act on — drop PR
+     chatter; keep the bug description.
+   - **`test_cmd` / `setup_cmd`** for non-`node --test` repos, e.g.
+     `--test-cmd "npm test" --setup-cmd "npm ci"`.
+
+3. **Write it.** Re-run without `--dry-run` (add `--datasets <dir>` / `--overlay
+   <dir>` to target a specific workspace). The instance is appended to
+   `<root>/<tier>/instances.json` (creating `tier.json` if the tier is new).
+
+4. **Verify end-to-end.** Run the case with the cheapest model just to confirm the
+   plumbing — the sandbox seeds from the real repo at `base_commit` and grading
+   runs the held-out tests:
+
+   ```bash
+   EVAL_INSTANCE=<instance_id> lastlight-evals run code-fix --model haiku
+   ```
+
+## Test runners & grading
+
+- **`node --test` (default).** Emits TAP; the CLI extracts per-test names and grades
+  each `FAIL_TO_PASS` / `PASS_TO_PASS` by name.
+- **Any other runner via `--test-cmd`.** If it emits TAP with stable names, named
+  grading still works. Otherwise the case runs in **suite mode**: `FAIL_TO_PASS`
+  stays empty and the case is resolved iff the test command exits 0 (after the
+  held-out `test_patch` is applied). Use `--setup-cmd` for install/build.
+
+## Triage from an issue
+
+`add-case --issue <url>` builds a triage case from a **resolved** issue — its
+content plus the human triage outcome:
+
+- `problem_statement` + the `issue` seed from the title/body. **Seed labels are
+  emptied on purpose** so the agent triages from scratch (the applied labels are
+  the gold it must reproduce, not an input).
+- `expect_github.labels_added` = the labels that were applied to the issue (read
+  from the issue **events** API, so the evidence block also shows *who* applied
+  each — maintainer vs bot), and `issue_closed: true` if the issue was closed.
+- An **evidence block** prints the applied labels and the reviewer comments
+  (author + first line) — the raw signal you turn into the gold decision.
+
+What you refine before running:
+
+- **`triage_gold`** (`{ category, state }`) — assign from the applied labels per
+  **the deployment's triage taxonomy** (the CLI can't know which label is the
+  category vs the workflow state, so it leaves `triage_gold` empty). `gradeTriage`
+  checks these label strings end up on the issue.
+- **Prune** any non-triage labels (e.g. `good first issue`) out of `labels_added`.
+- **Reviewer comments** — optionally turn a representative one into a
+  `comment_matches` regex (asserting the agent's triage comment covers the same
+  point), or move a genuinely *pre-triage* maintainer comment into `issue.comments`
+  as context the agent should see.
+
+See `instance-schema.md` for the full triage field reference.

package/plugins/lastlight/skills/lastlight-evals/references/instance-schema.md

@@ -21,12 +21,18 @@ Datasets are discovered from (overlay > user > built-in): `<overlay>/evals/datas
   // ── SWE-bench core ──
   "instance_id": "triage__my-case",      // unique id
   "repo": "owner/repo",                   // logical; fixture origin is a local bare repo
-  "base_commit": "0000000...",            // code-fix only
+  "base_commit": "0000000...",            // code-fix. UNUSED for vendored fixtures
+                                           // (harness synthesizes one); for a
+                                           // git-source case it's the real SHA
+                                           // checked out at run time.
   "problem_statement": "short issue text",
   "patch": "...",                          // gold patch — reference only, NOT graded
-  "test_patch": "...",                     // held-out tests (code-fix)
-  "FAIL_TO_PASS": ["test id 1"],          // must go red→green (code-fix)
+  "test_patch": "...",                     // held-out tests (code-fix), git-apply form
+  "FAIL_TO_PASS": ["test id 1"],          // must go red→green. Empty ⇒ suite mode
   "PASS_TO_PASS": ["test id 2"],          // must stay green (code-fix)
+  "test_cmd": ["npm", "test"],            // held-out test argv (default: node --test)
+  "setup_cmd": ["npm", "ci"],             // optional install/build before tests (git-source)
+  "head_commit": "abc123...",             // PR head SHA — reference/authoring only
 
   // ── Last Light extensions ──
   "workflow": "issue-triage",             // optional; defaults to the tier's defaultWorkflow
@@ -55,16 +61,27 @@ Append a `SweBenchInstance` to `datasets/triage/instances.json` with `issue`,
 `triage_gold`, and the `expect_github` assertions (e.g. `labels_added`). That's
 it — triage is graded on the triage decision + GitHub mutations.
 
-## Add a code-fix case (three things, keyed by `instance_id`)
+## Two flavors of code-fix case
+
+A code-fix instance gets its repo from **one of two** provenances; the rest of the
+machinery (grading, dashboard) is identical.
+
+**A. Vendored fixture** (three things, keyed by `instance_id`):
 
 1. **`datasets/code-fix/instances.json`** — append the instance with
-   `base_commit`, `FAIL_TO_PASS`, `PASS_TO_PASS`, `issue`, and `expect_github`
-   (e.g. `pr_opened`).
+   `FAIL_TO_PASS`, `PASS_TO_PASS`, `issue`, and `expect_github` (e.g. `pr_opened`).
 2. **`datasets/code-fix/repos/<instance_id>/`** — the fixture repo at the base
    commit (the buggy code *before* the fix; no held-out tests here).
 3. **`datasets/code-fix/tests/<instance_id>/`** — the held-out test files, copied
    into the repo at grade time and run to compute `FAIL_TO_PASS` / `PASS_TO_PASS`.
 
+**B. Git-source (from a real PR)** — set `repo` + a real `base_commit` and a
+`test_patch`; **no** `repos/<id>/` is vendored. At run time the harness clones the
+repo into the gitignored `./.eval-cache/` and checks out `base_commit`. Don't
+hand-build these — use `lastlight-evals add-case --pr <url>` (see
+**`authoring-from-pr.md`**), which fills `base_commit`, `head_commit`,
+`test_patch`, and the verdicts for you.
+
 ## Add a custom tier
 
 Create `datasets/<tier-name>/` with `tier.json` (`name`, `defaultWorkflow`,
@@ -78,7 +95,9 @@ and `tests/<id>/`. Discovery auto-finds it — no code change. Run it with
   (`expect_github`)?
 - **Triage:** did the decision match `triage_gold`?
 - **Execution (code-fix):** all `FAIL_TO_PASS` green AND all `PASS_TO_PASS` still
-  green after applying held-out tests.
+  green after applying held-out tests. When the runner emits no TAP test names
+  (a non-`node --test` `test_cmd` and an empty `FAIL_TO_PASS`), grading falls back
+  to **suite mode** — resolved iff the test command exits 0.
 - With `--runs N` (N>1) the binary verdict is **worst-case** (passes only if every
   trial passed); the scorecard also shows per-verdict pass counts to expose
   variance.