lastlight-evals 0.1.2 → 0.2.0

package/README.md

@@ -2,9 +2,9 @@
 
 ### Which model should run your agent? Find out — with receipts.
 
-[![Last Light eval scorecard — 9 models compared on pass rate, cost, and latency](docs/scorecard.png)](https://cliftonc.github.io/lastlight-evals/)
+[![Last Light eval scorecard — 9 models compared on pass rate, cost, and latency](docs/scorecard-v2.png)](https://evals.lastlight.dev/)
 
-> **[▶ Explore the live scorecard](https://cliftonc.github.io/lastlight-evals/)** — interactive, with per-instance detail. (Above: triage tier across 9 models.)
+> **[▶ Explore the live scorecard](https://evals.lastlight.dev/)** — interactive, with per-instance detail. (Above: code-fix tier across 9 models.)
 
 `lastlight-evals` takes [**Last Light**](https://lastlight.dev)'s *real*
 production workflows — the actual prompts, skills, and agent loop that ship — and
@@ -45,10 +45,37 @@ instance (SWE-bench shape)
 
 ## Get started
 
-Needs **Node 24+** and a provider API key. The fastest path is
-**`init`** — it scaffolds *your own* evals workspace (your workflows + your
-datasets, seeded from the built-in samples) and optionally creates a private
-GitHub repo for it:
+Needs **Node 24+** and a provider API key.
+
+### Easiest: let the Last Light agent skill set up *your own* workspace
+
+Want to eval **your own** deployment — your workflows, your agent persona, your
+config — not just the shipped samples? If you drive
+[Last Light](https://lastlight.dev) from an agent (e.g. Claude Code), install its
+skills once and then just *ask* — no flags to remember:
+
+```bash
+lastlight skills install            # installs the Last Light agent skills
+```
+
+Then, in a **new empty folder**, tell your agent (point it at *your* instance
+overlay repo):
+
+> *Let's set up an evals workspace here, using my existing Last Light instance
+> config in `cliftonc/lastlight-instance`.*
+
+The `lastlight-evals` skill scaffolds the workspace, clones your overlay into
+`instance/`, seeds the sample datasets, and wires it all up — under the hood it
+runs `lastlight-evals init . --clone cliftonc/lastlight-instance`, after which a
+bare `lastlight-evals run` "just works" (it auto-detects `./instance` as the
+overlay and `./evals/datasets`). Now you're evaluating *your* agent against the
+models you care about. Prefer to drive it by hand? Keep reading.
+
+### Manual: scaffold with `init`
+
+The fastest CLI path is **`init`** — it scaffolds *your own* evals workspace
+(your workflows + your datasets, seeded from the built-in samples) and optionally
+creates a private GitHub repo for it:
 
 ```bash
 npm install -g lastlight-evals
@@ -154,25 +181,64 @@ lastlight-evals run --overlay ~/work/lastlight-instance
 # add your own datasets dir without an overlay
 lastlight-evals run --datasets ~/my-evals/datasets
 
+# CONFIG run type — eval a deployment's REAL per-step model config (different
+# models per workflow phase, from the overlay's config.yaml) instead of forcing
+# one model. This is the setup you actually ship. Try the bundled sample overlay:
+lastlight-evals run code-fix --mode config --overlay examples/overlay
+lastlight-evals run code-fix --mode config --overlay A --overlay B   # 2 configs side-by-side
+
 # ad-hoc model set / focus one instance / no browser
 EVAL_MODELS="openai/gpt-5.5,anthropic/claude-sonnet-4-6" lastlight-evals run
 EVAL_INSTANCE=off-by-one lastlight-evals run code-fix
 lastlight-evals run triage --no-open
 ```
 
-The runner opens `index.html` and **rewrites it after every run** (auto-refresh,
-preserving the active tab + scroll), so you watch the scorecard fill in live.
-Output lands under `./eval-results/<tiers>/` (override with `LASTLIGHT_EVALS_OUT`):
+The report is a **JSON-driven dashboard**, not generated HTML — the harness only
+ever writes `scorecard.json`, updating it (atomically) as the run proceeds. The
+runner starts a tiny local server and opens `http://localhost:PORT` deep-linked
+at the run, so you watch the scorecard fill in live (the SPA polls the JSON).
+When the run finishes the server stays up so the dashboard keeps working — press
+`Ctrl-C` to stop it. Each run lands in its **own** timestamped folder, so runs
+accumulate instead of overwriting — `./eval-results/<tiers>/<runId>/` (override
+the root with `LASTLIGHT_EVALS_OUT`), where `runId` is `<timestamp>-<git-sha>`:
 
-- `index.html` — styled scorecard.
-- `scorecard.json` — structured roll-up per model.
+- `scorecard.json` — structured roll-up per model + per-instance results, carrying run `meta`.
 - `predictions.jsonl` — SWE-bench predictions shape.
 
+The dashboard's **overview** lists every run newest-first with a per-model trend
+sparkline and links into each run's full scorecard; the **run view** is the
+model-comparison table plus per-instance rows. To browse past runs anytime
+without running models, start the server on its own:
+
+```bash
+lastlight-evals serve            # opens the dashboard over ./eval-results
+lastlight-evals serve --port 4319
+```
+
 Needs a provider key (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` /
 `FIREWORKS_API_KEY` / `OPENROUTER_API_KEY`) in the environment or a cwd `.env`.
 The runner exits non-zero **only** if the harness itself errors — a weak model
 scoring poorly is the measurement, not a build failure.
 
+### Two run types
+
+A run compares N **arms** along one of two axes — pick with `--mode` (or, in a
+TTY with no model flags, you're asked):
+
+- **`models`** (default) — compare models, each **forced across every workflow
+  step**. `--model`/`--compare` select the set. Lands in `eval-results/<tier>/`
+  (or `<tier>-compare/`).
+- **`config`** (`--mode config`) — run a deployment's **real per-step model
+  config**: the `models`/`variants` maps from an overlay's `config.yaml`, merged
+  over core's `config/default.yaml` exactly as production does, so each phase can
+  run on a different model. The arm is the config/overlay; pass `--overlay` more
+  than once to compare configs side-by-side, or re-run over time to compare as
+  you tweak prompts/skills/workflow/model-config. `--model` overrides a config's
+  `default` for quick what-ifs. Lands in `eval-results/<tier>-config/`, on its
+  own trend line. The run view shows a **Per-step models** panel with each
+  phase's resolved model. See [`examples/overlay`](examples/overlay) for a
+  ready-to-run sample.
+
 ## Your own workflows + datasets (overlays)
 
 An **overlay** is a directory (often its own repo, like `lastlight-instance`)
@@ -194,7 +260,12 @@ lastlight-evals run --overlay ~/work/lastlight-instance     # or LASTLIGHT_OVERL
 - An overlay **`evals/models.json`** is picked up automatically (or pass
   `--models-file`).
 
-### `lastlight-evals init [dir]` — scaffold a fresh overlay+evals repo
+### `lastlight-evals init [dir]` — scaffold an evals workspace
+
+Two shapes, depending on whether you already have a deployment overlay repo:
+
+**Plain** — a self-contained overlay+evals repo (its own `workflows/` `skills/`
+`agent-context/` + `evals/`):
 
 ```bash
 lastlight-evals init my-evals
@@ -206,6 +277,21 @@ Scaffolds `workflows/` `skills/` `agent-context/` (empty, to fill in),
 `config.yaml`, and a `.gitignore`/`README`, then offers to `git init` + create a
 private GitHub repo via `gh` (reusing core's `lastlight server setup` flow).
 
+**Separate** (`--clone`) — the recommended shape when you already have a
+deployment overlay (e.g. `lastlight-instance`) and want to eval **its** config.
+The overlay is cloned into `<dir>/instance/` (its own git checkout, git-ignored)
+with the evals at the workspace root; a bare run auto-detects both, no flags:
+
+```bash
+lastlight-evals init my-evals --clone cliftonc/lastlight-instance
+cd my-evals && lastlight-evals run        # auto: overlay ./instance + ./evals/datasets
+```
+
+This is exactly what the [`lastlight-evals` agent skill](#easiest-let-the-last-light-agent-skill-set-up-your-own-workspace)
+does for you. Update the overlay later with `cd instance && git pull`; your evals
+stay out of the deployment repo. Run `lastlight-evals init --help` for all flags
+(`--yes`, `--no-git`, …).
+
 ## Datasets & tiers
 
 A **tier** is a directory containing `instances.json` (+ an optional `tier.json`