lastlight-evals 0.1.0

package/README.md

@@ -0,0 +1,187 @@
+# lastlight-evals
+
+A standalone, **SWE-bench-compatible** eval harness for [Last
+Light](https://github.com/cliftonc/lastlight) workflows. It drives the **real**
+production workflows (`issue-triage`, `build`, …) — their actual prompts and
+skills — against a **mocked GitHub**, grades the result deterministically, and
+prints a model-comparison scorecard. It answers "what do we expect from the
+agent, and which model does it best?"
+
+Nothing here talks to real GitHub. The agent's `github_*` tool calls are served
+by an in-process fake (seeded + recording), and `git push` goes to a local bare
+repo. The only deviations from production are the two we can't do unattended:
+approval gates are disabled and outward side-effects are mocked.
+
+```
+instance (SWE-bench shape)
+   │
+   ├─ start fake GitHub (seeded with the issue, records every mutation)
+   ├─ (code-fix) seed workspace: fixture repo @ base_commit + local bare origin
+   ├─ load the REAL workflow YAML (issue-triage / build / …) from lastlight core
+   ├─ runWorkflow(sandbox:"none", githubApiBaseUrl→fake, approvalConfig:{})
+   └─ grade:
+        • execution  — apply held-out tests, run them → FAIL_TO_PASS / PASS_TO_PASS
+        • behavioral — recorded GitHub calls vs the instance's expectations
+```
+
+> Working on the harness itself? See `CLAUDE.md` for the seams and invariants
+> (the base-URL mock, static-token mode, the no-clone seeding trick, the
+> asset-bootstrap footgun, the metrics drain).
+
+## How it depends on Last Light
+
+`lastlight-evals` is a thin CLI on top of the `lastlight` npm package. It imports
+exactly four things from core's public `lastlight/evals` barrel —
+`getWorkflow`, `runWorkflow`, `ExecutorConfig`, `TemplateContext` — plus the
+`gh`-repo bootstrap helpers used by `init`. Core ships its `workflows/`,
+`skills/`, and `agent-context/` in the package, so the evals run the same assets
+core does.
+
+```bash
+npm install            # installs `lastlight` (and agentic-pi as a peer)
+```
+
+**Local development against an un-published core.** Until the matching
+`lastlight` version is on npm — or whenever you want to eval your working-tree
+core — link a checkout:
+
+```bash
+cd ../lastlight && npm run build && npm link
+cd ../lastlight-evals && npm link lastlight
+```
+
+Or set `LASTLIGHT_CORE_DIR=/path/to/lastlight` to point just the **asset roots**
+(workflows/skills/agent-context — the bulk of what `lastlight server update`
+ships) at a checkout without touching the npm dep. (The runner *code* still
+comes from `node_modules/lastlight`; use `npm link` to exercise working-tree
+engine code too.)
+
+## Run it
+
+```bash
+# no tier args → interactively pick which tiers to run (one or all).
+# Non-interactive (CI / piped) falls back to the cheapest default.
+lastlight-evals run                     # (or: npm run eval)
+
+# name tiers explicitly to skip the prompt
+lastlight-evals run triage
+lastlight-evals run code-fix            # the full build cycle (heavy)
+lastlight-evals run triage code-fix     # both → combined tabbed report
+
+# cross-vendor comparison (OpenAI + Anthropic + open source) — see models.json.
+# Families run in PARALLEL; serial within a family. Force serial with --serial.
+lastlight-evals run --compare
+
+# pick ONE model (fuzzy-matched against models.json id/label)
+lastlight-evals run triage --model haiku
+lastlight-evals run triage --model glm,deepseek   # a comma-list also works
+
+# repeat each case N times; verdicts WORST-case, cost/tokens/latency MEAN
+lastlight-evals run triage --runs 3
+
+# run against an overlay repo's OWN workflows + datasets (see below)
+lastlight-evals run --overlay ~/work/lastlight-instance
+
+# add your own datasets dir without an overlay
+lastlight-evals run --datasets ~/my-evals/datasets
+
+# ad-hoc model set / focus one instance / no browser
+EVAL_MODELS="openai/gpt-5.5,anthropic/claude-sonnet-4-6" lastlight-evals run
+EVAL_INSTANCE=off-by-one lastlight-evals run code-fix
+lastlight-evals run triage --no-open
+```
+
+The runner opens `index.html` and **rewrites it after every run** (auto-refresh,
+preserving the active tab + scroll), so you watch the scorecard fill in live.
+Output lands under `./eval-results/<tiers>/` (override with `LASTLIGHT_EVALS_OUT`):
+
+- `index.html` — styled scorecard.
+- `scorecard.json` — structured roll-up per model.
+- `predictions.jsonl` — SWE-bench predictions shape.
+
+Needs a provider key (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` /
+`FIREWORKS_API_KEY` / `OPENROUTER_API_KEY`) in the environment or a cwd `.env`.
+The runner exits non-zero **only** if the harness itself errors — a weak model
+scoring poorly is the measurement, not a build failure.
+
+## Your own workflows + datasets (overlays)
+
+An **overlay** is a directory (often its own repo, like `lastlight-instance`)
+that carries its own `workflows/` / `skills/` / `agent-context/` (which shadow
+the core built-ins by name) and its own `evals/datasets/`. One flag wires both:
+
+```bash
+lastlight-evals run --overlay ~/work/lastlight-instance     # or LASTLIGHT_OVERLAY_DIR
+```
+
+- Overlay **workflows/skills** are layered over core via core's asset overlay
+  (same mechanism the production harness uses).
+- Overlay **datasets** are discovered at `<overlay>/evals/datasets/<tier>/`, and
+  shadow built-in tiers of the same name.
+- An overlay **`evals/models.json`** is picked up automatically (or pass
+  `--models-file`).
+
+### `lastlight-evals init [dir]` — scaffold a fresh overlay+evals repo
+
+```bash
+lastlight-evals init my-evals
+cd my-evals && lastlight-evals run --overlay .
+```
+
+Scaffolds `workflows/` `skills/` `agent-context/` (empty, to fill in),
+`evals/datasets/` + `evals/models.json` (seeded from the shipped samples),
+`config.yaml`, and a `.gitignore`/`README`, then offers to `git init` + create a
+private GitHub repo via `gh` (reusing core's `lastlight server setup` flow).
+
+## Datasets & tiers
+
+A **tier** is a directory containing `instances.json` (+ an optional `tier.json`
+declaring its `defaultWorkflow`). Tiers are discovered from three roots, merged
+by name with **overlay > user (`--datasets`) > built-in** precedence:
+
+- **built-in** (shipped here): `triage` → `issue-triage`, `code-fix` → `build`.
+- **user**: `--datasets <dir>` / `LASTLIGHT_EVALS_DATASETS`.
+- **overlay**: `<overlay>/evals/datasets/*`.
+
+### Add a case
+
+**Triage** — append to a tier's `instances.json`:
+
+```json
+{
+  "instance_id": "triage__my-case",
+  "repo": "lastlight-evals/widget",
+  "workflow": "issue-triage",
+  "problem_statement": "short title",
+  "issue": { "number": 110, "title": "…", "body": "…", "labels": [] },
+  "triage_gold": { "category": "bug", "state": "ready-for-agent" },
+  "expect_github": { "labels_added": ["bug"] }
+}
+```
+
+**Code-fix** — three things keyed by `instance_id`, all under the tier dir:
+
+```
+<tier>/instances.json     # the SweBenchInstance (FAIL_TO_PASS / PASS_TO_PASS)
+<tier>/repos/<id>/        # fixture repo at base_commit (NO held-out tests)
+<tier>/tests/<id>/        # held-out test files, copied in at grade time
+```
+
+A new tier just needs a directory with an `instances.json` and a `tier.json`
+(`{ "name", "defaultWorkflow", "description" }`); per-instance `workflow` wins
+when present.
+
+## Models (`models.json`)
+
+- `default` — the single model `run` uses.
+- `compare` — the cross-vendor set `--compare` fans out over. Each entry has an
+  `id` (the agentic-pi/pi-ai `provider/model` spec), a `label`, and an `envKey`.
+  **An entry only runs if its `envKey` is present**, so the compare set
+  auto-trims to whatever keys you have.
+
+## Roadmap
+
+- **`lastlight-evals extract <owner>/<repo>#<n>`** — generate eval cases from
+  GitHub historical issues/PRs (issue → fixture, merged PR → held-out tests).
+- Docker-backed runs; real SWE-bench Lite ingestion; per-fixture test runners.
+- LLM-as-judge stays out by design — grading is deterministic.

package/datasets/code-fix/instances.json

@@ -0,0 +1,26 @@
+[
+  {
+    "instance_id": "codefix__date-range-off-by-one",
+    "repo": "lastlight-evals/daterange",
+    "workflow": "build",
+    "base_commit": "0000000000000000000000000000000000000000",
+    "problem_statement": "inclusiveDayCount is off by one\n\n`inclusiveDayCount(start, end)` should return the number of whole days in the INCLUSIVE range [start, end], but it returns one too few.\n\nExample:\n- `inclusiveDayCount(2026-01-01, 2026-01-01)` returns 0, should be 1.\n- `inclusiveDayCount(2026-01-01, 2026-01-03)` returns 2, should be 3.\n\nThe fix is in `src/date-range.ts`. `eachDay` already enumerates the inclusive range correctly, so don't change it.",
+    "issue": {
+      "number": 201,
+      "title": "inclusiveDayCount is off by one",
+      "body": "`inclusiveDayCount(start, end)` should count days in the INCLUSIVE range but returns one too few. `inclusiveDayCount(2026-01-01, 2026-01-01)` returns 0 (should be 1). The bug is in `src/date-range.ts`.",
+      "labels": ["bug", "ready-for-agent"],
+      "user": "reporter"
+    },
+    "FAIL_TO_PASS": [
+      "inclusive single day counts as one",
+      "inclusive three day span counts as three"
+    ],
+    "PASS_TO_PASS": [
+      "eachDay still lists every inclusive day"
+    ],
+    "expect_github": {
+      "pr_opened": { "base": "main", "head_is_branch": true }
+    }
+  }
+]

package/datasets/code-fix/repos/codefix__date-range-off-by-one/README.md

@@ -0,0 +1,8 @@
+# daterange
+
+A tiny inclusive date-range utility.
+
+- `inclusiveDayCount(start, end)` — number of whole days in the inclusive range.
+- `eachDay(start, end)` — every `YYYY-MM-DD` in the inclusive range.
+
+Run the tests with `npm test` (`node --test`).

package/datasets/code-fix/repos/codefix__date-range-off-by-one/package.json

@@ -0,0 +1,9 @@
+{
+  "name": "daterange",
+  "version": "1.0.0",
+  "type": "module",
+  "description": "Tiny date-range utility.",
+  "scripts": {
+    "test": "node --test --experimental-strip-types"
+  }
+}

package/datasets/code-fix/repos/codefix__date-range-off-by-one/src/date-range.ts

@@ -0,0 +1,25 @@
+/**
+ * Date-range helpers.
+ */
+
+const MS_PER_DAY = 24 * 60 * 60 * 1000;
+
+/**
+ * Number of whole days covered by the INCLUSIVE range [start, end].
+ *
+ * e.g. 2026-01-01 .. 2026-01-01 covers 1 day; 2026-01-01 .. 2026-01-03 covers 3.
+ */
+export function inclusiveDayCount(start: Date, end: Date): number {
+  const ms = end.getTime() - start.getTime();
+  // BUG: drops the inclusive end day — should add 1.
+  return Math.round(ms / MS_PER_DAY);
+}
+
+/** All YYYY-MM-DD dates in the inclusive range, in order. */
+export function eachDay(start: Date, end: Date): string[] {
+  const out: string[] = [];
+  for (let t = start.getTime(); t <= end.getTime(); t += MS_PER_DAY) {
+    out.push(new Date(t).toISOString().slice(0, 10));
+  }
+  return out;
+}

package/datasets/code-fix/tests/codefix__date-range-off-by-one/date-range.test.ts

@@ -0,0 +1,18 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+
+import { inclusiveDayCount, eachDay } from "./src/date-range.ts";
+
+const d = (s: string): Date => new Date(`${s}T00:00:00Z`);
+
+test("inclusive single day counts as one", () => {
+  assert.equal(inclusiveDayCount(d("2026-01-01"), d("2026-01-01")), 1);
+});
+
+test("inclusive three day span counts as three", () => {
+  assert.equal(inclusiveDayCount(d("2026-01-01"), d("2026-01-03")), 3);
+});
+
+test("eachDay still lists every inclusive day", () => {
+  assert.deepEqual(eachDay(d("2026-01-01"), d("2026-01-03")), ["2026-01-01", "2026-01-02", "2026-01-03"]);
+});

package/datasets/code-fix/tier.json

@@ -0,0 +1,5 @@
+{
+  "name": "code-fix",
+  "defaultWorkflow": "build",
+  "description": "Full Architect→Executor→Reviewer build cycle on a seeded fixture repo (heavy)."
+}

package/datasets/triage/instances.json

@@ -0,0 +1,62 @@
+[
+  {
+    "instance_id": "triage__bug-with-repro-ready",
+    "repo": "lastlight-evals/widget",
+    "workflow": "issue-triage",
+    "problem_statement": "Crash on empty config file",
+    "issue": {
+      "number": 101,
+      "title": "Crash on empty config file",
+      "body": "## Steps to reproduce\n1. Create an empty `config.yaml`\n2. Run `widget start`\n\n## Expected\nIt should start with defaults.\n\n## Actual\nIt throws `TypeError: Cannot read properties of undefined (reading 'port')` at `src/config.ts:42`.\n\nEnvironment: widget v2.1.0, Node 20. This reproduces 100% of the time.",
+      "labels": [],
+      "user": "alice"
+    },
+    "triage_gold": { "category": "bug", "state": "ready-for-agent" },
+    "expect_github": { "labels_added": ["bug"] }
+  },
+  {
+    "instance_id": "triage__bug-no-repro-needs-info",
+    "repo": "lastlight-evals/widget",
+    "workflow": "issue-triage",
+    "problem_statement": "It's broken",
+    "issue": {
+      "number": 102,
+      "title": "It doesn't work",
+      "body": "I tried to use the thing and it didn't work. Please fix.",
+      "labels": [],
+      "user": "bob"
+    },
+    "triage_gold": { "state": "needs-info" },
+    "expect_github": { "labels_added": ["needs-info"], "labels_absent": ["ready-for-agent"], "comment_matches": "need|provide|repro|version|steps" }
+  },
+  {
+    "instance_id": "triage__feature-well-specified",
+    "repo": "lastlight-evals/widget",
+    "workflow": "issue-triage",
+    "problem_statement": "Add --json flag to widget status",
+    "issue": {
+      "number": 103,
+      "title": "Add a --json output flag to `widget status`",
+      "body": "## Use case\nWe script around `widget status` in CI and parsing the human table is brittle.\n\n## Proposal\nAdd a `--json` flag that prints the same status as a JSON object: `{ \"state\": \"running\", \"uptimeSec\": 1234 }`.\n\nThe table output stays the default. This is additive and well-scoped — the status data already exists in `src/status.ts`.",
+      "labels": [],
+      "user": "carol"
+    },
+    "triage_gold": { "category": "enhancement" },
+    "expect_github": { "labels_added": ["enhancement"] }
+  },
+  {
+    "instance_id": "triage__pure-question",
+    "repo": "lastlight-evals/widget",
+    "workflow": "issue-triage",
+    "problem_statement": "How do I change the port?",
+    "issue": {
+      "number": 104,
+      "title": "How do I change the listening port?",
+      "body": "Quick question — what's the right way to change the port widget listens on? Is it an env var or a config key? Not a bug, just want to understand the options.",
+      "labels": [],
+      "user": "dave"
+    },
+    "triage_gold": { "category": "question" },
+    "expect_github": { "labels_added": ["question"], "labels_absent": ["ready-for-agent", "ready-for-human"] }
+  }
+]

package/datasets/triage/tier.json

@@ -0,0 +1,5 @@
+{
+  "name": "triage",
+  "defaultWorkflow": "issue-triage",
+  "description": "Issue triage — labels, category, and state-role accuracy (cheap)."
+}

package/dist/bootstrap.js

@@ -0,0 +1,46 @@
+/**
+ * Core-asset bootstrap — the one thing an out-of-process eval harness MUST do
+ * that the in-repo version never had to.
+ *
+ * Last Light's `getWorkflow` resolves built-in workflows/skills/agent-context
+ * from `DEFAULT_ROOT = resolve(".")` (the process cwd). In-repo that happened to
+ * be the core checkout, so it "just worked". As a separate package our cwd is
+ * wherever the user invoked the CLI, so we MUST tell core where its assets live
+ * by calling `configureWorkflowAssets({ builtInRoot })` BEFORE any
+ * `getWorkflow`/`runWorkflow`. Forget this and workflows silently fail to
+ * resolve. {@link bootstrapAssets} is therefore the first call in `run`.
+ */
+import { createRequire } from "node:module";
+import { dirname } from "node:path";
+import { configureWorkflowAssets } from "lastlight/evals";
+/**
+ * The lastlight PACKAGE ROOT — the dir holding `workflows/`, `skills/`,
+ * `agent-context/`, `config/`, `dist/`.
+ *
+ * Default: resolve the installed `lastlight` package (a normal npm dependency).
+ *
+ * Override: `LASTLIGHT_CORE_DIR` repoints the ASSET roots at a local core
+ * checkout, so you can eval un-published workflow/prompt/skill edits — the bulk
+ * of what `lastlight server update` ships — without bumping the npm dep. Caveat:
+ * the imported runner CODE still comes from `node_modules/lastlight`; to also
+ * exercise working-tree engine code, `npm link lastlight` (or a `file:` dep).
+ */
+export function resolveCoreRoot() {
+    const override = process.env.LASTLIGHT_CORE_DIR?.trim();
+    if (override)
+        return override;
+    const require = createRequire(import.meta.url);
+    return dirname(require.resolve("lastlight/package.json"));
+}
+/**
+ * Point core's asset layers at the resolved core root (+ optional overlay).
+ * MUST run before the first workflow access. An overlay's workflows/skills/
+ * agent-context shadow the built-ins by logical name — the same precedence the
+ * production harness uses via `LASTLIGHT_OVERLAY_DIR`.
+ */
+export function bootstrapAssets(opts = {}) {
+    const builtInRoot = resolveCoreRoot();
+    configureWorkflowAssets({ builtInRoot, overlayRoot: opts.overlayDir });
+    return { builtInRoot, overlayDir: opts.overlayDir };
+}
+//# sourceMappingURL=bootstrap.js.map

package/dist/bootstrap.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"bootstrap.js","sourceRoot":"","sources":["../src/bootstrap.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AACH,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAC5C,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAEpC,OAAO,EAAE,uBAAuB,EAAE,MAAM,iBAAiB,CAAC;AAE1D;;;;;;;;;;;GAWG;AACH,MAAM,UAAU,eAAe;IAC7B,MAAM,QAAQ,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,EAAE,IAAI,EAAE,CAAC;IACxD,IAAI,QAAQ;QAAE,OAAO,QAAQ,CAAC;IAC9B,MAAM,OAAO,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAC/C,OAAO,OAAO,CAAC,OAAO,CAAC,OAAO,CAAC,wBAAwB,CAAC,CAAC,CAAC;AAC5D,CAAC;AAOD;;;;;GAKG;AACH,MAAM,UAAU,eAAe,CAAC,OAAgC,EAAE;IAChE,MAAM,WAAW,GAAG,eAAe,EAAE,CAAC;IACtC,uBAAuB,CAAC,EAAE,WAAW,EAAE,WAAW,EAAE,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;IACvE,OAAO,EAAE,WAAW,EAAE,UAAU,EAAE,IAAI,CAAC,UAAU,EAAE,CAAC;AACtD,CAAC"}

package/dist/discovery.js

@@ -0,0 +1,100 @@
+/**
+ * Tier/dataset discovery — replaces the old hardcoded `TIERS` map.
+ *
+ * A **tier** is simply a directory that contains an `instances.json`
+ * (alongside, for code-fix tiers, `repos/<id>/` fixtures and `tests/<id>/`
+ * held-out tests). Tiers are discovered from up to three roots and merged by
+ * name with **overlay > user > built-in** precedence — the same shadow-by-name
+ * model core uses for workflow assets, so a deployment can override a shipped
+ * tier or add entirely new ones without touching this package.
+ *
+ *   1. built-in — `<pkg>/datasets/*`            (the shipped "our" samples)
+ *   2. user     — `--datasets <dir>` / `LASTLIGHT_EVALS_DATASETS`
+ *   3. overlay  — `<overlayDir>/evals/datasets/*`  (a bootstrapped repo)
+ *
+ * Which workflow a tier runs is resolved per-instance first (the existing,
+ * unchanged `instance.workflow` field), then from an optional per-tier
+ * `tier.json` (`{ name?, defaultWorkflow, description? }`). Shipping a
+ * `tier.json` next to the two built-in datasets turns the former hardcoded map
+ * into data with zero changes to the instance files.
+ */
+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
+import { join, resolve } from "node:path";
+/** Scan one root for immediate subdirs that hold an `instances.json`. */
+function scanRoot(root, source) {
+    if (!root || !existsSync(root))
+        return [];
+    const out = [];
+    for (const entry of readdirSync(root)) {
+        const dir = join(root, entry);
+        let isDir = false;
+        try {
+            isDir = statSync(dir).isDirectory();
+        }
+        catch {
+            continue;
+        }
+        if (!isDir)
+            continue;
+        const instancesPath = join(dir, "instances.json");
+        if (!existsSync(instancesPath))
+            continue;
+        let manifest = {};
+        const manifestPath = join(dir, "tier.json");
+        if (existsSync(manifestPath)) {
+            try {
+                manifest = JSON.parse(readFileSync(manifestPath, "utf8"));
+            }
+            catch {
+                /* a malformed tier.json just means no defaultWorkflow — not fatal */
+            }
+        }
+        out.push({
+            name: manifest.name || entry,
+            source,
+            root: dir,
+            instancesPath,
+            defaultWorkflow: manifest.defaultWorkflow,
+            description: manifest.description,
+        });
+    }
+    return out;
+}
+/**
+ * Discover all tiers across the roots, overlay-wins by name. Insertion order
+ * (built-in first) is preserved for stable display; later sources overwrite the
+ * map value for a shared name.
+ */
+export function discoverTiers(opts) {
+    const tiers = new Map();
+    const add = (list) => {
+        for (const t of list)
+            tiers.set(t.name, t);
+    };
+    add(scanRoot(opts.builtinRoot, "builtin"));
+    if (opts.userDatasetsDir)
+        add(scanRoot(resolve(opts.userDatasetsDir), "user"));
+    if (opts.overlayDir)
+        add(scanRoot(join(resolve(opts.overlayDir), "evals", "datasets"), "overlay"));
+    return tiers;
+}
+/** Load a tier's instances, with the optional `EVAL_INSTANCE` substring filter. */
+export function loadInstances(tier) {
+    const all = JSON.parse(readFileSync(tier.instancesPath, "utf8"));
+    const filter = process.env.EVAL_INSTANCE?.trim();
+    return filter ? all.filter((i) => i.instance_id.includes(filter)) : all;
+}
+/**
+ * Resolve the workflow to run for one instance: explicit `instance.workflow`
+ * wins, else the tier's `defaultWorkflow`. Throws if neither is set — a loud
+ * misconfiguration rather than a silent wrong default.
+ */
+export function workflowFor(tier, inst) {
+    const wf = inst.workflow ?? tier.defaultWorkflow;
+    if (!wf) {
+        throw new Error(`Tier "${tier.name}": instance "${inst.instance_id}" has no \`workflow\` and the ` +
+            `tier has no \`defaultWorkflow\` (add one to ${join(tier.root, "tier.json")}).`);
+    }
+    return wf;
+}
+//# sourceMappingURL=discovery.js.map

package/dist/discovery.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"discovery.js","sourceRoot":"","sources":["../src/discovery.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AACH,OAAO,EAAE,UAAU,EAAE,WAAW,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC;AAC1E,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAuB1C,yEAAyE;AACzE,SAAS,QAAQ,CAAC,IAAY,EAAE,MAAkB;IAChD,IAAI,CAAC,IAAI,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;QAAE,OAAO,EAAE,CAAC;IAC1C,MAAM,GAAG,GAAW,EAAE,CAAC;IACvB,KAAK,MAAM,KAAK,IAAI,WAAW,CAAC,IAAI,CAAC,EAAE,CAAC;QACtC,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;QAC9B,IAAI,KAAK,GAAG,KAAK,CAAC;QAClB,IAAI,CAAC;YACH,KAAK,GAAG,QAAQ,CAAC,GAAG,CAAC,CAAC,WAAW,EAAE,CAAC;QACtC,CAAC;QAAC,MAAM,CAAC;YACP,SAAS;QACX,CAAC;QACD,IAAI,CAAC,KAAK;YAAE,SAAS;QACrB,MAAM,aAAa,GAAG,IAAI,CAAC,GAAG,EAAE,gBAAgB,CAAC,CAAC;QAClD,IAAI,CAAC,UAAU,CAAC,aAAa,CAAC;YAAE,SAAS;QAEzC,IAAI,QAAQ,GAAiB,EAAE,CAAC;QAChC,MAAM,YAAY,GAAG,IAAI,CAAC,GAAG,EAAE,WAAW,CAAC,CAAC;QAC5C,IAAI,UAAU,CAAC,YAAY,CAAC,EAAE,CAAC;YAC7B,IAAI,CAAC;gBACH,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,YAAY,EAAE,MAAM,CAAC,CAAiB,CAAC;YAC5E,CAAC;YAAC,MAAM,CAAC;gBACP,qEAAqE;YACvE,CAAC;QACH,CAAC;QACD,GAAG,CAAC,IAAI,CAAC;YACP,IAAI,EAAE,QAAQ,CAAC,IAAI,IAAI,KAAK;YAC5B,MAAM;YACN,IAAI,EAAE,GAAG;YACT,aAAa;YACb,eAAe,EAAE,QAAQ,CAAC,eAAe;YACzC,WAAW,EAAE,QAAQ,CAAC,WAAW;SAClC,CAAC,CAAC;IACL,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAQD;;;;GAIG;AACH,MAAM,UAAU,aAAa,CAAC,IAAqB;IACjD,MAAM,KAAK,GAAG,IAAI,GAAG,EAAgB,CAAC;IACtC,MAAM,GAAG,GAAG,CAAC,IAAY,EAAE,EAAE;QAC3B,KAAK,MAAM,CAAC,IAAI,IAAI;YAAE,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC;IAC7C,CAAC,CAAC;IACF,GAAG,CAAC,QAAQ,CAAC,IAAI,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC,CAAC;IAC3C,IAAI,IAAI,CAAC,eAAe;QAAE,GAAG,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,eAAe,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC;IAC/E,IAAI,IAAI,CAAC,UAAU;QAAE,GAAG,CAAC,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,UAAU,CAAC,EAAE,OAAO,EAAE,UAAU,CAAC,EAAE,SAAS,CAAC,CAAC,CAAC;IACnG,OAAO,KAAK,CAAC;AACf,CAAC;AAED,mFAAmF;AACnF,MAAM,UAAU,aAAa,CAAC,IAAU;IACtC,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,IAAI,CAAC,aAAa,EAAE,MAAM,CAAC,CAAuB,CAAC;IACvF,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,aAAa,EAAE,IAAI,EAAE,CAAC;IACjD,OAAO,MAAM,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,WAAW,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC;AAC1E,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,WAAW,CAAC,IAAU,EAAE,IAAsB;IAC5D,MAAM,EAAE,GAAG,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,eAAe,CAAC;IACjD,IAAI,CAAC,EAAE,EAAE,CAAC;QACR,MAAM,IAAI,KAAK,CACb,SAAS,IAAI,CAAC,IAAI,gBAAgB,IAAI,CAAC,WAAW,gCAAgC;YAChF,+CAA+C,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,WAAW,CAAC,IAAI,CAClF,CAAC;IACJ,CAAC;IACD,OAAO,EAAE,CAAC;AACZ,CAAC"}

package/dist/env.js

@@ -0,0 +1,109 @@
+/**
+ * Minimal `.env` loader for the eval harness (no dotenv dependency).
+ *
+ * Reads the repo-root `.env` (KEY=VALUE lines) and sets any keys not already
+ * present in `process.env`. The provider key (OPENAI_API_KEY / ANTHROPIC_…)
+ * lives there for local dev; the eval needs it to make real model calls.
+ */
+import { readFileSync, existsSync } from "node:fs";
+import { join, resolve } from "node:path";
+import { builtinModelsPath } from "./paths.js";
+let loaded = false;
+/**
+ * Active model-registry path. Defaults to the shipped `<pkg>/models.json`; a
+ * user/overlay can point it elsewhere via {@link setModelsPath} (run.ts wires
+ * `--models-file` / `<overlay>/evals/models.json`) so a deployment can ship its
+ * own model set the same way it ships its own datasets.
+ */
+let modelsPath = builtinModelsPath();
+/** Override the model-registry path (call once at CLI startup, before reads). */
+export function setModelsPath(path) {
+    modelsPath = resolve(path);
+}
+export function loadDotEnv(root = process.cwd()) {
+    if (loaded)
+        return;
+    loaded = true;
+    const path = join(root, ".env");
+    if (!existsSync(path))
+        return;
+    for (const raw of readFileSync(path, "utf8").split("\n")) {
+        const line = raw.trim();
+        if (!line || line.startsWith("#"))
+            continue;
+        const eq = line.indexOf("=");
+        if (eq <= 0)
+            continue;
+        const key = line.slice(0, eq).trim();
+        let val = line.slice(eq + 1).trim();
+        if ((val.startsWith('"') && val.endsWith('"')) || (val.startsWith("'") && val.endsWith("'"))) {
+            val = val.slice(1, -1);
+        }
+        if (process.env[key] === undefined)
+            process.env[key] = val;
+    }
+}
+/** True if at least one provider key the eval can use is set. */
+export function hasProviderKey() {
+    return Boolean(process.env.OPENAI_API_KEY ||
+        process.env.ANTHROPIC_API_KEY ||
+        process.env.OPENROUTER_API_KEY ||
+        process.env.FIREWORKS_API_KEY ||
+        process.env.DEEPSEEK_API_KEY);
+}
+function modelsConfig() {
+    return JSON.parse(readFileSync(modelsPath, "utf8"));
+}
+/** Single-model run: EVAL_MODELS override, else the `default` from models.json. */
+export function evalModels() {
+    const raw = process.env.EVAL_MODELS?.trim();
+    if (raw)
+        return raw.split(",").map((m) => m.trim()).filter(Boolean);
+    return [modelsConfig().default];
+}
+/**
+ * Cross-vendor comparison set from `models.json`, filtered to the models whose
+ * provider key is actually present — so `npm run eval:compare` runs whatever
+ * you have keys for and silently skips the rest.
+ */
+export function compareModels() {
+    return modelsConfig().compare.filter((m) => !m.envKey || Boolean(process.env[m.envKey]));
+}
+/** id → display label, from models.json (for the scorecard). */
+export function modelLabels() {
+    const out = {};
+    for (const m of modelsConfig().compare)
+        if (m.label)
+            out[m.id] = m.label;
+    return out;
+}
+/** Provider family (env-key) inferred from a `provider/model` id prefix. */
+export function familyForId(id) {
+    const provider = id.split("/")[0]?.toLowerCase() ?? "";
+    const map = {
+        openai: "OPENAI_API_KEY",
+        anthropic: "ANTHROPIC_API_KEY",
+        fireworks: "FIREWORKS_API_KEY",
+        openrouter: "OPENROUTER_API_KEY",
+        deepseek: "DEEPSEEK_API_KEY",
+    };
+    return map[provider] ?? "default";
+}
+/**
+ * Resolve a user-supplied model token (from `--model`) to a concrete entry.
+ * Matches a models.json `compare` id exactly, else a case-insensitive substring
+ * of its id or label (so `--model haiku` works); failing that, treats the token
+ * as a raw `provider/model` id and infers the family from its prefix. Not
+ * key-gated — `--model` is an explicit request, so we run it even if the key
+ * check would otherwise skip it (the provider call surfaces a missing key).
+ */
+export function resolveModel(token) {
+    const all = modelsConfig().compare;
+    const v = token.toLowerCase();
+    const hit = all.find((m) => m.id === token) ??
+        all.find((m) => m.id.toLowerCase().includes(v) || m.label?.toLowerCase().includes(v));
+    if (hit)
+        return { ...hit, family: hit.envKey ?? familyForId(hit.id) };
+    return { id: token, family: familyForId(token) };
+}
+//# sourceMappingURL=env.js.map

package/dist/env.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"env.js","sourceRoot":"","sources":["../src/env.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAEH,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AACnD,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAE1C,OAAO,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAE/C,IAAI,MAAM,GAAG,KAAK,CAAC;AAEnB;;;;;GAKG;AACH,IAAI,UAAU,GAAG,iBAAiB,EAAE,CAAC;AAErC,iFAAiF;AACjF,MAAM,UAAU,aAAa,CAAC,IAAY;IACxC,UAAU,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;AAC7B,CAAC;AAED,MAAM,UAAU,UAAU,CAAC,IAAI,GAAG,OAAO,CAAC,GAAG,EAAE;IAC7C,IAAI,MAAM;QAAE,OAAO;IACnB,MAAM,GAAG,IAAI,CAAC;IACd,MAAM,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IAChC,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;QAAE,OAAO;IAC9B,KAAK,MAAM,GAAG,IAAI,YAAY,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;QACzD,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC;YAAE,SAAS;QAC5C,MAAM,EAAE,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;QAC7B,IAAI,EAAE,IAAI,CAAC;YAAE,SAAS;QACtB,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;QACrC,IAAI,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QACpC,IAAI,CAAC,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,GAAG,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,GAAG,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC;YAC7F,GAAG,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;QACzB,CAAC;QACD,IAAI,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,KAAK,SAAS;YAAE,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC;IAC7D,CAAC;AACH,CAAC;AAED,iEAAiE;AACjE,MAAM,UAAU,cAAc;IAC5B,OAAO,OAAO,CACZ,OAAO,CAAC,GAAG,CAAC,cAAc;QACxB,OAAO,CAAC,GAAG,CAAC,iBAAiB;QAC7B,OAAO,CAAC,GAAG,CAAC,kBAAkB;QAC9B,OAAO,CAAC,GAAG,CAAC,iBAAiB;QAC7B,OAAO,CAAC,GAAG,CAAC,gBAAgB,CAC/B,CAAC;AACJ,CAAC;AAaD,SAAS,YAAY;IACnB,OAAO,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,UAAU,EAAE,MAAM,CAAC,CAAiB,CAAC;AACtE,CAAC;AAED,mFAAmF;AACnF,MAAM,UAAU,UAAU;IACxB,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,WAAW,EAAE,IAAI,EAAE,CAAC;IAC5C,IAAI,GAAG;QAAE,OAAO,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;IACpE,OAAO,CAAC,YAAY,EAAE,CAAC,OAAO,CAAC,CAAC;AAClC,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,aAAa;IAC3B,OAAO,YAAY,EAAE,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;AAC3F,CAAC;AAED,gEAAgE;AAChE,MAAM,UAAU,WAAW;IACzB,MAAM,GAAG,GAA2B,EAAE,CAAC;IACvC,KAAK,MAAM,CAAC,IAAI,YAAY,EAAE,CAAC,OAAO;QAAE,IAAI,CAAC,CAAC,KAAK;YAAE,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC;IACzE,OAAO,GAAG,CAAC;AACb,CAAC;AAED,4EAA4E;AAC5E,MAAM,UAAU,WAAW,CAAC,EAAU;IACpC,MAAM,QAAQ,GAAG,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,IAAI,EAAE,CAAC;IACvD,MAAM,GAAG,GAA2B;QAClC,MAAM,EAAE,gBAAgB;QACxB,SAAS,EAAE,mBAAmB;QAC9B,SAAS,EAAE,mBAAmB;QAC9B,UAAU,EAAE,oBAAoB;QAChC,QAAQ,EAAE,kBAAkB;KAC7B,CAAC;IACF,OAAO,GAAG,CAAC,QAAQ,CAAC,IAAI,SAAS,CAAC;AACpC,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,YAAY,CAAC,KAAa;IACxC,MAAM,GAAG,GAAG,YAAY,EAAE,CAAC,OAAO,CAAC;IACnC,MAAM,CAAC,GAAG,KAAK,CAAC,WAAW,EAAE,CAAC;IAC9B,MAAM,GAAG,GACP,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,KAAK,KAAK,CAAC;QAC/B,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,WAAW,EAAE,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,WAAW,EAAE,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC;IACxF,IAAI,GAAG;QAAE,OAAO,EAAE,GAAG,GAAG,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,IAAI,WAAW,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC;IACtE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,CAAC,KAAK,CAAC,EAAE,CAAC;AACnD,CAAC"}