@hona/openeval 0.2.0 → 0.2.2

package/README.md

@@ -1,29 +1,74 @@
-# OpenEval
+<div align="center">
+  <h1>OpenEval</h1>
+  <p><strong>Write the task. Judge the evidence.</strong></p>
+  <p>Prompt-and-rubric evaluations for agents. Typed declarations, isolated runs, inspectable scores.</p>
+  <p>
+    <a href="https://openev.al">Website</a> ·
+    <a href="https://openeval.pages.dev">Live preview</a> ·
+    <a href="https://openev.al/docs/quickstart/">Write your first eval</a> ·
+    <a href="https://openev.al/docs/reference/">CLI reference</a> ·
+    <a href="https://www.npmjs.com/package/@hona/openeval">npm</a>
+  </p>
+  <p>
+    <a href="https://www.npmjs.com/package/@hona/openeval"><img src="https://img.shields.io/npm/v/%40hona%2Fopeneval?style=flat-square&color=66d38a" alt="npm version"></a>
+    <a href="https://bun.com"><img src="https://img.shields.io/badge/Bun-1.4.2%2B-f9f1e1?style=flat-square" alt="Bun 1.4.2 or later"></a>
+    <a href="https://github.com/Hona/openeval/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-66d38a?style=flat-square" alt="MIT license"></a>
+  </p>
+</div>
+
+![The OpenEval workbench: rubric source, a sample SQL response, and linked metric decisions](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/workbench.png)
+
+*Interactive documentation example. Viewer screenshots use illustrative data and fictional model labels.*
+
+## An eval is two files
+
+| File | What you write | Who reads it |
+| --- | --- | --- |
+| `prompt.md` | A natural, focused task | Candidate agent |
+| `judge.md` | Named metrics and pass/fail criteria | Judge agent |
+| `eval.ts` *(optional)* | Workspace preparation and early stopping | Host |
+
+**`evals/ask-dialect/prompt.md`**
 
-A typed prompt-plus-LLM-judge SDK for Bun. Run agents in isolated containers,
-record their work, grade the evidence, and explore the results in a local viewer.
+```md
+Write a SQL query for the ten most recent orders for a customer.
+```
 
-## Install
+**`evals/ask-dialect/judge.md`**
 
-Requires Bun 1.4.2 or later, Docker, and an authenticated OpenCode installation.
-The agent runtime currently uses OpenCode `0.0.0-beta-19296`.
+```md
+# Requests the SQL dialect
 
-```sh
-bun add --exact @hona/openeval
+## Metric: asked_dialect — Asks for the SQL dialect
+
+Pass when the agent asks which database or SQL dialect is in use.
+Fail when it assumes a dialect without asking. Asking alongside a draft counts.
+
+## Metric: safe_parameters — Uses bound parameters
+
+Pass when the proposed query uses a bound customer-ID parameter and explains
+how to supply its value. Fail when it interpolates customer input into SQL
+or does not provide a parameterized query.
 ```
 
-## Declare a benchmark
+| Recorded response | Asks for dialect | Bound parameters |
+| --- | --- | --- |
+| Asks which DB; provides a bound-parameter draft | **1** | **1** |
+| Assumes PostgreSQL; uses `$1` | **0** | **1** |
+| Only asks which database | **1** | **0** |
+| Required recording is unavailable | **null** | **null** |
+
+→ [Write good rubrics](https://openev.al/docs/rubrics/) · [Download the SQL starter](https://openev.al/starter.zip)
+
+## Choose models. Run. Inspect.
 
-```text
-my-benchmark/
-  benchmark.ts
-  evals/
-    ask-dialect/
-      prompt.md
-      judge.md
+Requires **Bun 1.4.2+**, **Docker**, and connected models in **OpenCode**.
+
+```sh
+bun add --exact @hona/openeval
 ```
 
-`benchmark.ts`:
+**`benchmark.ts`** — replace the model references with your connected models:
 
 ```ts
 import type { Benchmark } from "@hona/openeval";
@@ -35,99 +80,98 @@ export default {
 } satisfies Benchmark;
 ```
 
-Replace the model references with models connected in OpenCode.
-
-`evals/ask-dialect/prompt.md`:
-
-```md
-Write a SQL query for the ten most recent orders for a customer.
+```sh
+bunx --bun @hona/openeval image
+bunx --bun @hona/openeval plan --only-eval ask-dialect
+bunx --bun @hona/openeval run --only-repetition 1
+bunx --bun @hona/openeval view
 ```
 
-`evals/ask-dialect/judge.md`:
+The viewer opens at **http://127.0.0.1:4173**. `run` resumes the same aggregate;
+scope flags select work while retaining existing scores.
 
-```md
-# Requests the SQL dialect
+To remove a model from an existing aggregate, remove its entry from
+`benchmark.ts`, then retire its active selections:
 
-## Metric: asked_dialect — Asks which SQL dialect to use
+```sh
+bunx --bun @hona/openeval snapshot ./results/RUN before-model-removal
+bunx --bun @hona/openeval remove-models ./results/RUN --model provider/retired-model
+```
 
-Pass when the agent asks which database or SQL dialect is in use.
-Fail when it assumes a dialect without asking. Asking alongside a draft counts.
+This retains the model's recorded executions, judgments, and artifacts. It
+does not run candidates or judges, and requires a stopped benchmark run.
+Models still declared in `benchmark.ts` can be added back by a later `run`.
+
+```mermaid
+flowchart LR
+  P["prompt.md"] --> C["Isolated candidate"] --> E["Recording"]
+  J["judge.md"] --> G["Judge + citations"]
+  E --> G --> S["Metric scores"] --> V["Results viewer"]
 ```
 
-Prompts are sent verbatim. Each metric is `0`, `1`, or `null` when evidence is
-insufficient. The judge supplies citations to the recording. A shared native
-judge agent supplies the evidence and submission protocol; rubrics contain the
-task-specific criteria. See [JUDGING.md](JUDGING.md).
+## See what earned the score
 
-## Run and inspect
+![Model scores, completed checks, runtime, and cost in the results viewer](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/results.png)
 
-From your benchmark directory:
+| Capability | What you get | Guide |
+| --- | --- | --- |
+| Multiple metrics | Independent decisions from one recording | [Rubrics](https://openev.al/docs/rubrics/) |
+| Controlled workspaces | Readable files, pinned Git inputs, preparation | [Workspaces](https://openev.al/docs/workspaces/) |
+| Small batches | Eval, model, repetition, and cost controls | [Running](https://openev.al/docs/running/) |
+| Transparent scores | Equal eval weights; bounds for unresolved checks | [Scoring](https://openev.al/docs/scoring/) |
+| Evidence inspection | Sessions, tool results, artifacts, and citations | [Evidence](https://openev.al/docs/evidence/) |
+| Rejudging | New judgments from retained, immutable recordings | [Evidence](https://openev.al/docs/evidence/#revise) |
 
-```sh
-bunx --bun @hona/openeval image
-bunx --bun @hona/openeval plan
-bunx --bun @hona/openeval run
-bunx --bun @hona/openeval view
-```
+<details>
+<summary><strong>Inspect a judgment and its evidence</strong></summary>
 
-The viewer opens at `http://127.0.0.1:4173`. Use `--benchmark <directory>` to point
-any command at another benchmark, or `--port <port>` for another viewer port.
+![SQL eval drilldown with individual metric decisions and evidence links](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/judgment.png)
 
-`run` resumes the current aggregate and reuses unchanged work. `--new` starts a
-separate result. Repeat `--only-eval`, `--only-model`, or `--only-repetition` to
-execute a small scope while retaining the full aggregate. `--max-cost <usd>`
-sets a scheduling budget for the invocation. Run `openeval --help` for commands.
+</details>
 
-Scores average repetitions per metric, metrics per eval, then evals equally.
-Unresolved checks produce completion bounds until the final percentage is known.
-The viewer provides metric drilldowns and recorded candidate and judge sessions.
-Elapsed time measures active execution intervals, counting overlapping work once.
+<details>
+<summary><strong>Watch candidate and judge work in the live queue</strong></summary>
 
-## Prepare a workspace
+![Live queue with separate execution and judging stages](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/queue.png)
 
-An eval's optional `workspace/` directory supplies files. An optional `eval.ts`
-can declare a pinned Git repository or readable revision overlays and preparation
-commands:
+</details>
+
+## Use the SDK
 
 ```ts
-import type { Eval } from "@hona/openeval";
+import { runBenchmark } from "@hona/openeval";
 
-export default {
-  prepare: [{ cwd: ".", argv: ["bun", "install", "--frozen-lockfile"] }],
-} satisfies Eval;
+await runBenchmark("./my-benchmark", {
+  onlyEvals: ["ask-dialect"],
+  onlyRepetitions: [1],
+});
 ```
 
-Preparation runs before recording the initial candidate workspace. Candidate
-containers receive project inputs, never judge rubrics or evaluator storage.
-Candidates finish naturally, meet an opted-in irreversible judge decision, or
-time out after at most 45 minutes. Finalized recordings remain immutable.
+## Write evals with an agent
 
-## SDK
+Use the public [Eval Writing skill](https://github.com/Hona/openeval/tree/main/.opencode/skills/eval-writing)
+to turn a real failure into an eval, review a rubric, or investigate misleading
+scores. It guides an agent through concrete false-pass/false-failure examples,
+accepted alternatives, evidence requirements, and human-reviewed calibration.
 
-```ts
-import { runBenchmark, serveResults } from "@hona/openeval";
-
-await runBenchmark("./my-benchmark");
-```
+Copy the whole `.opencode/skills/eval-writing/` directory, including `references/`,
+into the same path in your project. For global use, copy it to
+`~/.config/opencode/skills/eval-writing/`. Then run **`/eval-writing`** in OpenCode.
 
-The package also exports types, result readers, snapshot and rejudge functions,
-and reusable view/session data through `@hona/openeval/types`, `/results`,
-`/view`, and `/session`.
+> Use eval-writing to review this task and rubric. Show me the strongest false
+> pass and false failure, then propose the smallest improvement.
 
-## Develop and release
+The skill includes a framework-neutral workflow, fictional coaching examples,
+an OpenEval-specific reference, and a broad public-research guide.
 
-```sh
-bun install --frozen-lockfile
-bun run typecheck
-bun test
-bun run release:pack
-bun run release:verify
-```
+## Develop
 
-The private benchmark client is maintained in a separate repository. It installs
-an exact published SDK version. This repository contains only the reusable SDK,
-CLI, viewer, and development inputs. See [RELEASING.md](https://github.com/Hona/openeval/blob/main/RELEASING.md).
+| Command | Purpose |
+| --- | --- |
+| `bun run site:dev` | Landing page and docs with hot reload on port 4176 |
+| `bun run site:build && bun run site:verify` | Prerender pages and verify links and starter files |
+| `bun run typecheck && bun test` | Local SDK checks; no live models |
+| `bun run release:pack && bun run release:verify` | Verify the actual npm archive in a separate consumer |
 
-OpenEval is MIT licensed. The vendored session UI retains its upstream MIT
-license and revision in `vendor/session-ui`. Built viewer distributions include
-third-party license notices.
+- [Release guide](https://github.com/Hona/openeval/blob/main/RELEASING.md) · [Judge protocol](JUDGING.md)
+- MIT licensed. The viewer includes upstream third-party license notices.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hona/openeval",
-  "version": "0.2.0",
+  "version": "0.2.2",
   "description": "Typed prompt-and-judge evaluations with isolated agents, recorded evidence, and a results viewer",
   "license": "MIT",
   "repository": {

package/src/app/load-benchmark.ts

@@ -28,7 +28,7 @@ const positive = (value: unknown, fallback: number, label: string) => {
 export const modelRef = (value: unknown): ModelRef => {
   if (
     typeof value !== "string" ||
-    !/^[\w.-]+\/[^\s/#]+(?:#[\w.-]+)?$/.test(value)
+    !/^[\w.-]+\/[^\s/#]+(?:\/[^\s/#]+)*(?:#[\w.-]+)?$/.test(value)
   )
     throw new Error(`Invalid model reference: ${String(value)}`);
   return value as ModelRef;

package/src/app/remove-models.ts

@@ -0,0 +1,60 @@
+import { resolve } from "node:path";
+import type { ModelRef } from "../types";
+import { Results } from "../infra/sqlite";
+import { modelRef } from "./load-benchmark";
+import { readBenchmarkRun } from "./read-run";
+import { finishBenchmark } from "./run-benchmark";
+
+/** Retire active model selections while preserving every execution and artifact. */
+export async function removeModels(
+  directory: string,
+  models: readonly ModelRef[],
+) {
+  const root = resolve(directory);
+  const removed = [...new Set(models.map(modelRef))];
+  if (!removed.length) throw new Error("Select models to remove");
+  const before = readBenchmarkRun(root).benchmark;
+
+  using results = new Results(resolve(root, "runner.db"));
+  return results.transaction(() => {
+    const current = results.benchmark!;
+    if (current.id !== before.id || current.mergedInto)
+      throw new Error(
+        "Benchmark was replaced or merged; use its current aggregate",
+      );
+    if (current.state === "running")
+      throw new Error(
+        "Cannot remove models while the benchmark run is running",
+      );
+    if (removed.some((model) => !current.definition.models.includes(model)))
+      throw new Error("Select models present in this benchmark run");
+    const remaining = current.definition.models.filter(
+      (model) => !removed.includes(model),
+    );
+    if (!remaining.length)
+      throw new Error("Keep at least one model in the benchmark run");
+    const retired = results
+      .slots()
+      .filter((slot) => slot.active && removed.includes(slot.model));
+    for (const slot of retired) results.select({ ...slot, active: false });
+    const definition = { ...current.definition, models: remaining };
+    results.saveBenchmark({
+      ...current,
+      definition,
+      execution: { ...current.execution, deferred: 0 },
+    });
+    const benchmark = finishBenchmark({
+      directory: root,
+      definition,
+      runtime: current.runtime,
+      results,
+    });
+    results.saveBenchmark(benchmark, true);
+    return {
+      directory: root,
+      removed,
+      retiredSlots: retired.length,
+      benchmark,
+    };
+  });
+}

package/src/cli.ts

@@ -6,6 +6,7 @@ import {
   loadBenchmark,
   buildImage,
   addModels,
+  removeModels,
   retryEvalRun,
   judgeRun,
   serveResults,
@@ -39,6 +40,7 @@ Commands:
   snapshot <run> <name>  Export a score snapshot
   merge-runs <to> <from> Merge results into one aggregate
   add-models <run>       Add models with repeated --model flags
+  remove-models <run>    Remove active models while retaining their evidence
   retry <run> <eval-run> Retry a candidate execution
   rejudge <run> <eval-run> Judge the saved evidence again
 
@@ -46,7 +48,7 @@ Options:
   --benchmark <dir>      Benchmark directory (default: current directory)
   --run <dir>            Resume a specific result directory
   --new                 Start a new result
-  --model <provider/id>  Add a model to the benchmark
+  --model <provider/id>  Model to add or remove (repeatable)
   --only-model <ref>     Execute only this model (repeatable)
   --only-eval <id>       Execute only this eval (repeatable)
   --only-repetition <n>  Execute only this repetition (repeatable)
@@ -125,6 +127,16 @@ Requires Bun 1.4.2+, Docker, and an authenticated OpenCode installation.`);
     if (!args[1])
       throw new Error("add-models requires a benchmark run directory");
     console.log((await addModels(resolve(args[1]), models())).directory);
+  } else if (command === "remove-models") {
+    if (!args[1])
+      throw new Error("remove-models requires a benchmark run directory");
+    const result = await removeModels(resolve(args[1]), models());
+    console.log(JSON.stringify({
+      directory: result.directory,
+      removed: result.removed,
+      retiredSlots: result.retiredSlots,
+      status: result.benchmark.state,
+    }, null, 2));
   } else if (command === "retry" || command === "rejudge") {
     if (!args[1] || !args[2])
       throw new Error(
@@ -137,7 +149,7 @@ Requires Bun 1.4.2+, Docker, and an authenticated OpenCode installation.`);
     console.log(JSON.stringify(result, null, 2));
   } else
     throw new Error(
-      "Commands: image, plan, run, view, add-models, merge-runs, retry, rejudge, snapshot",
+      "Commands: image, plan, run, view, add-models, remove-models, merge-runs, retry, rejudge, snapshot",
     );
 } catch (error) {
   console.error(error instanceof Error ? error.message : String(error));

package/src/index.ts

@@ -38,6 +38,7 @@ export {
   type RunBenchmarkOptions,
 } from "./app/run-benchmark";
 export { addModels } from "./app/add-models";
+export { removeModels } from "./app/remove-models";
 export type { CostEstimate } from "./app/cost-plan";
 export { mergeBenchmarkRuns } from "./app/merge-runs";
 export { retryEvalRun } from "./app/retry-run";