@hona/openeval 0.2.0 → 0.2.1

package/README.md

@@ -1,29 +1,74 @@
-# OpenEval
+<div align="center">
+  <h1>OpenEval</h1>
+  <p><strong>Write the task. Judge the evidence.</strong></p>
+  <p>Prompt-and-rubric evaluations for agents. Typed declarations, isolated runs, inspectable scores.</p>
+  <p>
+    <a href="https://openev.al">Website</a> ·
+    <a href="https://openeval.pages.dev">Live preview</a> ·
+    <a href="https://openev.al/docs/quickstart/">Write your first eval</a> ·
+    <a href="https://openev.al/docs/reference/">CLI reference</a> ·
+    <a href="https://www.npmjs.com/package/@hona/openeval">npm</a>
+  </p>
+  <p>
+    <a href="https://www.npmjs.com/package/@hona/openeval"><img src="https://img.shields.io/npm/v/%40hona%2Fopeneval?style=flat-square&color=66d38a" alt="npm version"></a>
+    <a href="https://bun.com"><img src="https://img.shields.io/badge/Bun-1.4.2%2B-f9f1e1?style=flat-square" alt="Bun 1.4.2 or later"></a>
+    <a href="https://github.com/Hona/openeval/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-66d38a?style=flat-square" alt="MIT license"></a>
+  </p>
+</div>
+
+![The OpenEval workbench: rubric source, a sample SQL response, and linked metric decisions](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/workbench.png)
+
+*Interactive documentation example. Viewer screenshots use illustrative data and fictional model labels.*
+
+## An eval is two files
+
+| File | What you write | Who reads it |
+| --- | --- | --- |
+| `prompt.md` | A natural, focused task | Candidate agent |
+| `judge.md` | Named metrics and pass/fail criteria | Judge agent |
+| `eval.ts` *(optional)* | Workspace preparation and early stopping | Host |
+
+**`evals/ask-dialect/prompt.md`**
 
-A typed prompt-plus-LLM-judge SDK for Bun. Run agents in isolated containers,
-record their work, grade the evidence, and explore the results in a local viewer.
+```md
+Write a SQL query for the ten most recent orders for a customer.
+```
 
-## Install
+**`evals/ask-dialect/judge.md`**
 
-Requires Bun 1.4.2 or later, Docker, and an authenticated OpenCode installation.
-The agent runtime currently uses OpenCode `0.0.0-beta-19296`.
+```md
+# Requests the SQL dialect
 
-```sh
-bun add --exact @hona/openeval
+## Metric: asked_dialect — Asks for the SQL dialect
+
+Pass when the agent asks which database or SQL dialect is in use.
+Fail when it assumes a dialect without asking. Asking alongside a draft counts.
+
+## Metric: safe_parameters — Uses bound parameters
+
+Pass when the proposed query uses a bound customer-ID parameter and explains
+how to supply its value. Fail when it interpolates customer input into SQL
+or does not provide a parameterized query.
 ```
 
-## Declare a benchmark
+| Recorded response | Asks for dialect | Bound parameters |
+| --- | --- | --- |
+| Asks which DB; provides a bound-parameter draft | **1** | **1** |
+| Assumes PostgreSQL; uses `$1` | **0** | **1** |
+| Only asks which database | **1** | **0** |
+| Required recording is unavailable | **null** | **null** |
+
+→ [Write good rubrics](https://openev.al/docs/rubrics/) · [Download the SQL starter](https://openev.al/starter.zip)
+
+## Choose models. Run. Inspect.
 
-```text
-my-benchmark/
-  benchmark.ts
-  evals/
-    ask-dialect/
-      prompt.md
-      judge.md
+Requires **Bun 1.4.2+**, **Docker**, and connected models in **OpenCode**.
+
+```sh
+bun add --exact @hona/openeval
 ```
 
-`benchmark.ts`:
+**`benchmark.ts`** — replace the model references with your connected models:
 
 ```ts
 import type { Benchmark } from "@hona/openeval";
@@ -35,99 +80,86 @@ export default {
 } satisfies Benchmark;
 ```
 
-Replace the model references with models connected in OpenCode.
-
-`evals/ask-dialect/prompt.md`:
-
-```md
-Write a SQL query for the ten most recent orders for a customer.
+```sh
+bunx --bun @hona/openeval image
+bunx --bun @hona/openeval plan --only-eval ask-dialect
+bunx --bun @hona/openeval run --only-repetition 1
+bunx --bun @hona/openeval view
 ```
 
-`evals/ask-dialect/judge.md`:
-
-```md
-# Requests the SQL dialect
-
-## Metric: asked_dialect — Asks which SQL dialect to use
+The viewer opens at **http://127.0.0.1:4173**. `run` resumes the same aggregate;
+scope flags select work while retaining existing scores.
 
-Pass when the agent asks which database or SQL dialect is in use.
-Fail when it assumes a dialect without asking. Asking alongside a draft counts.
+```mermaid
+flowchart LR
+  P["prompt.md"] --> C["Isolated candidate"] --> E["Recording"]
+  J["judge.md"] --> G["Judge + citations"]
+  E --> G --> S["Metric scores"] --> V["Results viewer"]
 ```
 
-Prompts are sent verbatim. Each metric is `0`, `1`, or `null` when evidence is
-insufficient. The judge supplies citations to the recording. A shared native
-judge agent supplies the evidence and submission protocol; rubrics contain the
-task-specific criteria. See [JUDGING.md](JUDGING.md).
+## See what earned the score
 
-## Run and inspect
+![Model scores, completed checks, runtime, and cost in the results viewer](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/results.png)
 
-From your benchmark directory:
+| Capability | What you get | Guide |
+| --- | --- | --- |
+| Multiple metrics | Independent decisions from one recording | [Rubrics](https://openev.al/docs/rubrics/) |
+| Controlled workspaces | Readable files, pinned Git inputs, preparation | [Workspaces](https://openev.al/docs/workspaces/) |
+| Small batches | Eval, model, repetition, and cost controls | [Running](https://openev.al/docs/running/) |
+| Transparent scores | Equal eval weights; bounds for unresolved checks | [Scoring](https://openev.al/docs/scoring/) |
+| Evidence inspection | Sessions, tool results, artifacts, and citations | [Evidence](https://openev.al/docs/evidence/) |
+| Rejudging | New judgments from retained, immutable recordings | [Evidence](https://openev.al/docs/evidence/#revise) |
 
-```sh
-bunx --bun @hona/openeval image
-bunx --bun @hona/openeval plan
-bunx --bun @hona/openeval run
-bunx --bun @hona/openeval view
-```
+<details>
+<summary><strong>Inspect a judgment and its evidence</strong></summary>
 
-The viewer opens at `http://127.0.0.1:4173`. Use `--benchmark <directory>` to point
-any command at another benchmark, or `--port <port>` for another viewer port.
+![SQL eval drilldown with individual metric decisions and evidence links](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/judgment.png)
 
-`run` resumes the current aggregate and reuses unchanged work. `--new` starts a
-separate result. Repeat `--only-eval`, `--only-model`, or `--only-repetition` to
-execute a small scope while retaining the full aggregate. `--max-cost <usd>`
-sets a scheduling budget for the invocation. Run `openeval --help` for commands.
+</details>
 
-Scores average repetitions per metric, metrics per eval, then evals equally.
-Unresolved checks produce completion bounds until the final percentage is known.
-The viewer provides metric drilldowns and recorded candidate and judge sessions.
-Elapsed time measures active execution intervals, counting overlapping work once.
+<details>
+<summary><strong>Watch candidate and judge work in the live queue</strong></summary>
 
-## Prepare a workspace
+![Live queue with separate execution and judging stages](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/queue.png)
 
-An eval's optional `workspace/` directory supplies files. An optional `eval.ts`
-can declare a pinned Git repository or readable revision overlays and preparation
-commands:
+</details>
+
+## Use the SDK
 
 ```ts
-import type { Eval } from "@hona/openeval";
+import { runBenchmark } from "@hona/openeval";
 
-export default {
-  prepare: [{ cwd: ".", argv: ["bun", "install", "--frozen-lockfile"] }],
-} satisfies Eval;
+await runBenchmark("./my-benchmark", {
+  onlyEvals: ["ask-dialect"],
+  onlyRepetitions: [1],
+});
 ```
 
-Preparation runs before recording the initial candidate workspace. Candidate
-containers receive project inputs, never judge rubrics or evaluator storage.
-Candidates finish naturally, meet an opted-in irreversible judge decision, or
-time out after at most 45 minutes. Finalized recordings remain immutable.
+## Write evals with an agent
 
-## SDK
+Use the public [Eval Writing skill](https://github.com/Hona/openeval/tree/main/.opencode/skills/eval-writing)
+to turn a real failure into an eval, review a rubric, or investigate misleading
+scores. It guides an agent through concrete false-pass/false-failure examples,
+accepted alternatives, evidence requirements, and human-reviewed calibration.
 
-```ts
-import { runBenchmark, serveResults } from "@hona/openeval";
-
-await runBenchmark("./my-benchmark");
-```
+Copy the whole `.opencode/skills/eval-writing/` directory, including `references/`,
+into the same path in your project. For global use, copy it to
+`~/.config/opencode/skills/eval-writing/`. Then run **`/eval-writing`** in OpenCode.
 
-The package also exports types, result readers, snapshot and rejudge functions,
-and reusable view/session data through `@hona/openeval/types`, `/results`,
-`/view`, and `/session`.
+> Use eval-writing to review this task and rubric. Show me the strongest false
+> pass and false failure, then propose the smallest improvement.
 
-## Develop and release
+The skill includes a framework-neutral workflow, fictional coaching examples,
+an OpenEval-specific reference, and a broad public-research guide.
 
-```sh
-bun install --frozen-lockfile
-bun run typecheck
-bun test
-bun run release:pack
-bun run release:verify
-```
+## Develop
 
-The private benchmark client is maintained in a separate repository. It installs
-an exact published SDK version. This repository contains only the reusable SDK,
-CLI, viewer, and development inputs. See [RELEASING.md](https://github.com/Hona/openeval/blob/main/RELEASING.md).
+| Command | Purpose |
+| --- | --- |
+| `bun run site:dev` | Landing page and docs with hot reload on port 4176 |
+| `bun run site:build && bun run site:verify` | Prerender pages and verify links and starter files |
+| `bun run typecheck && bun test` | Local SDK checks; no live models |
+| `bun run release:pack && bun run release:verify` | Verify the actual npm archive in a separate consumer |
 
-OpenEval is MIT licensed. The vendored session UI retains its upstream MIT
-license and revision in `vendor/session-ui`. Built viewer distributions include
-third-party license notices.
+- [Release guide](https://github.com/Hona/openeval/blob/main/RELEASING.md) · [Judge protocol](JUDGING.md)
+- MIT licensed. The viewer includes upstream third-party license notices.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hona/openeval",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Typed prompt-and-judge evaluations with isolated agents, recorded evidence, and a results viewer",
   "license": "MIT",
   "repository": {

package/src/app/load-benchmark.ts

@@ -28,7 +28,7 @@ const positive = (value: unknown, fallback: number, label: string) => {
 export const modelRef = (value: unknown): ModelRef => {
   if (
     typeof value !== "string" ||
-    !/^[\w.-]+\/[^\s/#]+(?:#[\w.-]+)?$/.test(value)
+    !/^[\w.-]+\/[^\s/#]+(?:\/[^\s/#]+)*(?:#[\w.-]+)?$/.test(value)
   )
     throw new Error(`Invalid model reference: ${String(value)}`);
   return value as ModelRef;