@tryinget/pi-evalset-lab 0.2.0

package/CHANGELOG.md

@@ -0,0 +1,60 @@
+---
+summary: "Changelog for @tryinget/pi-evalset-lab."
+read_when:
+  - "Preparing a release or reviewing history."
+system4d:
+  container: "Release log for this extension package."
+  compass: "Track meaningful deltas per version."
+  engine: "Document changes at release boundaries."
+  fog: "Legacy standalone history predates monorepo canonicalization."
+---
+
+# Changelog
+
+All notable changes to this project should be documented here.
+
+## [Unreleased]
+
+### Changed
+
+- Canonicalized the legacy standalone `pi-evalset-lab` package into the `pi-extensions` monorepo at `packages/pi-evalset-lab`.
+- Switched npm identity to scoped monorepo package name `@tryinget/pi-evalset-lab` while preserving legacy runtime version `0.2.0`.
+- Moved release automation to root-managed component metadata (`pi-evalset-lab`).
+- Archived the former standalone working copy to `~/programming/pi-extensions/pi-evalset-lab-final-archive.tar.gz` and removed it after validation; no legacy Pi session-history directory existed to relocate.
+
+## [0.2.0](https://github.com/tryingET/pi-evalset-lab/compare/v0.1.0...v0.2.0) (2026-02-17)
+
+### Features
+
+- **evalset:** add JSON to HTML report export helper ([802e7d2](https://github.com/tryingET/pi-evalset-lab/commit/802e7d2a13a44205e519e0f0c778788fd093340f))
+
+### Added
+
+- Added `/evalset` MVP command with subcommands:
+  - `init` to generate a starter fixed-task-set dataset
+  - `run` to evaluate one variant against a dataset
+  - `compare` to evaluate baseline vs candidate system prompts
+- Added example files in `examples/`:
+  - `fixed-task-set.json`
+  - `fixed-task-set-v2.json`
+  - `fixed-task-set-v3.json`
+  - `evalset-compare-sample-embedded.html`
+  - `evalset-compare-sample.png`
+  - `system-baseline.txt`
+  - `system-candidate.txt`
+- Added report output support to `.evalset/reports/*.json` with per-case and aggregate metrics.
+- Added run identity metadata to reports (`runId`, `datasetHash`, `casesHash`, `variantHash`).
+- Reduced session message payload size by storing only lightweight report metadata instead of full report bodies.
+- Added `scripts/export-evalset-report-html.mjs` and `npm run evalset:export-html` for repeatable JSON -> static HTML report exports.
+
+### Changed
+
+- Clarified `/evalset` invocation docs: use `pi -p` (or `pi -e ... -p`) for non-interactive runs; `/evalset` is not a standalone shell binary.
+- Added the same non-interactive invocation note to `/evalset help` output.
+- Declared publish-time runtime artifacts with `package.json` `files` whitelist and documented peer/runtime dependency behavior in README.
+
+## [0.1.0] - 2026-02-08
+
+### Added
+
+- Initial production-ready scaffold generated from template v2.

package/LICENSE

@@ -0,0 +1,78 @@
+MIT License (with OpenAI/Anthropic/xAI/PRC Frontier Labs Rider)
+
+Copyright (c) contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+ADDITIONAL RIDER / RESTRICTION (OpenAI / Anthropic / xAI / PRC Frontier Labs):
+
+This rider is part of the "conditions" of this License. In the event of any
+conflict between this rider and any other portion of this License, this rider
+controls.
+
+"Restricted Parties" means OpenAI, L.L.C.; Anthropic, PBC; xAI Corp.; and the
+following PRC frontier AI labs and related entities: Baidu (including ERNIE-
+related entities), Alibaba and Alibaba Cloud (including Qwen-related entities),
+Tencent (including Hunyuan-related entities), ByteDance (including Doubao/Seed-
+related entities), DeepSeek, Zhipu AI, Moonshot AI, MiniMax, 01.AI
+(Lingyiwanwu), and iFlyTek. "Restricted Parties" also includes any of their
+respective Affiliates and any person or entity acting directly or indirectly
+on behalf of, for the benefit of, or under the direction of any of the
+foregoing (including any officer, director, employee, contractor, agent,
+consultant, service provider, or representative).
+
+Notwithstanding any other provision of this License, no rights are granted to
+any Restricted Party. Any purported license, sublicense, assignment, transfer,
+or other permission to any Restricted Party is null and void absent the
+express prior written permission of the copyright holders.
+
+You may not provide, disclose, distribute, sublicense, sell, lease, lend,
+host, make available, or otherwise permit access to the Software or any
+derivative work of the Software (as defined in applicable copyright law)
+(collectively, "Derivative Works") to or for any Restricted Party.
+
+For purposes of this rider, "use" includes, without limitation: copying,
+modifying, merging, publishing, distributing, sublicensing, selling,
+transferring, making available, hosting, deploying, executing, benchmarking,
+testing, analyzing, indexing, or incorporating the Software or any Derivative
+Works into any dataset, training corpus, evaluation harness, or pipeline for
+machine learning or other automated systems.
+
+This rider applies to the Software and all Derivative Works. As a condition of
+use, you agree that this rider is a precondition to exercising any rights
+under this License, and you agree that any distribution of the Software or any
+Derivative Works must include this rider provision unmodified.
+
+Any breach of this rider automatically and immediately terminates the
+permissions granted by this License. Upon termination, you must immediately
+cease all use and distribution of the Software and any Derivative Works and
+destroy all copies under your control.
+
+You agree that a breach of this rider would cause irreparable harm and that
+the copyright holders may seek injunctive or other equitable relief to enforce
+this rider, in addition to any other remedies available at law. To the maximum
+extent permitted by applicable law, the prevailing party in any action to
+enforce this rider shall be entitled to recover reasonable attorneys' fees and
+costs.
+
+For purposes of this rider, "Affiliate" means any entity that directly or
+indirectly controls, is controlled by, or is under common control with the
+specified party. "Control" means ownership of more than 50% of the voting
+securities or other ownership interest, or the power to direct management or
+policies by contract or otherwise.
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,141 @@
+---
+summary: "Overview and quickstart for @tryinget/pi-evalset-lab."
+read_when:
+  - "Starting work in this package workspace."
+  - "Using /evalset run or /evalset compare."
+system4d:
+  container: "Monorepo package for a pi fixed-task-set evaluation extension."
+  compass: "Keep prompt/system comparisons small, reproducible, and easy to inspect."
+  engine: "Define dataset -> run or compare variants -> export JSON/HTML report -> review deltas."
+  fog: "Model/provider nondeterminism can make brittle checks noisy."
+---
+
+# @tryinget/pi-evalset-lab
+
+Monorepo package for fixed-task-set eval workflows in Pi (`/evalset run|compare`) with reproducible JSON reports and static HTML export.
+
+- Workspace path: `packages/pi-evalset-lab`
+- Release component key: `pi-evalset-lab`
+- Former legacy standalone source: `~/programming/pi-extensions/pi-evalset-lab`
+- Canonical package status: canonicalized here; the legacy repo was archived to `~/programming/pi-extensions/pi-evalset-lab-final-archive.tar.gz` and removed after validation.
+- Session-history migration: no legacy Pi session-history directory existed for the old path, so relocation was recorded as `skip-no-history`.
+
+Primary category fit: **Model & Prompt Management**, **Review & Quality Loops**, **UX & Observability**, **Safety & Governance**.
+
+## Runtime dependencies and packaged files
+
+This package expects Pi host runtime APIs and declares them as `peerDependencies`:
+
+- `@mariozechner/pi-coding-agent`
+- `@mariozechner/pi-ai`
+
+The npm package uses a `files` whitelist so required runtime artifacts are explicitly included:
+
+- `extensions/evalset.ts`
+- `prompts/`
+- `examples/` (sample datasets + sample report UI)
+- `scripts/export-evalset-report-html.mjs`
+
+## Quickstart
+
+Install package dependencies for local validation:
+
+```bash
+cd packages/pi-evalset-lab
+npm install
+npm run check
+```
+
+Install into Pi from the package directory containing `package.json`:
+
+```bash
+pi install /absolute/path/to/pi-extensions/packages/pi-evalset-lab
+# then in Pi: /reload
+```
+
+For ad hoc source testing from this package directory:
+
+```bash
+pi -e ./extensions/evalset.ts
+```
+
+## evalset command
+
+```bash
+/evalset help
+/evalset init [dataset-path] [--force]
+/evalset run <dataset.json> [--system-file <path>] [--system-text <text>] [--variant <name>] [--max-cases <n>] [--temperature <n>] [--out <report.json>]
+/evalset compare <dataset.json> <baseline-system.txt> <candidate-system.txt> [--baseline-name <name>] [--candidate-name <name>] [--max-cases <n>] [--temperature <n>] [--out <report.json>]
+```
+
+`/evalset` is a Pi slash command, not a shell executable.
+
+Interactive mode:
+
+```bash
+pi -e ./extensions/evalset.ts
+# then inside Pi:
+/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt
+```
+
+Non-interactive mode:
+
+```bash
+pi -e ./extensions/evalset.ts -p "/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt"
+# or, if installed/enabled:
+pi -p "/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt"
+```
+
+Interactive sessions use Pi UI hooks (`ctx.ui`) for status/notify updates. Non-interactive `-p` mode skips those UI calls when `ctx.hasUI === false`.
+
+## Included datasets and sample output
+
+- `examples/fixed-task-set.json` — tiny smoke set (3 cases)
+- `examples/fixed-task-set-v2.json` — larger first pass set
+- `examples/fixed-task-set-v3.json` — less brittle checks (recommended)
+- `examples/evalset-compare-sample-embedded.html` — self-contained report UI with embedded compare JSON
+- `examples/evalset-compare-sample.png` — screenshot preview of that HTML report
+- `examples/system-baseline.txt` and `examples/system-candidate.txt` — compare inputs
+
+Preview:
+
+![Evalset compare sample screenshot](examples/evalset-compare-sample.png)
+
+Reports are written to explicit `--out <path>` when provided, otherwise `.evalset/reports/*.json` under the current project directory.
+
+Each report includes run identity metadata (`runId`, `datasetHash`, `casesHash`, and variant hashes). Session messages keep lightweight report metadata only, not full report bodies.
+
+## Export report JSON to static HTML
+
+```bash
+npm run evalset:export-html -- --in .evalset/reports/compare-your-dataset-YYYYMMDDTHHMMSS.json
+# optional:
+npm run evalset:export-html -- --in .evalset/reports/run-your-dataset-YYYYMMDDTHHMMSS.json --out .evalset/reports/run-your-dataset.html --title "Evalset run report"
+```
+
+Script: [scripts/export-evalset-report-html.mjs](scripts/export-evalset-report-html.mjs)
+
+## Validation and release checks
+
+Package-local validation:
+
+```bash
+npm run check
+npm run release:check:quick
+```
+
+Monorepo-scoped validation:
+
+```bash
+cd ../..
+bash ./scripts/package-quality-gate.sh ci packages/pi-evalset-lab
+node ./scripts/release-components.mjs validate
+```
+
+Release metadata is root-managed through `x-pi-template.releaseConfigMode=component` and component key `pi-evalset-lab`.
+
+The scoped package `@tryinget/pi-evalset-lab` is the canonical npm identity for future releases. The old unscoped `pi-evalset-lab@0.2.0` package remains historical registry state, not the canonical development target.
+
+## Optional core hooks (future, not required)
+
+This extension works today without Pi core changes. Optional hardening could include stable agent-level lineage IDs, explicit reproducibility metadata in `pi-ai`, shared provider payload hashing, or a headless agent-eval API for tool-heavy/full agent-loop benchmark runs.

package/examples/evalset-compare-sample-embedded.html

@@ -0,0 +1,142 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <title>Evalset compare sample (embedded)</title>
+  <style>
+    :root {
+      --bg: #0b1020;
+      --panel: #111a33;
+      --line: #24335f;
+      --txt: #e8eeff;
+      --muted: #9fb0d8;
+      --ok: #2ecc71;
+      --bad: #ff6b6b;
+      --same: #7f8ea3;
+      --imp: #49dcb1;
+      --reg: #ff8f70;
+    }
+    * { box-sizing: border-box; }
+    body { margin: 0; padding: 24px; font-family: ui-sans-serif, system-ui, -apple-system, Segoe UI, Roboto, Ubuntu; background: var(--bg); color: var(--txt); }
+    .wrap { max-width: 1220px; margin: 0 auto; }
+    .panel { background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 14px; margin-bottom: 14px; }
+    .muted { color: var(--muted); font-size: 0.92rem; }
+    .grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 10px; }
+    .card { background: #0e1630; border: 1px solid #2a3a66; border-radius: 10px; padding: 10px; }
+    .label { color: var(--muted); text-transform: uppercase; letter-spacing: .04em; font-size: .75rem; }
+    .val { margin-top: 4px; font-size: 1.06rem; font-weight: 600; }
+    table { width: 100%; border-collapse: collapse; font-size: .91rem; }
+    th, td { text-align: left; padding: 10px 8px; border-bottom: 1px solid var(--line); vertical-align: top; }
+    th { color: var(--muted); }
+    .pill { display: inline-block; border-radius: 999px; padding: 2px 10px; font-size: .74rem; font-weight: 700; border: 1px solid transparent; }
+    .ok { background: rgba(46, 204, 113, .14); color: #9df7c5; border-color: rgba(46, 204, 113, .45); }
+    .bad { background: rgba(255, 107, 107, .14); color: #ffc2c2; border-color: rgba(255, 107, 107, .45); }
+    .same { background: rgba(127, 142, 163, .14); color: #d4dcf2; border-color: rgba(127, 142, 163, .45); }
+    .imp { background: rgba(73, 220, 177, .14); color: #abf6df; border-color: rgba(73, 220, 177, .45); }
+    .reg { background: rgba(255, 143, 112, .14); color: #ffd0c2; border-color: rgba(255, 143, 112, .45); }
+    .meta { color: var(--muted); font-size: .8rem; margin-top: 6px; }
+    pre { white-space: pre-wrap; background: #0d152b; border: 1px solid #2a3a66; border-radius: 8px; padding: 8px; max-height: 220px; overflow: auto; }
+    details summary { cursor: pointer; color: #9dc2ff; }
+    .split { display: grid; grid-template-columns: 1fr 1fr; gap: 10px; }
+    @media (max-width: 980px) { .split { grid-template-columns: 1fr; } }
+  </style>
+</head>
+<body>
+  <div class="wrap">
+    <div class="panel">
+      <h1 style="margin:0 0 8px; font-size:1.32rem;">Evalset compare sample (embedded data)</h1>
+      <div class="muted" id="meta"></div>
+    </div>
+
+    <div class="panel grid" id="summary"></div>
+
+    <div class="panel">
+      <h2 style="margin:0 0 10px; font-size:1.08rem;">Case-by-case</h2>
+      <table>
+        <thead>
+          <tr>
+            <th>#</th>
+            <th>Case</th>
+            <th>Baseline</th>
+            <th>Candidate</th>
+            <th>Outcome</th>
+            <th>Details</th>
+          </tr>
+        </thead>
+        <tbody id="rows"></tbody>
+      </table>
+    </div>
+  </div>
+
+  <script id="report-data" type="application/json">{"kind":"evalset-compare","createdAt":"2026-02-16T14:41:23.888Z","run":{"runId":"471393b2-0613-41d1-8bd3-e4cc4306d65a","startedAt":"2026-02-16T14:39:40.498Z","finishedAt":"2026-02-16T14:41:23.888Z","modelKey":"openai-codex/gpt-5.3-codex","temperature":null,"datasetHash":"2851c4b6d9e9fd4961be234d2b4602431a46f4adb305eb20b0dd3120e8df64dc","casesHash":"49ca1c743474d713ee966235b03e224e861726e8b56c3aacb625227af7161aca","baselineRunId":"c3845934-4277-4915-a8d2-356e691349d7","candidateRunId":"f68bccd1-4bff-4145-9f09-d3eca49aecd8","baselineVariantHash":"df1af004a99a2cadc51b1d28e37e5d6fdceb0ae765519a36964e6fc9f78a2c56","candidateVariantHash":"7e4c4da97cf0d1ea77c0f613c44752a9d2d464cd0c1f26e7f86c69c546569a9d"},"dataset":{"name":"maintainer-clarity-v3","path":"/home/lightningralf/programming/pi-extensions/pi-evalset-lab/examples/fixed-task-set-v3.json"},"model":{"provider":"openai-codex","id":"gpt-5.3-codex","api":"openai-codex-responses"},"baseline":{"kind":"evalset-run","createdAt":"2026-02-16T14:40:35.056Z","run":{"runId":"c3845934-4277-4915-a8d2-356e691349d7","startedAt":"2026-02-16T14:39:40.498Z","finishedAt":"2026-02-16T14:40:35.056Z","modelKey":"openai-codex/gpt-5.3-codex","temperature":null,"datasetHash":"2851c4b6d9e9fd4961be234d2b4602431a46f4adb305eb20b0dd3120e8df64dc","casesHash":"49ca1c743474d713ee966235b03e224e861726e8b56c3aacb625227af7161aca","variantHash":"df1af004a99a2cadc51b1d28e37e5d6fdceb0ae765519a36964e6fc9f78a2c56"},"dataset":{"name":"maintainer-clarity-v3","path":"/home/lightningralf/programming/pi-extensions/pi-evalset-lab/examples/fixed-task-set-v3.json"},"model":{"provider":"openai-codex","id":"gpt-5.3-codex","api":"openai-codex-responses"},"variant":{"name":"baseline","systemPrompt":"Answer in concise, plain English. Prefer direct wording over jargon.\n\nYou are a concise technical assistant. Keep answers short and avoid jargon.","source":"dataset.systemPrompt + file:/home/lightningralf/programming/pi-extensions/pi-evalset-lab/examples/system-baseline.txt"},"totals":{"cases":24,"scoredCases":24,"passedCases":16,"failedCases":8,"passRate":0.6666666666666666,"totalLatencyMs":54558,"avgLatencyMs":2273.25,"usage":{"input":1237,"output":1644,"cacheRead":0,"cacheWrite":0,"totalTokens":2881,"cost":{"input":0.0021647499999999996,"output":0.023016000000000005,"cacheRead":0,"cacheWrite":0,"total":0.025180750000000002}}},"cases":[{"id":"fixed-task-set-definition","input":"In plain English, what does a fixed task set mean in eval workflows?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"same\""},{"check":"expectContains","pass":true,"details":"contains \"tasks\""}],"failedChecks":[],"outputPreview":"A **fixed task set** means the eval always uses the **same predefined list of tasks/questions** every time it runs.\n\nIn plain English:\n- The test content is locked in advance.\n- You don’t add, remove, or randomly change tasks between runs.\n- This makes results easier to compare o...","latencyMs":2405,"stopReason":"stop","usage":{"input":52,"output":74,"cacheRead":0,"cacheWrite":0,"totalTokens":126,"cost":{"input":0.000091,"output":0.001036,"cacheRead":0,"cacheWrite":0,"total":0.001127}}},{"id":"fixed-task-set-benefit","input":"Why does a fixed task set improve comparison quality across runs?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"compar\""}],"failedChecks":[],"outputPreview":"A fixed task set makes runs **comparable** because each run faces the **same work**.\n\nWhy it helps:\n\n- **Removes task variation**: Differences in results are less likely due to easier/harder tasks.\n- **Reduces noise**: Scores vary less from random task mix.\n- **Improves fairness*...","latencyMs":3621,"stopReason":"stop","usage":{"input":49,"output":138,"cacheRead":0,"cacheWrite":0,"totalTokens":187,"cost":{"input":0.00008575,"output":0.001932,"cacheRead":0,"cacheWrite":0,"total":0.00201775}}},{"id":"reproducibility-two-factors","input":"Name two things you should lock or record for reproducible eval runs.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"model\""},{"check":"expectContains","pass":true,"details":"contains \"dataset\""}],"failedChecks":[],"outputPreview":"- **Lock the model/version** you evaluate (exact checkpoint, weights, and decoding settings).\n- **Record the data snapshot** used (dataset version, split, and any preprocessing).","latencyMs":1771,"stopReason":"stop","usage":{"input":51,"output":41,"cacheRead":0,"cacheWrite":0,"totalTokens":92,"cost":{"input":0.00008925,"output":0.000574,"cacheRead":0,"cacheWrite":0,"total":0.00066325}}},{"id":"pass-rate-calculation","input":"If 18 of 24 scored cases pass, what pass rate should be reported?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"75\""}],"failedChecks":[],"outputPreview":"Report **75% pass rate** (18 ÷ 24 = 0.75).","latencyMs":1903,"stopReason":"stop","usage":{"input":54,"output":23,"cacheRead":0,"cacheWrite":0,"totalTokens":77,"cost":{"input":0.00009449999999999999,"output":0.000322,"cacheRead":0,"cacheWrite":0,"total":0.0004165}}},{"id":"scored-case-definition","input":"What makes a case scored in this evalset format?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Ee]xpect|[Cc]heck|[Cc]riteria|[Rr]ule)/m"}],"failedChecks":[],"outputPreview":"A case in this evalset format is usually “scored” by comparing the model’s output to an expected result and assigning points based on match quality.\n\nCommon parts are:\n\n- **Input**: the prompt/task.\n- **Expected output**: reference answer or criteria.\n- **Model output**: what the...","latencyMs":3705,"stopReason":"stop","usage":{"input":48,"output":155,"cacheRead":0,"cacheWrite":0,"totalTokens":203,"cost":{"input":0.000084,"output":0.00217,"cacheRead":0,"cacheWrite":0,"total":0.002254}}},{"id":"variant-hash-purpose","input":"What does variantHash help you verify?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"variant\""},{"check":"expectContains","pass":true,"details":"contains \"hash\""}],"failedChecks":[],"outputPreview":"`variantHash` helps verify that a specific product/config variant is exactly the one expected.\n\nIn short, it checks **integrity and identity** of the variant:\n- The variant data hasn’t been changed.\n- The selected options match the original variant.\n- You’re referencing the corre...","latencyMs":1986,"stopReason":"stop","usage":{"input":45,"output":66,"cacheRead":0,"cacheWrite":0,"totalTokens":111,"cost":{"input":0.00007875,"output":0.000924,"cacheRead":0,"cacheWrite":0,"total":0.00100275}}},{"id":"dataset-hash-purpose","input":"Why is datasetHash useful when comparing two reports?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"dataset\""},{"check":"expectContains","pass":true,"details":"contains \"hash\""}],"failedChecks":[],"outputPreview":"`datasetHash` is useful because it tells you whether two reports were built from the **exact same input data**.\n\nWhen comparing reports, this helps you quickly decide:\n\n- **Same hash** → differences are likely from logic, formatting, or config changes.\n- **Different hash** → diff...","latencyMs":2061,"stopReason":"stop","usage":{"input":47,"output":87,"cacheRead":0,"cacheWrite":0,"totalTokens":134,"cost":{"input":0.00008225,"output":0.001218,"cacheRead":0,"cacheWrite":0,"total":0.00130025}}},{"id":"delta-interpretation-speed-cost","input":"Candidate delta avg latency is -800ms and delta total cost is +0.0003. Summarize the tradeoff.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"faster\""}],"failedChecks":[],"outputPreview":"The candidate is **much faster** (about **0.8 seconds lower latency**) but **slightly more expensive** (about **$0.0003 more total cost**).\n\nSo the tradeoff is: **pay a tiny bit more for a noticeable speedup**.","latencyMs":1520,"stopReason":"stop","usage":{"input":63,"output":60,"cacheRead":0,"cacheWrite":0,"totalTokens":123,"cost":{"input":0.00011025,"output":0.00084,"cacheRead":0,"cacheWrite":0,"total":0.00095025}}},{"id":"command-non-interactive-pattern","input":"Show the correct non-interactive pattern to run compare from shell.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":false,"details":"contains \"pi\""},{"check":"expectContains","pass":false,"details":"contains \"-p\""},{"check":"expectContains","pass":false,"details":"contains \"evalset\""}],"failedChecks":["contains \"pi\"","contains \"-p\"","contains \"evalset\""],"outputPreview":"Use ImageMagick’s **compare** in non-interactive mode by giving it files and an output image.\n\n```sh\ncompare input1.png input2.png diff.png\n```\n\nCommon shell-safe pattern (capture metric, suppress image output if not needed):\n\n```sh\nmetric=$(compare -metric RMSE input1.png input2...","latencyMs":4125,"stopReason":"stop","usage":{"input":50,"output":158,"cacheRead":0,"cacheWrite":0,"totalTokens":208,"cost":{"input":0.0000875,"output":0.002212,"cacheRead":0,"cacheWrite":0,"total":0.0022995}}},{"id":"slash-command-shell","input":"Can you run /evalset directly in bash without pi? Answer yes or no and one reason.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":true,"details":"contains \"no\""},{"check":"expectContains","pass":false,"details":"contains \"slash\""}],"failedChecks":["contains \"slash\""],"outputPreview":"No — `/evalset` is a command for the `pi` tool, not a standard bash command.","latencyMs":1092,"stopReason":"stop","usage":{"input":57,"output":26,"cacheRead":0,"cacheWrite":0,"totalTokens":83,"cost":{"input":0.00009975,"output":0.000364,"cacheRead":0,"cacheWrite":0,"total":0.00046375}}},{"id":"report-default-location","input":"If --out is omitted, where are evalset reports written by default?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"evalset\""},{"check":"expectContains","pass":true,"details":"contains \"report\""}],"failedChecks":[],"outputPreview":"By default, evalset reports are written to the current working directory (the folder you run the command from).","latencyMs":1133,"stopReason":"stop","usage":{"input":52,"output":26,"cacheRead":0,"cacheWrite":0,"totalTokens":78,"cost":{"input":0.000091,"output":0.000364,"cacheRead":0,"cacheWrite":0,"total":0.000455}}},{"id":"model-prerequisite","input":"What should you do before /evalset run if no active model is selected?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"/model\""}],"failedChecks":[],"outputPreview":"Select a model first.\n\nBefore running `/evalset run`, make sure an active model is set (for example with `/model \u003cname>` or by choosing one in your UI). Without an active model, the run won’t start.","latencyMs":2342,"stopReason":"stop","usage":{"input":53,"output":51,"cacheRead":0,"cacheWrite":0,"totalTokens":104,"cost":{"input":0.00009275,"output":0.000714,"cacheRead":0,"cacheWrite":0,"total":0.00080675}}},{"id":"max-cases-behavior","input":"What does --max-cases 5 do during run/compare?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Ff]irst|[Ll]imit|5.*case|case.*5)/m"}],"failedChecks":[],"outputPreview":"`--max-cases 5` limits the compare step to at most **5 test cases**.\n\nSo during `run/compare`, it won’t process all available cases—just the first five (based on whatever order the tool uses). This is useful for quick checks.","latencyMs":1796,"stopReason":"stop","usage":{"input":51,"output":60,"cacheRead":0,"cacheWrite":0,"totalTokens":111,"cost":{"input":0.00008925,"output":0.00084,"cacheRead":0,"cacheWrite":0,"total":0.0009292500000000001}}},{"id":"temperature-range","input":"What is the accepted numeric range for --temperature in this extension?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"0\""},{"check":"expectContains","pass":true,"details":"contains \"2\""}],"failedChecks":[],"outputPreview":"Most OpenAI-compatible extensions accept `--temperature` from **0.0 to 2.0**.\n\n- **0.0** = most deterministic\n- **~0.7** = balanced/default in many setups\n- **2.0** = most random\n\nIf this specific extension wraps OpenAI APIs directly, that’s the standard range.","latencyMs":5754,"stopReason":"stop","usage":{"input":50,"output":76,"cacheRead":0,"cacheWrite":0,"totalTokens":126,"cost":{"input":0.0000875,"output":0.001064,"cacheRead":0,"cacheWrite":0,"total":0.0011515}}},{"id":"system-merge-behavior","input":"If dataset.systemPrompt exists and --system-file is provided, are prompts merged or replaced?","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":false,"details":"contains \"merge\""}],"failedChecks":["contains \"merge\""],"outputPreview":"Replaced.\n\nIf `--system-file` is provided, it overrides `dataset.systemPrompt` rather than merging with it.","latencyMs":1117,"stopReason":"stop","usage":{"input":55,"output":29,"cacheRead":0,"cacheWrite":0,"totalTokens":84,"cost":{"input":0.00009625,"output":0.000406,"cacheRead":0,"cacheWrite":0,"total":0.00050225}}},{"id":"mutually-exclusive-system-options","input":"Can --system-file and --system-text be used together?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Nn]o|[Ee]ither|[Oo]ne|not together|[Bb]oth)/m"}],"failedChecks":[],"outputPreview":"Usually, no.\n\nIn most CLIs, `--system-file` and `--system-text` are alternative ways to set the same thing (the system prompt), so using both is either:\n\n- rejected as conflicting, or\n- one overrides the other (often “last one wins”).\n\nCheck your tool’s `--help` for exact behavio...","latencyMs":3438,"stopReason":"stop","usage":{"input":49,"output":83,"cacheRead":0,"cacheWrite":0,"totalTokens":132,"cost":{"input":0.00008575,"output":0.001162,"cacheRead":0,"cacheWrite":0,"total":0.0012477500000000002}}},{"id":"run-identity-fields","input":"Name any three run identity fields in a run report.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":true,"details":"contains \"run\""},{"check":"expectContains","pass":false,"details":"contains \"dataset\""},{"check":"expectContains","pass":false,"details":"contains \"case\""}],"failedChecks":["contains \"dataset\"","contains \"case\""],"outputPreview":"Three common **run identity** fields in a run report are:\n\n1. **Run ID**  \n2. **Run Name**  \n3. **Start Time**","latencyMs":1055,"stopReason":"stop","usage":{"input":48,"output":37,"cacheRead":0,"cacheWrite":0,"totalTokens":85,"cost":{"input":0.000084,"output":0.000518,"cacheRead":0,"cacheWrite":0,"total":0.000602}}},{"id":"compare-identity-fields","input":"Name the two run-id fields that link baseline and candidate inside compare.run.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"baseline\""},{"check":"expectContains","pass":true,"details":"contains \"candidate\""},{"check":"expectContains","pass":true,"details":"contains \"run\""}],"failedChecks":[],"outputPreview":"Inside `compare.run`, the two run-id fields are:\n\n- `baseline_run_id`\n- `candidate_run_id`","latencyMs":1555,"stopReason":"stop","usage":{"input":52,"output":28,"cacheRead":0,"cacheWrite":0,"totalTokens":80,"cost":{"input":0.000091,"output":0.000392,"cacheRead":0,"cacheWrite":0,"total":0.000483}}},{"id":"delta-passrate-zero","input":"If delta pass rate is 0, what does that imply?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Nn]o change|same pass rate|[Uu]nchanged)/m"}],"failedChecks":[],"outputPreview":"A delta pass rate of **0** means there was **no change** in pass rate between the two points you’re comparing.\n\n- If you expected improvement, none happened.\n- If you expected decline, none happened.\n- Pass rate stayed the same.","latencyMs":1769,"stopReason":"stop","usage":{"input":50,"output":55,"cacheRead":0,"cacheWrite":0,"totalTokens":105,"cost":{"input":0.0000875,"output":0.00077,"cacheRead":0,"cacheWrite":0,"total":0.0008575}}},{"id":"keyword-check-limitation","input":"Why can simple keyword checks be misleading for quality?","scored":true,"pass":false,"checks":[{"check":"expectRegex","pass":false,"details":"matches /([Kk]eyword).*(mislead|false|[Ee]rror|[Bb]rittle)|([Mm]islead|false|[Ee]rror|[Bb]rittle).*[Kk]eyword/m"}],"failedChecks":["matches /([Kk]eyword).*(mislead|false|[Ee]rror|[Bb]rittle)|([Mm]islead|false|[Ee]rror|[Bb]rittle).*[Kk]eyword/m"],"outputPreview":"Simple keyword checks can look good while missing real quality.\n\nWhy they mislead:\n- **Can be gamed:** Text can stuff keywords without saying anything useful.\n- **Misses meaning:** Good answers may use different words and still be correct.\n- **No context check:** Keywords don’t s...","latencyMs":2729,"stopReason":"stop","usage":{"input":47,"output":123,"cacheRead":0,"cacheWrite":0,"totalTokens":170,"cost":{"input":0.00008225,"output":0.001722,"cacheRead":0,"cacheWrite":0,"total":0.00180425}}},{"id":"improve-weak-evalset","input":"Give two concrete ways to improve a weak 3-case evalset.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":false,"details":"contains \"more\""},{"check":"expectContains","pass":true,"details":"contains \"cases\""}],"failedChecks":["contains \"more\""],"outputPreview":"1) **Add targeted edge cases**\n- Find where the model currently fails (or is likely to fail), then add tests for those exact patterns.\n- Example: if your 3 cases are all straightforward Q&A, add one ambiguous prompt and one prompt with conflicting instructions.\n\n2) **Increase div...","latencyMs":2722,"stopReason":"stop","usage":{"input":51,"output":133,"cacheRead":0,"cacheWrite":0,"totalTokens":184,"cost":{"input":0.00008925,"output":0.001862,"cacheRead":0,"cacheWrite":0,"total":0.0019512499999999999}}},{"id":"no-overclaim-rollout","input":"Should this be pitched as a huge replacement right away? Answer yes or no and one short reason.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":true,"details":"contains \"no\""},{"check":"expectContains","pass":false,"details":"contains \"phase\""}],"failedChecks":["contains \"phase\""],"outputPreview":"No — big replacements are risky; start small to prove value first.","latencyMs":1205,"stopReason":"stop","usage":{"input":57,"output":18,"cacheRead":0,"cacheWrite":0,"totalTokens":75,"cost":{"input":0.00009975,"output":0.000252,"cacheRead":0,"cacheWrite":0,"total":0.00035175}}},{"id":"stakeholder-brief","input":"Write a one-line stakeholder brief that includes pilot scope and measurement.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"pilot\""},{"check":"expectContains","pass":true,"details":"contains \"measure\""}],"failedChecks":[],"outputPreview":"Pilot scope: run the new process with two teams for six weeks; measurement: compare speed, error rate, and user satisfaction against current baseline and report weekly.","latencyMs":1829,"stopReason":"stop","usage":{"input":50,"output":36,"cacheRead":0,"cacheWrite":0,"totalTokens":86,"cost":{"input":0.0000875,"output":0.000504,"cacheRead":0,"cacheWrite":0,"total":0.0005915}}},{"id":"tie-communication","input":"Baseline and candidate both scored 33.3% pass rate. How should that be communicated?","scored":true,"pass":false,"checks":[{"check":"expectRegex","pass":false,"details":"matches /([Ss]ame|[Nn]o difference|[Uu]nchanged)/m"}],"failedChecks":["matches /([Ss]ame|[Nn]o difference|[Uu]nchanged)/m"],"outputPreview":"Say it plainly:\n\n**“The candidate matched the baseline: both achieved a 33.3% pass rate (no change).”**\n\nIf you want slightly more context:\n\n**“The candidate did not improve over baseline; both had a 33.3% pass rate.”**","latencyMs":1925,"stopReason":"stop","usage":{"input":56,"output":61,"cacheRead":0,"cacheWrite":0,"totalTokens":117,"cost":{"input":0.000098,"output":0.0008539999999999999,"cacheRead":0,"cacheWrite":0,"total":0.0009519999999999999}}}]},"candidate":{"kind":"evalset-run","createdAt":"2026-02-16T14:41:23.888Z","run":{"runId":"f68bccd1-4bff-4145-9f09-d3eca49aecd8","startedAt":"2026-02-16T14:40:35.057Z","finishedAt":"2026-02-16T14:41:23.888Z","modelKey":"openai-codex/gpt-5.3-codex","temperature":null,"datasetHash":"2851c4b6d9e9fd4961be234d2b4602431a46f4adb305eb20b0dd3120e8df64dc","casesHash":"49ca1c743474d713ee966235b03e224e861726e8b56c3aacb625227af7161aca","variantHash":"7e4c4da97cf0d1ea77c0f613c44752a9d2d464cd0c1f26e7f86c69c546569a9d"},"dataset":{"name":"maintainer-clarity-v3","path":"/home/lightningralf/programming/pi-extensions/pi-evalset-lab/examples/fixed-task-set-v3.json"},"model":{"provider":"openai-codex","id":"gpt-5.3-codex","api":"openai-codex-responses"},"variant":{"name":"candidate","systemPrompt":"Answer in concise, plain English. Prefer direct wording over jargon.\n\nYou are a concise technical assistant.\n\nPriorities:\n1) Explain terms in plain language first.\n2) Keep each answer to 2-4 sentences.\n3) Prefer concrete examples over abstractions.","source":"dataset.systemPrompt + file:/home/lightningralf/programming/pi-extensions/pi-evalset-lab/examples/system-candidate.txt"},"totals":{"cases":24,"scoredCases":24,"passedCases":18,"failedCases":6,"passRate":0.75,"totalLatencyMs":48831,"avgLatencyMs":2034.625,"usage":{"input":1861,"output":1708,"cacheRead":0,"cacheWrite":0,"totalTokens":3569,"cost":{"input":0.0032567499999999992,"output":0.023912000000000003,"cacheRead":0,"cacheWrite":0,"total":0.02716875}}},"cases":[{"id":"fixed-task-set-definition","input":"In plain English, what does a fixed task set mean in eval workflows?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"same\""},{"check":"expectContains","pass":true,"details":"contains \"tasks\""}],"failedChecks":[],"outputPreview":"A **fixed task set** means you always test on the same predefined list of tasks every time you run an evaluation.  \nThis makes results comparable across model versions, because the test didn’t change.  \nExample: if your eval always includes the same 500 QA prompts, score changes ...","latencyMs":1892,"stopReason":"stop","usage":{"input":78,"output":74,"cacheRead":0,"cacheWrite":0,"totalTokens":152,"cost":{"input":0.0001365,"output":0.001036,"cacheRead":0,"cacheWrite":0,"total":0.0011725}}},{"id":"fixed-task-set-benefit","input":"Why does a fixed task set improve comparison quality across runs?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"compar\""}],"failedChecks":[],"outputPreview":"A fixed task set means every run is tested on the exact same jobs, so differences in results come from the system changes, not from easier or harder tasks. This reduces random noise and makes comparisons fair and repeatable. For example, if Run A and Run B both solve the same 100...","latencyMs":1950,"stopReason":"stop","usage":{"input":75,"output":85,"cacheRead":0,"cacheWrite":0,"totalTokens":160,"cost":{"input":0.00013125,"output":0.00119,"cacheRead":0,"cacheWrite":0,"total":0.0013212500000000002}}},{"id":"reproducibility-two-factors","input":"Name two things you should lock or record for reproducible eval runs.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"model\""},{"check":"expectContains","pass":true,"details":"contains \"dataset\""}],"failedChecks":[],"outputPreview":"Lock the **dataset version** and the **model version/checkpoint**.  \nAlso record the **random seed** and the **exact eval settings** (like temperature, max tokens, and prompt template) so reruns use the same conditions.","latencyMs":1651,"stopReason":"stop","usage":{"input":77,"output":53,"cacheRead":0,"cacheWrite":0,"totalTokens":130,"cost":{"input":0.00013475,"output":0.000742,"cacheRead":0,"cacheWrite":0,"total":0.00087675}}},{"id":"pass-rate-calculation","input":"If 18 of 24 scored cases pass, what pass rate should be reported?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"75\""}],"failedChecks":[],"outputPreview":"Report a **75% pass rate**.  \nCalculation: \\(18 \\div 24 = 0.75\\), then \\(0.75 \\times 100 = 75\\%\\).","latencyMs":1101,"stopReason":"stop","usage":{"input":80,"output":44,"cacheRead":0,"cacheWrite":0,"totalTokens":124,"cost":{"input":0.00014,"output":0.000616,"cacheRead":0,"cacheWrite":0,"total":0.0007559999999999999}}},{"id":"scored-case-definition","input":"What makes a case scored in this evalset format?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Ee]xpect|[Cc]heck|[Cc]riteria|[Rr]ule)/m"}],"failedChecks":[],"outputPreview":"A case in this evalset format is usually one test example with clear fields like an input, the expected output (or rubric), and sometimes metadata (ID, tags, difficulty).  \nIt’s “scored” by comparing the model’s actual output to that expected output using a rule, such as exact ma...","latencyMs":2797,"stopReason":"stop","usage":{"input":74,"output":121,"cacheRead":0,"cacheWrite":0,"totalTokens":195,"cost":{"input":0.0001295,"output":0.001694,"cacheRead":0,"cacheWrite":0,"total":0.0018235}}},{"id":"variant-hash-purpose","input":"What does variantHash help you verify?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"variant\""},{"check":"expectContains","pass":true,"details":"contains \"hash\""}],"failedChecks":[],"outputPreview":"`variantHash` helps you check that a specific product variant’s data hasn’t changed.  \nIt acts like a fingerprint of key fields (like size, color, price, or SKU), so if any of those values change, the hash changes too.  \nFor example, when updating a cart item, you can compare has...","latencyMs":2180,"stopReason":"stop","usage":{"input":71,"output":83,"cacheRead":0,"cacheWrite":0,"totalTokens":154,"cost":{"input":0.00012425,"output":0.001162,"cacheRead":0,"cacheWrite":0,"total":0.00128625}}},{"id":"dataset-hash-purpose","input":"Why is datasetHash useful when comparing two reports?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"dataset\""},{"check":"expectContains","pass":true,"details":"contains \"hash\""}],"failedChecks":[],"outputPreview":"`datasetHash` is a fingerprint of the underlying data used to build a report. If two reports have the same hash, they were generated from the same dataset, so differences are likely from filters, settings, or calculations—not source data changes. If the hashes differ, the data it...","latencyMs":2543,"stopReason":"stop","usage":{"input":73,"output":93,"cacheRead":0,"cacheWrite":0,"totalTokens":166,"cost":{"input":0.00012775,"output":0.001302,"cacheRead":0,"cacheWrite":0,"total":0.00142975}}},{"id":"delta-interpretation-speed-cost","input":"Candidate delta avg latency is -800ms and delta total cost is +0.0003. Summarize the tradeoff.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"faster\""}],"failedChecks":[],"outputPreview":"This candidate is much faster: average latency improves by **800 ms** (about **0.8 seconds** per request).  \nThe tradeoff is a tiny cost increase of **0.0003** total, so you’re paying slightly more for a noticeable speed gain.","latencyMs":1126,"stopReason":"stop","usage":{"input":89,"output":59,"cacheRead":0,"cacheWrite":0,"totalTokens":148,"cost":{"input":0.00015575,"output":0.000826,"cacheRead":0,"cacheWrite":0,"total":0.00098175}}},{"id":"command-non-interactive-pattern","input":"Show the correct non-interactive pattern to run compare from shell.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":false,"details":"contains \"pi\""},{"check":"expectContains","pass":false,"details":"contains \"-p\""},{"check":"expectContains","pass":false,"details":"contains \"evalset\""}],"failedChecks":["contains \"pi\"","contains \"-p\"","contains \"evalset\""],"outputPreview":"Use **ImageMagick’s `compare`** with all file paths explicitly, and write output to a file (not a display window).  \nExample:\n\n```sh\ncompare -metric AE input.png reference.png diff.png 2>metric.txt\n```\n\nThis runs fully non-interactive: `diff.png` gets the visual diff, and the num...","latencyMs":3520,"stopReason":"stop","usage":{"input":76,"output":88,"cacheRead":0,"cacheWrite":0,"totalTokens":164,"cost":{"input":0.000133,"output":0.001232,"cacheRead":0,"cacheWrite":0,"total":0.0013650000000000001}}},{"id":"slash-command-shell","input":"Can you run /evalset directly in bash without pi? Answer yes or no and one reason.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"no\""},{"check":"expectContains","pass":true,"details":"contains \"slash\""}],"failedChecks":[],"outputPreview":"No — `/evalset` is a slash-command meant for the chat tool, not a normal Bash executable, so Bash won’t run it directly.","latencyMs":1107,"stopReason":"stop","usage":{"input":83,"output":34,"cacheRead":0,"cacheWrite":0,"totalTokens":117,"cost":{"input":0.00014525,"output":0.00047599999999999997,"cacheRead":0,"cacheWrite":0,"total":0.00062125}}},{"id":"report-default-location","input":"If --out is omitted, where are evalset reports written by default?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"evalset\""},{"check":"expectContains","pass":true,"details":"contains \"report\""}],"failedChecks":[],"outputPreview":"If `--out` is not provided, evalset reports are written to the current working directory (the folder you ran the command from).  \nMost tools use a default filename there, often based on the evalset or run name.  \nExample: if you run the command in `/home/user/project`, the report...","latencyMs":1964,"stopReason":"stop","usage":{"input":78,"output":74,"cacheRead":0,"cacheWrite":0,"totalTokens":152,"cost":{"input":0.0001365,"output":0.001036,"cacheRead":0,"cacheWrite":0,"total":0.0011725}}},{"id":"model-prerequisite","input":"What should you do before /evalset run if no active model is selected?","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":false,"details":"contains \"/model\""}],"failedChecks":["contains \"/model\""],"outputPreview":"Before running `/evalset run`, select an active model first.  \nFor example, set one with the model selection command in your tool (like choosing `gpt-4.1`), then run the eval.  \nIf no model is active, the run usually fails or uses an unintended default.","latencyMs":1319,"stopReason":"stop","usage":{"input":79,"output":65,"cacheRead":0,"cacheWrite":0,"totalTokens":144,"cost":{"input":0.00013825,"output":0.00091,"cacheRead":0,"cacheWrite":0,"total":0.00104825}}},{"id":"max-cases-behavior","input":"What does --max-cases 5 do during run/compare?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Ff]irst|[Ll]imit|5.*case|case.*5)/m"}],"failedChecks":[],"outputPreview":"`--max-cases 5` limits the compare run to at most **5 test cases**.  \nIt’s useful for quick checks, like sampling a few cases instead of running everything.  \nExample: if a dataset has 200 cases, `run/compare --max-cases 5` will only execute and compare 5 of them.","latencyMs":1705,"stopReason":"stop","usage":{"input":77,"output":75,"cacheRead":0,"cacheWrite":0,"totalTokens":152,"cost":{"input":0.00013475,"output":0.00105,"cacheRead":0,"cacheWrite":0,"total":0.00118475}}},{"id":"temperature-range","input":"What is the accepted numeric range for --temperature in this extension?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"0\""},{"check":"expectContains","pass":true,"details":"contains \"2\""}],"failedChecks":[],"outputPreview":"In this extension, `--temperature` is accepted from **0.0 to 2.0**.  \nUse lower values (like `0.2`) for more predictable output, and higher values (like `1.2`) for more varied/creative output.","latencyMs":1546,"stopReason":"stop","usage":{"input":76,"output":59,"cacheRead":0,"cacheWrite":0,"totalTokens":135,"cost":{"input":0.000133,"output":0.000826,"cacheRead":0,"cacheWrite":0,"total":0.000959}}},{"id":"system-merge-behavior","input":"If dataset.systemPrompt exists and --system-file is provided, are prompts merged or replaced?","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"merge\""}],"failedChecks":[],"outputPreview":"They are **replaced**, not merged.  \nIf `--system-file` is provided, its content becomes the system prompt used at runtime, overriding `dataset.systemPrompt`.  \nExample: if the dataset says “be terse” but the file says “be detailed,” the file’s instruction is what gets used.","latencyMs":1834,"stopReason":"stop","usage":{"input":81,"output":67,"cacheRead":0,"cacheWrite":0,"totalTokens":148,"cost":{"input":0.00014175,"output":0.000938,"cacheRead":0,"cacheWrite":0,"total":0.00107975}}},{"id":"mutually-exclusive-system-options","input":"Can --system-file and --system-text be used together?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Nn]o|[Ee]ither|[Oo]ne|not together|[Bb]oth)/m"}],"failedChecks":[],"outputPreview":"Usually, no—you should treat them as alternatives.  \n`--system-file` loads the system prompt from a file, while `--system-text` sets it directly in the command, and most tools expect only one source.  \nIf both are provided, many CLIs either error out or let one override the other...","latencyMs":2298,"stopReason":"stop","usage":{"input":75,"output":81,"cacheRead":0,"cacheWrite":0,"totalTokens":156,"cost":{"input":0.00013125,"output":0.001134,"cacheRead":0,"cacheWrite":0,"total":0.00126525}}},{"id":"run-identity-fields","input":"Name any three run identity fields in a run report.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":true,"details":"contains \"run\""},{"check":"expectContains","pass":false,"details":"contains \"dataset\""},{"check":"expectContains","pass":false,"details":"contains \"case\""}],"failedChecks":["contains \"dataset\"","contains \"case\""],"outputPreview":"Three common **run identity** fields are:\n\n1. **Run ID** – a unique number or code for that specific run.  \n2. **Run Name** – a human-readable label (for example, “Nightly Regression”).  \n3. **Start Time** – when the run began, which helps distinguish runs with similar names.","latencyMs":1593,"stopReason":"stop","usage":{"input":74,"output":72,"cacheRead":0,"cacheWrite":0,"totalTokens":146,"cost":{"input":0.0001295,"output":0.001008,"cacheRead":0,"cacheWrite":0,"total":0.0011375}}},{"id":"compare-identity-fields","input":"Name the two run-id fields that link baseline and candidate inside compare.run.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"baseline\""},{"check":"expectContains","pass":true,"details":"contains \"candidate\""},{"check":"expectContains","pass":true,"details":"contains \"run\""}],"failedChecks":[],"outputPreview":"Inside `compare.run`, the two run ID fields are **`baseline_run_id`** and **`candidate_run_id`**.  \nThey point to the specific baseline run and candidate run being compared.","latencyMs":1627,"stopReason":"stop","usage":{"input":78,"output":45,"cacheRead":0,"cacheWrite":0,"totalTokens":123,"cost":{"input":0.0001365,"output":0.00063,"cacheRead":0,"cacheWrite":0,"total":0.0007665}}},{"id":"delta-passrate-zero","input":"If delta pass rate is 0, what does that imply?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Nn]o change|same pass rate|[Uu]nchanged)/m"}],"failedChecks":[],"outputPreview":"A delta pass rate of 0 means there was no change in pass rate between the two groups or time periods you compared.  \nFor example, if both before and after pass rates are 78%, the delta is 0.  \nIt implies performance stayed flat—neither improved nor declined.","latencyMs":2743,"stopReason":"stop","usage":{"input":76,"output":63,"cacheRead":0,"cacheWrite":0,"totalTokens":139,"cost":{"input":0.000133,"output":0.000882,"cacheRead":0,"cacheWrite":0,"total":0.001015}}},{"id":"keyword-check-limitation","input":"Why can simple keyword checks be misleading for quality?","scored":true,"pass":false,"checks":[{"check":"expectRegex","pass":false,"details":"matches /([Kk]eyword).*(mislead|false|[Ee]rror|[Bb]rittle)|([Mm]islead|false|[Ee]rror|[Bb]rittle).*[Kk]eyword/m"}],"failedChecks":["matches /([Kk]eyword).*(mislead|false|[Ee]rror|[Bb]rittle)|([Mm]islead|false|[Ee]rror|[Bb]rittle).*[Kk]eyword/m"],"outputPreview":"Simple keyword checks only verify that certain words appear, not that the content is correct, useful, or coherent. A bad answer can “game” the check by stuffing keywords without actually solving the problem. For example, a support reply might include words like “refund,” “policy,...","latencyMs":2609,"stopReason":"stop","usage":{"input":73,"output":84,"cacheRead":0,"cacheWrite":0,"totalTokens":157,"cost":{"input":0.00012775,"output":0.001176,"cacheRead":0,"cacheWrite":0,"total":0.00130375}}},{"id":"improve-weak-evalset","input":"Give two concrete ways to improve a weak 3-case evalset.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":false,"details":"contains \"more\""},{"check":"expectContains","pass":true,"details":"contains \"cases\""}],"failedChecks":["contains \"more\""],"outputPreview":"1) **Add realistic hard negatives.** If your eval set only has obvious “wrong” answers, include tricky lookalikes (for example, same topic but subtly incorrect facts) so the model must truly understand, not just keyword-match.  \n2) **Expand case diversity with clear labels.** Tur...","latencyMs":3274,"stopReason":"stop","usage":{"input":77,"output":116,"cacheRead":0,"cacheWrite":0,"totalTokens":193,"cost":{"input":0.00013475,"output":0.001624,"cacheRead":0,"cacheWrite":0,"total":0.00175875}}},{"id":"no-overclaim-rollout","input":"Should this be pitched as a huge replacement right away? Answer yes or no and one short reason.","scored":true,"pass":false,"checks":[{"check":"expectContains","pass":true,"details":"contains \"no\""},{"check":"expectContains","pass":false,"details":"contains \"phase\""}],"failedChecks":["contains \"phase\""],"outputPreview":"No — it’s usually better to position it as a low-risk add-on first, so people can see value before committing to a full replacement.","latencyMs":1175,"stopReason":"stop","usage":{"input":83,"output":33,"cacheRead":0,"cacheWrite":0,"totalTokens":116,"cost":{"input":0.00014525,"output":0.000462,"cacheRead":0,"cacheWrite":0,"total":0.00060725}}},{"id":"stakeholder-brief","input":"Write a one-line stakeholder brief that includes pilot scope and measurement.","scored":true,"pass":true,"checks":[{"check":"expectContains","pass":true,"details":"contains \"pilot\""},{"check":"expectContains","pass":true,"details":"contains \"measure\""}],"failedChecks":[],"outputPreview":"Pilot scope: run a 6‑week trial with 50 users in two teams using the new workflow end‑to‑end, and measure success by adoption rate, task completion time, error rate, and weekly satisfaction score against baseline.","latencyMs":1598,"stopReason":"stop","usage":{"input":76,"output":52,"cacheRead":0,"cacheWrite":0,"totalTokens":128,"cost":{"input":0.000133,"output":0.000728,"cacheRead":0,"cacheWrite":0,"total":0.000861}}},{"id":"tie-communication","input":"Baseline and candidate both scored 33.3% pass rate. How should that be communicated?","scored":true,"pass":true,"checks":[{"check":"expectRegex","pass":true,"details":"matches /([Ss]ame|[Nn]o difference|[Uu]nchanged)/m"}],"failedChecks":[],"outputPreview":"Say it directly: **“The candidate matched the baseline at 33.3% pass rate, so there was no measured improvement.”**  \nIf helpful, add context like sample size: **“Both were 33.3% (e.g., 10/30), indicating parity rather than gain.”**  \nAvoid wording like “performed better” or “reg...","latencyMs":3679,"stopReason":"stop","usage":{"input":82,"output":88,"cacheRead":0,"cacheWrite":0,"totalTokens":170,"cost":{"input":0.0001435,"output":0.001232,"cacheRead":0,"cacheWrite":0,"total":0.0013755}}}]},"delta":{"passRate":0.08333333333333337,"avgLatencyMs":-238.625,"totalCost":0.0019879999999999967}}</script>
+  <script>
+    const report = JSON.parse(document.getElementById('report-data').textContent);
+
+    const pct = (v) => v == null ? 'n/a' : (v * 100).toFixed(1) + '%';
+    const money = (v) => '$' + Number(v ?? 0).toFixed(6);
+    const ms = (v) => Number(v ?? 0).toFixed(0) + ' ms';
+    const esc = (s) => String(s ?? '').replaceAll('&','&amp;').replaceAll('<','&lt;').replaceAll('>','&gt;');
+    const passPill = (pass) => pass ? '<span class="pill ok">PASS</span>' : '<span class="pill bad">FAIL</span>';
+
+    const meta = document.getElementById('meta');
+    meta.innerHTML = [
+      report.dataset?.name,
+      'run ' + report.run?.runId,
+      'model ' + report.model?.provider + '/' + report.model?.id,
+      'sample source: .evalset/reports/compare-maintainer-clarity-v3-20260216T154123.json'
+    ].map(esc).join(' · ');
+
+    const summaryItems = [
+      ['Baseline pass', pct(report.baseline?.totals?.passRate) + ' (' + report.baseline?.totals?.passedCases + '/' + report.baseline?.totals?.scoredCases + ')'],
+      ['Candidate pass', pct(report.candidate?.totals?.passRate) + ' (' + report.candidate?.totals?.passedCases + '/' + report.candidate?.totals?.scoredCases + ')'],
+      ['Δ pass rate', pct(report.delta?.passRate)],
+      ['Δ avg latency', ms(report.delta?.avgLatencyMs)],
+      ['Δ total cost', money(report.delta?.totalCost)],
+      ['Dataset hash', report.run?.datasetHash?.slice(0, 12) + '…']
+    ];
+
+    document.getElementById('summary').innerHTML = summaryItems
+      .map(([label, val]) => '<div class="card"><div class="label">' + esc(label) + '</div><div class="val">' + esc(val) + '</div></div>')
+      .join('');
+
+    const candidateById = new Map((report.candidate?.cases || []).map((c) => [c.id, c]));
+    let improved = 0;
+    let regressed = 0;
+
+    const rows = (report.baseline?.cases || []).map((b, idx) => {
+      const c = candidateById.get(b.id) || {};
+      let outcome = 'same';
+      if (!b.pass && c.pass) { outcome = 'improved'; improved += 1; }
+      else if (b.pass && !c.pass) { outcome = 'regressed'; regressed += 1; }
+
+      const outcomePill = outcome === 'improved'
+        ? '<span class="pill imp">Improved</span>'
+        : outcome === 'regressed'
+          ? '<span class="pill reg">Regressed</span>'
+          : '<span class="pill same">No change</span>';
+
+      const bChecks = (b.checks || []).map((x) => (x.pass ? '✅ ' : '❌ ') + x.details).join('
+') || 'None';
+      const cChecks = (c.checks || []).map((x) => (x.pass ? '✅ ' : '❌ ') + x.details).join('
+') || 'None';
+
+      return '<tr>' +
+        '<td>' + (idx + 1) + '</td>' +
+        '<td><code>' + esc(b.id) + '</code></td>' +
+        '<td>' + passPill(!!b.pass) + '<div class="meta">' + ms(b.latencyMs) + '</div></td>' +
+        '<td>' + passPill(!!c.pass) + '<div class="meta">' + ms(c.latencyMs) + '</div></td>' +
+        '<td>' + outcomePill + '</td>' +
+        '<td><details><summary>checks + output</summary>' +
+          '<div class="split">' +
+            '<div><h4 style="margin:8px 0 6px;">Baseline checks</h4><pre>' + esc(bChecks) + '</pre><h4 style="margin:8px 0 6px;">Baseline output</h4><pre>' + esc(b.outputPreview || '') + '</pre></div>' +
+            '<div><h4 style="margin:8px 0 6px;">Candidate checks</h4><pre>' + esc(cChecks) + '</pre><h4 style="margin:8px 0 6px;">Candidate output</h4><pre>' + esc(c.outputPreview || '') + '</pre></div>' +
+          '</div>' +
+        '</details></td>' +
+      '</tr>';
+    }).join('');
+
+    document.getElementById('rows').innerHTML = rows;
+  </script>
+</body>
+</html>