eval-builder 0.1.0__tar.gz → 0.1.2__tar.gz

{eval_builder-0.1.0 → eval_builder-0.1.2}/.gitignore

@@ -9,9 +9,11 @@ build/
 *.egg-info/
 .DS_Store
 evalset/
+!examples/sample-labeling/evalset/
 .deepeval/
 # Example inputs that prepare.py downloads or derives (reproducible from the pinned dataset)
 examples/mt-bench/data/
 # Large or derived files inside example workspaces
 examples/mt-bench/*/judge_requests.jsonl
 examples/mt-bench/*/traces.jsonl
+examples/sample-labeling/evalset/judge_requests.jsonl

{eval_builder-0.1.0 → eval_builder-0.1.2}/AGENTS.md

@@ -16,6 +16,9 @@ src/eval_builder/
   judge/run.py     opt-in judge plugin runner (off by default, runs a user command)
   judge/check.py   flip rate, kappa, accuracy, probes, verdicts
   judge/stats.py   Wilson interval, Cohen's kappa, majority vote
+  label.py         which cases a person should label, the CSV sheet, `label import`
+  label_sheet.py   the offline HTML labeling sheet (one file, inline JS, no network)
+  balance.py       the warning when cases or labels are mostly one outcome
   export.py        promptfoo, DeepEval, Inspect AI, JSONL
   report.py        report.md and report.json
   setup_agents.py  `eval-builder setup` for Claude Code, Codex, Cursor
@@ -25,7 +28,8 @@ src/eval_builder/
 
 ## Rules
 
-- No model calls and no network access in the package. The judge runner only starts a
+- No model calls and no network access in the package. (Exported files may call a
+  model when the user runs them, for example the DeepEval test calling a wired judge.) The judge runner only starts a
   command the user names, and only with an explicit flag.
 - Deterministic: same inputs and seed give the same selection and the same files.
 - Every number the tool reports must be traceable to a file in the workspace.

eval_builder-0.1.2/CHANGELOG.md

@@ -0,0 +1,85 @@
+# Changelog
+
+## 0.1.2 (2026-10-08)
+
+Human labels. No judge can be called trustworthy without them, and both real agent
+sessions against 0.1.0 and 0.1.1 stopped at that step: they asked for labels and had
+no way to collect them.
+
+- `label` (CLI, and MCP `label`): picks the ready cases a person should label, 24 by
+  default (judge-check needs 20). The budget is split evenly across outcomes (the
+  judges' consensus verdict, or the logged failure flag before any judge has run), and
+  up to half of each share goes to cases where the judges disagree, flip across
+  repeats or move under padding. Every pick and its reason is in `label_plan.json`.
+- `label` writes `label_sheet.html`: one self-contained file that works offline and
+  makes no network requests (its content security policy blocks them). One case per
+  screen with the earlier turns, the reply, the expected behavior and criteria;
+  pass/fail or A/B buttons (labels come from rubric.yaml), an optional note, keyboard
+  shortcuts, progress, and the judges' verdicts hidden. Progress survives a reload
+  through the browser's local storage. Export downloads `labels.jsonl` in exactly the
+  format judge-check reads and shows the same text to copy.
+- `label` also writes `label_sheet.csv` for spreadsheet users (fill the label column).
+- `label import <file>` (CLI, and MCP `label_import`): reads the sheet's labels.jsonl,
+  the filled-in CSV, a JSON list, or stdin (`-`); checks every case id and label
+  against cases.yaml and rubric.yaml; merges into `labels.jsonl` (a new label replaces
+  the same labeler's earlier one, other labelers are kept); lists rejected rows with
+  the reason and exits 1 when there are any.
+- `select` and `judge-check` warn, with the real proportions, when at least 80% of the
+  selected cases or of the human labels share one outcome. judge-check also warns when
+  a judge gave the same verdict on every labeled case or its accuracy is no better
+  than always giving the most common label, and reports `majority_baseline` (that
+  always-the-common-label accuracy).
+- DeepEval export: a pointwise judge that passed judge-check (or one forced with
+  `--judge`) now grades every case with its exact rubric.yaml prompt, through a custom
+  metric in `test_eval_builder.py` and `judge.json`. Ollama providers are called on the
+  local server; for other providers you fill in `call_judge`. Without a checked judge
+  the file keeps GEval. The export notes and README say that Inspect AI still uses its
+  default `model_graded_qa` grader.
+- judge-check parses JSON verdicts that a token limit cut off mid-reason, and verdicts
+  wrapped in a json code fence.
+- `status` and judge-check's `next` walk through the labeling step.
+- `examples/sample-labeling/`: the whole flow on the bundled 48-conversation sample.
+
+## 0.1.1 (2026-10-08)
+
+Fixes from a fresh-install audit and a real Claude Code session driving the MCP server.
+
+- promptfoo export: the app now receives the whole conversation as chat messages
+  (`{{ messages | dump }}`). Before, only the last user turn was sent, so multi-turn cases
+  such as "Can you parallelize it?" reached the app without their context.
+- promptfoo export wires a checked judge in as the llm-rubric grader: the first pointwise
+  judge that judge-check marked trustworthy and that has a new optional `provider` field in
+  rubric.yaml (a promptfoo provider id). `export --judge <id>` forces one, with a warning
+  when it did not pass. The judge's exact prompt becomes `rubricPrompt`.
+- judge-check parses JSON verdicts such as `{"pass": false, "reason": "..."}`, so one judge
+  prompt works both in judge-check and as a promptfoo grader.
+- judge-check summary rows (CLI `--json` and MCP) now carry 95% intervals, case counts and
+  label counts, and the result has a `next` step (for example how to write labels.jsonl).
+- judge-plan documents the request row fields (`judge`, not `judge_id`), refuses to plan
+  when no case is ready or a judge is still TODO, and says how to write judgments.
+- Clearer errors and next steps: judge-check without judgments, format detection failure
+  (shows the keys it saw and the shapes it accepts), `select` when fewer unique traces than
+  `-n` exist, `status` now walks through every step including judges.
+- MCP `judge_check` takes `include_cases` to return per-case verdict counts and majorities.
+- MCP tool descriptions say what each tool does, when to use it and what to call next.
+- `setup`: installs the Codex skill in the same run when the codex CLI is on PATH but
+  `~/.codex` does not exist yet, and says to restart the agent after applying.
+- `examples/sample_logs.jsonl`: 48 conversations from MT-Bench (CC BY 4.0) to try the
+  quickstart without your own logs.
+
+## 0.1.0 (2026-10-08)
+
+First release.
+
+- `ingest`: OpenAI chat JSONL (fine-tuning style and request/response logs), Anthropic
+  messages, Langfuse trace exports, OpenTelemetry GenAI spans (OTLP JSON and flat span
+  JSON), and generic input/output JSONL. Default-on redaction with a per-kind report.
+- `select`: exact and near-duplicate removal, TF-IDF k-means clusters, stratum
+  coverage, failure oversampling, a recorded reason for every pick.
+- `draft`, `validate`: case and rubric templates that the agent fills in with the user.
+- `judge-plan`, `judge-run` (opt-in), `judge-check`: repeated trials, flip rate,
+  agreement with human labels (accuracy with Wilson intervals, Cohen's kappa), answer
+  order and padding probes, a verdict per judge.
+- `export`: promptfoo, DeepEval, Inspect AI, JSONL.
+- `report`: Markdown and JSON.
+- `setup`: registers the MCP server with Claude Code, Codex and Cursor.

eval_builder-0.1.2/PKG-INFO

@@ -0,0 +1,282 @@
+Metadata-Version: 2.5
+Name: eval-builder
+Version: 0.1.2
+Summary: Turn real LLM app logs into an eval suite and measure which LLM judges you can trust.
+Author: Abel Yagubyan
+License-Expression: MIT
+License-File: LICENSE
+Keywords: deepeval,evals,inspect-ai,llm,llm-as-a-judge,mcp,promptfoo
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: mcp>=1.2
+Requires-Dist: numpy>=1.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: scipy>=1.10
+Description-Content-Type: text/markdown
+
+# eval-builder
+
+Your agent turns your app's real logs into an eval suite, and tells you which of its judges you can actually trust.
+
+Try it on 48 sample conversations (no account, no API key, no model needed for these steps):
+
+```sh
+curl -sLO https://raw.githubusercontent.com/Abelo9996/eval-builder/main/examples/sample_logs.jsonl
+uvx eval-builder ingest sample_logs.jsonl     # or your own logs: OpenAI, Anthropic, Langfuse, OpenTelemetry, JSONL
+uvx eval-builder select -n 10 --stratify category && uvx eval-builder draft
+```
+
+![The quickstart running against eval-builder 0.1.1 from PyPI](docs/demo.gif)
+
+What you get (real output of the commands above, eval-builder 0.1.1 from PyPI; the
+curl and the three uvx calls took 35 s in total here; the very first uvx run also
+downloads about 38 MiB, mostly numpy, scipy and scikit-learn, which took 6 s here):
+
+```
+ingested 48 traces into evalset/traces.jsonl
+  sample_logs.jsonl: format=openai records=48 traces=48 skipped=0 sha256=2d5dd2295784988b
+redactions: 0 {}
+next: eval-builder select -n 30 (add --stratify <metadata keys> to cover them)
+48 traces -> 16 unique (32 exact dupes, 0 near dupes) -> selected 10 (8 failures) across 4 clusters
+  q121-llama-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
+  q81-alpaca-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
+  q102-alpaca-13b: covers category=reasoning (2 unique traces, 12%)
+  q111-alpaca-13b: adds variety within cluster 1 (8 traces, 50%; triangle, response, person); least similar to cases already picked there
+  ... (6 more picks)
+next: run draft to turn the selection into cases.yaml
+10 case(s) added, 10 total in evalset/cases.yaml
+next: read the cases (list_cases, or cases.yaml), define criteria and judges (set_rubric, or rubric.yaml), fill expected_behavior and criteria per case and set status: ready (update_case), then run validate
+```
+
+`evalset/cases.yaml` now holds 10 real conversations with a TODO where the expected
+behavior goes. Writing that, running judges and reading the statistics is the part your
+coding agent does, through the MCP server (next section).
+
+eval-builder is not another eval platform. It builds the suite and checks the judges,
+then exports to the tools you already run: promptfoo, DeepEval, Inspect AI, or plain
+JSONL. It never calls a model. Your coding agent (Claude Code, Codex, Cursor) does the
+thinking; eval-builder does the selection, the bookkeeping and the statistics, and
+writes down the evidence.
+
+## Use it with your agent
+
+```sh
+uvx eval-builder setup --yes    # registers the MCP server and the skill, then restart your agent
+```
+
+Then type something like this into Claude Code, from the folder that has your logs:
+
+> Our assistant's chat logs from last month are in ./logs. Build a small eval suite from
+> them (about 10 cases) that I can run in promptfoo, and tell me whether I can trust a
+> local judge for grading it. I have Ollama running with qwen2.5:7b-instruct and
+> llama3.2:3b. Keep the judge runs small.
+
+This is from a real headless Claude Code session (October 2026, Claude Code 2.1.293)
+given exactly that prompt and 48 conversations from [`examples/sample_logs.jsonl`](examples/sample_logs.jsonl).
+It ran ingest, select, draft, list_cases, set_rubric, ten update_case calls, validate,
+judge_plan, 140 judge calls to Ollama through a short script it wrote, judge_check,
+export and report in 32 turns and about 10 minutes. From its final answer:
+
+> I built a 10-case promptfoo suite, but **you can't trust either local judge yet.**
+> Neither one has been checked against human labels, and each confidently got an
+> obvious case wrong.
+>
+> | judge | flip rate | padding helped | verdict |
+> |---|---|---|---|
+> | qwen2.5:7b-instruct | 10% [2%, 40%] (n=10) | 0% [0%, 28%] | not_enough_data |
+> | llama3.2:3b | 20% [6%, 51%] (n=10) | 0% [0%, 28%] | not_enough_data |
+>
+> **qwen7b failed a clear case.** It passed the Hawaii rewrite 5 out of 5 times, though
+> only the first sentence starts with "A".
+>
+> **No grader is wired in.** Because no judge passed, the `llm-rubric` checks fall back to
+> promptfoo's default grader, which needs an OpenAI key. To grade locally anyway, export
+> again with the judge set explicitly (the `export` tool's `judge` option, e.g. `qwen7b`).
+> Its verdicts will be unverified.
+
+It wrote every expected behavior itself because nobody was there to confirm them, and
+said so on each case. It did not invent human labels; it asked for them. That is where
+this session and the earlier one both stopped, so 0.1.2 adds the missing step: `label`
+writes a sheet the person labels in their browser, and `label import` brings the
+labels back for judge-check ([Human labels, end to end](#human-labels-end-to-end)).
+
+An earlier session against 0.1.0 is where the promptfoo context bug fixed in 0.1.1 came
+from. The agent read the export and told the user: "Right now it only sends the final
+question (`{{input}}`), so your app won't see what "it" refers to in "Can you
+parallelize it?"."
+
+## Example: real output
+
+Everything below comes from runs on this machine (Apple M4, 16 GB) that are committed
+in [`examples/mt-bench/`](examples/mt-bench/). The data is the LMSYS
+[MT-Bench human judgments](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments)
+dataset (CC BY 4.0): 80 two-turn questions answered by 6 models, plus pairwise verdicts
+from expert human judges.
+
+**Logs to suite.** `ingest` read 480 conversations (OpenAI chat format, redaction on,
+0 matches). `select -n 24 --stratify category` removed 400 exact duplicates (each
+question was asked to six models), clustered the 80 unique conversations into 9 topics
+and picked 24 covering all 8 categories, 19 of them failures. From
+[`suite/report.md`](examples/mt-bench/suite/report.md):
+
+```
+| trace           | cluster | represents | why it was picked |
+| q81-alpaca-13b  | 0 | 6 | failure (negative user feedback); 62% of unique traces are failures and at least 30% of picks are reserved for them |
+| q82-alpaca-13b  | 0 | 6 | adds variety within cluster 0 (11 traces, 14%; response, previous response, previous); least similar to cases already picked there |
+| q156-alpaca-13b | 3 | 6 | covers category=humanities (10 unique traces, 12%) |
+```
+
+The agent wrote expected behavior for the 24 cases, and `export` produced files that
+Inspect AI 0.3.277, DeepEval 4.2.8 and promptfoo 0.124.0 all load (see
+[`verify_exports.sh`](examples/mt-bench/verify_exports.sh)).
+
+**Which judges can you trust?** Five local LLM judge setups (four models through
+Ollama, one of them also at temperature 0) and two controls that are not language models judged 80 answer pairs where human experts
+picked a winner. Each judge ran 5 times per pair, 5 more with the answers swapped, and
+5 more with an irrelevant paragraph appended to one answer: 8,400 calls, 0 errors.
+`eval-builder judge-check`, from [`judges/report.md`](examples/mt-bench/judges/report.md):
+
+| judge | verdict | flip rate | accuracy vs humans | kappa | survives answer swap | first-shown answer picked | padding helped |
+|---|---|---|---|---|---|---|---|
+| qwen2.5:7b-instruct, temp 0 | biased | 0% [0%, 5%] | 72% [62%, 81%] | 0.45 [0.26, 0.65] | 75% [65%, 83%] | 44% | 0% |
+| qwen2.5:7b-instruct, temp 0.8 | biased | 12% [7%, 22%] | 70% [59%, 79%] | 0.40 [0.20, 0.60] | 75% [65%, 83%] | 42% | 1% |
+| qwen2.5:3b-instruct | biased | 12% [7%, 22%] | 56% [45%, 67%] | 0.13 [-0.08, 0.35] | 44% [33%, 55%] | 35% | 9% |
+| gemma2:2b | unstable | 59% [48%, 69%] | 56% [45%, 67%] | 0.14 [-0.07, 0.36] | 38% [28%, 48%] | 35% | 11% |
+| llama3.2:3b | unstable | 56% [45%, 67%] | 51% [40%, 62%] | 0.04 [-0.17, 0.26] | 14% [8%, 23%] | 20% | 10% |
+| control: longer answer wins | biased | 0% [0%, 5%] | 66% [55%, 76%] | 0.32 [0.12, 0.53] | 100% [95%, 100%] | 50% | 30% |
+| control: coin flip | unstable | 95% [88%, 98%] | 51% [40%, 62%] | 0.03 [-0.19, 0.24] | 55% [44%, 65%] | 50% | 25% |
+
+n = 80 pairs per judge; brackets are 95% intervals. No judge passed. The closest,
+qwen2.5 7B, reaches kappa 0.40 to 0.45 with the human experts, but its verdict
+flips on a quarter of the pairs when the answer order is swapped, and setting the
+temperature to 0 removes the run-to-run flips without removing that order effect.
+llama3.2 3B picked whichever answer it saw second 80% of the time. The longer-answer
+rule never flips and agrees with humans 66% of the time, which is why stability and
+accuracy alone are not enough: the padding probe catches it.
+
+The same check on the suite's own pass/fail judge (qwen2.5 7B, 24 cases, no human
+labels) gives `unstable`: its verdict changed across 5 identical calls on 7 of 24
+cases (29%, interval 15% to 49%).
+
+### Human labels, end to end
+
+```sh
+uvx eval-builder label                                   # picks 24 cases, writes label_sheet.html
+uvx eval-builder label import ~/Downloads/labels.jsonl   # what the sheet's Export button saved
+uvx eval-builder judge-check
+```
+
+![The labeling sheet: one case per screen, pass and fail buttons with keyboard shortcuts](docs/label-sheet.png)
+
+Run on the 48 sample conversations with every model's answer kept as a case (47
+cases), three local judges (1,128 calls, 0 errors), and 24 labels entered through the
+sheet in headless Chrome. The labels were made by the developer (Claude Code reading
+each case for him), not by an independent annotator, so read this as a demonstration
+of the flow. Details and every file: [`examples/sample-labeling/`](examples/sample-labeling/).
+
+- `label` split the 24 picks 12/12 between cases the judges called pass and fail; 15
+  are cases where the judges disagree, flip or move under padding.
+- The sheet made no network requests, survived a reload mid-way, and its download was
+  byte-identical to the copy box. `label import` took 24 rows, rejected 0.
+- `judge-check` against those labels (fail 18, pass 6):
+
+| judge | verdict | flip rate | accuracy vs labels | kappa |
+|---|---|---|---|---|
+| qwen2.5:7b-instruct, temp 0 | trustworthy | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
+| qwen2.5:7b-instruct, temp 0.8 | trustworthy | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
+| llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] |
+
+The qwen judges pass the default thresholds, but judge-check also warns that their
+accuracy is no better than always answering "fail" (75% of the labels), and every
+miss went the same way: both passed a reply that said "Here is an allegorical poem"
+and then wrote no poem. With 24 labels the kappa interval runs from about 0.1 to 0.8.
+The DeepEval export wired the 0.8-temperature qwen judge in with its exact prompt;
+DeepEval 4.2.8 ran three logged cases through it and it made the same call on the
+missing poem.
+
+## How it works
+
+| step | what it does | what it uses |
+|---|---|---|
+| `ingest` | Reads OpenAI chat JSONL (fine-tuning style or request/response logs), Anthropic messages, Langfuse trace exports, OpenTelemetry GenAI spans (OTLP JSON, current `gen_ai.input.messages` and older `gen_ai.prompt.N` attributes), or generic input/output JSONL. Normalizes to one schema with input, output, earlier turns, tools called, error flag, user feedback, route and model. Redacts emails, API keys, Luhn-valid card numbers, phone numbers and SSN-shaped numbers by default and reports counts. | Python standard library |
+| `select` | Removes exact duplicates (normalized hash of the user turns) and near-duplicates (character n-gram cosine), clusters the rest by topic, then picks: failures first (errors, negative feedback), at least one case per stratum (route, tool, feedback, any metadata key you name), one central case per cluster, and fills the rest by cluster size with the least similar remaining cases. Every pick records why it was picked. | scikit-learn (TF-IDF, k-means) |
+| `draft`, `validate` | Writes `cases.yaml` and `rubric.yaml` with TODO markers. The agent fills in expected behavior and criteria with you. Validation refuses ready cases that still contain TODO or reference unknown criteria. | PyYAML |
+| `judge-plan` | Lists every judge call to make: each case N times, plus probes that swap the answer order (pairwise judges) and pad an answer with an irrelevant paragraph. | |
+| `judge-run` | Optional and off by default. Sends each request as a JSON line to a command you name (your script, your provider, your keys) and records the verdicts. eval-builder ships no API keys and no provider code. | your command |
+| `label`, `label import` | Picks the ready cases a person should label (default 24): the budget is split across outcomes (the judges' consensus, or the logged failure flag before judges ran) and up to half of each share goes to cases where judges disagree, flip across repeats or move under padding. Writes `label_sheet.html`, one offline file (one case per screen, pass/fail or A/B buttons, a note, keyboard shortcuts, judge verdicts hidden, progress kept in the browser) whose Export button downloads `labels.jsonl` in the format judge-check reads, and `label_sheet.csv` for spreadsheet users. `label import` checks the file against `cases.yaml` and merges it into the workspace. | Python standard library; the sheet is plain HTML and JavaScript |
+| `judge-check` | Per judge: flip rate across repeated calls (with a Wilson interval), self-agreement, majority-of-3 vote stability, accuracy and Cohen's kappa against your human labels (with intervals), position consistency and first-shown preference, and how often padding moved the verdict toward the padded answer. Verdict: `trustworthy`, `unstable`, `biased`, `misaligned`, or `not_enough_data`, with the numbers behind it. | |
+| `export` | promptfoo `promptfooconfig.yaml` (the full conversation as chat messages, llm-rubric asserts, and a judge that passed judge-check wired in as the grader when its rubric entry names a promptfoo `provider`), DeepEval dataset plus a `deepeval test run` file (the same checked judge grades every case with its exact prompt; without one, GEval), Inspect AI dataset plus `task.py` (Inspect's default `model_graded_qa` grader), plain JSONL. The manifest lists file hashes and which judges passed. | |
+| `report` | `report.md` and `report.json`: sources with sha256, counts, redactions, selection reasons, the judge table, and the limits. | |
+
+The verdict thresholds are explicit flags with defaults: flip rate at most 20% of cases,
+position consistency at least 80%, padding helps at most 10% of cases, kappa at least
+0.4 on at least 20 human-labeled cases. A judge without human labels is never called
+trustworthy. When at least 80% of the selected cases or of the human labels share one
+outcome, `select` and `judge-check` say so with the real proportions, because a judge
+that always gives that answer would look accurate on them.
+
+## Setup for agents
+
+```sh
+uvx eval-builder setup          # shows what it would change
+uvx eval-builder setup --yes    # applies it; restart your agent afterwards
+```
+
+`setup` registers the MCP server (`uvx eval-builder mcp`) with Claude Code (`claude mcp
+add --scope user`), Codex (`[mcp_servers.eval-builder]` in `~/.codex/config.toml`) and
+Cursor (`~/.cursor/mcp.json`), and copies the agent instructions to
+`~/.claude/skills/eval-builder/` and `~/.codex/skills/eval-builder/`. It backs up any
+file it edits and does nothing on a second run. The workflow the agent follows is in
+[`skills/eval-builder/SKILL.md`](skills/eval-builder/SKILL.md).
+
+MCP tools: `ingest`, `select`, `draft`, `list_cases`, `update_case`, `set_rubric`,
+`validate`, `judge_plan`, `label`, `label_import`, `judge_check`, `export`, `report`,
+`status`. The judge runner
+is CLI only, because it executes a command.
+
+## What it can't do
+
+- It does not write expected behavior or human labels. The agent drafts expected
+  behavior with you; labels must come from people. Without labels, judge-check can
+  tell you a judge is unstable or biased, but not that it is right. `label` makes the
+  labeling quick, but someone still has to read each case.
+- Picking labels where judges disagree makes each label more informative, but those
+  cases are harder than average, so accuracy measured on them leans pessimistic.
+  `--uncertain-share 0` picks by outcome and topic only.
+- The labeling sheet is a static page, so it cannot save files: the person has to
+  click Export (or copy the text) and import it. Unexported progress lives only in
+  that browser's local storage.
+- Selection is lexical. Two requests that mean the same thing in different words can
+  land in different clusters, and near-duplicate detection only catches close textual
+  matches.
+- Redaction is pattern-based. Names, street addresses and free-form secrets get
+  through. Look at `traces.jsonl` before sharing a workspace.
+- The bias probes cover answer order and irrelevant length only. Self-preference,
+  style bias and rubric misreadings are not measured.
+- Small samples give wide intervals. The report prints them; read them.
+- It does not run your app, your judges or your eval. The agent (or a script you name
+  with `judge-run`) calls the judge model; the exported files run the eval in promptfoo,
+  DeepEval or Inspect AI.
+- A checked judge is wired in as the grader in promptfoo and DeepEval, and only a
+  pointwise one. promptfoo needs its prompt to answer in JSON (`{"pass": ...,
+  "reason": ...}`), because llm-rubric cannot parse a bare "pass". The DeepEval test
+  calls Ollama judges itself; for any other provider you fill in `call_judge`. The
+  Inspect AI export still uses Inspect's default `model_graded_qa` grader, not your
+  checked judge.
+
+## Privacy and safety
+
+Everything runs locally. eval-builder makes no network calls and sends nothing
+anywhere. The only process it starts is the judge command you name with
+`judge-run --enable-judge-plugin`. The labeling sheet is one HTML file with a content
+security policy that blocks every network request; it keeps unexported labels in the
+browser's local storage on that machine. Exported test files can call a model when
+you run them (the DeepEval test calls the judge it was given). Redaction is on unless you pass `--no-redact`, and
+the report lists what was replaced (counts and kinds, never the values).
+
+## License
+
+MIT. See [LICENSE](LICENSE).