eval-builder 0.1.1__tar.gz → 0.1.2__tar.gz

{eval_builder-0.1.1 → eval_builder-0.1.2}/.gitignore

@@ -9,9 +9,11 @@ build/
 *.egg-info/
 .DS_Store
 evalset/
+!examples/sample-labeling/evalset/
 .deepeval/
 # Example inputs that prepare.py downloads or derives (reproducible from the pinned dataset)
 examples/mt-bench/data/
 # Large or derived files inside example workspaces
 examples/mt-bench/*/judge_requests.jsonl
 examples/mt-bench/*/traces.jsonl
+examples/sample-labeling/evalset/judge_requests.jsonl

{eval_builder-0.1.1 → eval_builder-0.1.2}/AGENTS.md

@@ -16,6 +16,9 @@ src/eval_builder/
   judge/run.py     opt-in judge plugin runner (off by default, runs a user command)
   judge/check.py   flip rate, kappa, accuracy, probes, verdicts
   judge/stats.py   Wilson interval, Cohen's kappa, majority vote
+  label.py         which cases a person should label, the CSV sheet, `label import`
+  label_sheet.py   the offline HTML labeling sheet (one file, inline JS, no network)
+  balance.py       the warning when cases or labels are mostly one outcome
   export.py        promptfoo, DeepEval, Inspect AI, JSONL
   report.py        report.md and report.json
   setup_agents.py  `eval-builder setup` for Claude Code, Codex, Cursor
@@ -25,7 +28,8 @@ src/eval_builder/
 
 ## Rules
 
-- No model calls and no network access in the package. The judge runner only starts a
+- No model calls and no network access in the package. (Exported files may call a
+  model when the user runs them, for example the DeepEval test calling a wired judge.) The judge runner only starts a
   command the user names, and only with an explicit flag.
 - Deterministic: same inputs and seed give the same selection and the same files.
 - Every number the tool reports must be traceable to a file in the workspace.

{eval_builder-0.1.1 → eval_builder-0.1.2}/CHANGELOG.md

@@ -1,5 +1,45 @@
 # Changelog
 
+## 0.1.2 (2026-10-08)
+
+Human labels. No judge can be called trustworthy without them, and both real agent
+sessions against 0.1.0 and 0.1.1 stopped at that step: they asked for labels and had
+no way to collect them.
+
+- `label` (CLI, and MCP `label`): picks the ready cases a person should label, 24 by
+  default (judge-check needs 20). The budget is split evenly across outcomes (the
+  judges' consensus verdict, or the logged failure flag before any judge has run), and
+  up to half of each share goes to cases where the judges disagree, flip across
+  repeats or move under padding. Every pick and its reason is in `label_plan.json`.
+- `label` writes `label_sheet.html`: one self-contained file that works offline and
+  makes no network requests (its content security policy blocks them). One case per
+  screen with the earlier turns, the reply, the expected behavior and criteria;
+  pass/fail or A/B buttons (labels come from rubric.yaml), an optional note, keyboard
+  shortcuts, progress, and the judges' verdicts hidden. Progress survives a reload
+  through the browser's local storage. Export downloads `labels.jsonl` in exactly the
+  format judge-check reads and shows the same text to copy.
+- `label` also writes `label_sheet.csv` for spreadsheet users (fill the label column).
+- `label import <file>` (CLI, and MCP `label_import`): reads the sheet's labels.jsonl,
+  the filled-in CSV, a JSON list, or stdin (`-`); checks every case id and label
+  against cases.yaml and rubric.yaml; merges into `labels.jsonl` (a new label replaces
+  the same labeler's earlier one, other labelers are kept); lists rejected rows with
+  the reason and exits 1 when there are any.
+- `select` and `judge-check` warn, with the real proportions, when at least 80% of the
+  selected cases or of the human labels share one outcome. judge-check also warns when
+  a judge gave the same verdict on every labeled case or its accuracy is no better
+  than always giving the most common label, and reports `majority_baseline` (that
+  always-the-common-label accuracy).
+- DeepEval export: a pointwise judge that passed judge-check (or one forced with
+  `--judge`) now grades every case with its exact rubric.yaml prompt, through a custom
+  metric in `test_eval_builder.py` and `judge.json`. Ollama providers are called on the
+  local server; for other providers you fill in `call_judge`. Without a checked judge
+  the file keeps GEval. The export notes and README say that Inspect AI still uses its
+  default `model_graded_qa` grader.
+- judge-check parses JSON verdicts that a token limit cut off mid-reason, and verdicts
+  wrapped in a json code fence.
+- `status` and judge-check's `next` walk through the labeling step.
+- `examples/sample-labeling/`: the whole flow on the bundled 48-conversation sample.
+
 ## 0.1.1 (2026-10-08)
 
 Fixes from a fresh-install audit and a real Claude Code session driving the MCP server.

{eval_builder-0.1.1 → eval_builder-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: eval-builder
-Version: 0.1.1
+Version: 0.1.2
 Summary: Turn real LLM app logs into an eval suite and measure which LLM judges you can trust.
 Author: Abel Yagubyan
 License-Expression: MIT
@@ -29,18 +29,26 @@ uvx eval-builder ingest sample_logs.jsonl     # or your own logs: OpenAI, Anthro
 uvx eval-builder select -n 10 --stratify category && uvx eval-builder draft
 ```
 
-What you get (real output, eval-builder 0.1.1 on the sample file):
+![The quickstart running against eval-builder 0.1.1 from PyPI](docs/demo.gif)
+
+What you get (real output of the commands above, eval-builder 0.1.1 from PyPI; the
+curl and the three uvx calls took 35 s in total here; the very first uvx run also
+downloads about 38 MiB, mostly numpy, scipy and scikit-learn, which took 6 s here):
 
 ```
 ingested 48 traces into evalset/traces.jsonl
   sample_logs.jsonl: format=openai records=48 traces=48 skipped=0 sha256=2d5dd2295784988b
 redactions: 0 {}
+next: eval-builder select -n 30 (add --stratify <metadata keys> to cover them)
 48 traces -> 16 unique (32 exact dupes, 0 near dupes) -> selected 10 (8 failures) across 4 clusters
   q121-llama-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
+  q81-alpaca-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
   q102-alpaca-13b: covers category=reasoning (2 unique traces, 12%)
   q111-alpaca-13b: adds variety within cluster 1 (8 traces, 50%; triangle, response, person); least similar to cases already picked there
-  ...
+  ... (6 more picks)
+next: run draft to turn the selection into cases.yaml
 10 case(s) added, 10 total in evalset/cases.yaml
+next: read the cases (list_cases, or cases.yaml), define criteria and judges (set_rubric, or rubric.yaml), fill expected_behavior and criteria per case and set status: ready (update_case), then run validate
 ```
 
 `evalset/cases.yaml` now holds 10 real conversations with a TODO where the expected
@@ -90,7 +98,10 @@ export and report in 32 turns and about 10 minutes. From its final answer:
 > Its verdicts will be unverified.
 
 It wrote every expected behavior itself because nobody was there to confirm them, and
-said so on each case. It did not invent human labels; it asked for them.
+said so on each case. It did not invent human labels; it asked for them. That is where
+this session and the earlier one both stopped, so 0.1.2 adds the missing step: `label`
+writes a sheet the person labels in their browser, and `label import` brings the
+labels back for judge-check ([Human labels, end to end](#human-labels-end-to-end)).
 
 An earlier session against 0.1.0 is where the promptfoo context bug fixed in 0.1.1 came
 from. The agent read the export and told the user: "Right now it only sends the final
@@ -150,6 +161,42 @@ The same check on the suite's own pass/fail judge (qwen2.5 7B, 24 cases, no huma
 labels) gives `unstable`: its verdict changed across 5 identical calls on 7 of 24
 cases (29%, interval 15% to 49%).
 
+### Human labels, end to end
+
+```sh
+uvx eval-builder label                                   # picks 24 cases, writes label_sheet.html
+uvx eval-builder label import ~/Downloads/labels.jsonl   # what the sheet's Export button saved
+uvx eval-builder judge-check
+```
+
+![The labeling sheet: one case per screen, pass and fail buttons with keyboard shortcuts](docs/label-sheet.png)
+
+Run on the 48 sample conversations with every model's answer kept as a case (47
+cases), three local judges (1,128 calls, 0 errors), and 24 labels entered through the
+sheet in headless Chrome. The labels were made by the developer (Claude Code reading
+each case for him), not by an independent annotator, so read this as a demonstration
+of the flow. Details and every file: [`examples/sample-labeling/`](examples/sample-labeling/).
+
+- `label` split the 24 picks 12/12 between cases the judges called pass and fail; 15
+  are cases where the judges disagree, flip or move under padding.
+- The sheet made no network requests, survived a reload mid-way, and its download was
+  byte-identical to the copy box. `label import` took 24 rows, rejected 0.
+- `judge-check` against those labels (fail 18, pass 6):
+
+| judge | verdict | flip rate | accuracy vs labels | kappa |
+|---|---|---|---|---|
+| qwen2.5:7b-instruct, temp 0 | trustworthy | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
+| qwen2.5:7b-instruct, temp 0.8 | trustworthy | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
+| llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] |
+
+The qwen judges pass the default thresholds, but judge-check also warns that their
+accuracy is no better than always answering "fail" (75% of the labels), and every
+miss went the same way: both passed a reply that said "Here is an allegorical poem"
+and then wrote no poem. With 24 labels the kappa interval runs from about 0.1 to 0.8.
+The DeepEval export wired the 0.8-temperature qwen judge in with its exact prompt;
+DeepEval 4.2.8 ran three logged cases through it and it made the same call on the
+missing poem.
+
 ## How it works
 
 | step | what it does | what it uses |
@@ -159,14 +206,17 @@ cases (29%, interval 15% to 49%).
 | `draft`, `validate` | Writes `cases.yaml` and `rubric.yaml` with TODO markers. The agent fills in expected behavior and criteria with you. Validation refuses ready cases that still contain TODO or reference unknown criteria. | PyYAML |
 | `judge-plan` | Lists every judge call to make: each case N times, plus probes that swap the answer order (pairwise judges) and pad an answer with an irrelevant paragraph. | |
 | `judge-run` | Optional and off by default. Sends each request as a JSON line to a command you name (your script, your provider, your keys) and records the verdicts. eval-builder ships no API keys and no provider code. | your command |
+| `label`, `label import` | Picks the ready cases a person should label (default 24): the budget is split across outcomes (the judges' consensus, or the logged failure flag before judges ran) and up to half of each share goes to cases where judges disagree, flip across repeats or move under padding. Writes `label_sheet.html`, one offline file (one case per screen, pass/fail or A/B buttons, a note, keyboard shortcuts, judge verdicts hidden, progress kept in the browser) whose Export button downloads `labels.jsonl` in the format judge-check reads, and `label_sheet.csv` for spreadsheet users. `label import` checks the file against `cases.yaml` and merges it into the workspace. | Python standard library; the sheet is plain HTML and JavaScript |
 | `judge-check` | Per judge: flip rate across repeated calls (with a Wilson interval), self-agreement, majority-of-3 vote stability, accuracy and Cohen's kappa against your human labels (with intervals), position consistency and first-shown preference, and how often padding moved the verdict toward the padded answer. Verdict: `trustworthy`, `unstable`, `biased`, `misaligned`, or `not_enough_data`, with the numbers behind it. | |
-| `export` | promptfoo `promptfooconfig.yaml` (the full conversation as chat messages, llm-rubric asserts, and a judge that passed judge-check wired in as the grader when its rubric entry names a promptfoo `provider`), DeepEval dataset plus a `deepeval test run` file, Inspect AI dataset plus `task.py`, plain JSONL. The manifest lists file hashes and which judges passed. | |
+| `export` | promptfoo `promptfooconfig.yaml` (the full conversation as chat messages, llm-rubric asserts, and a judge that passed judge-check wired in as the grader when its rubric entry names a promptfoo `provider`), DeepEval dataset plus a `deepeval test run` file (the same checked judge grades every case with its exact prompt; without one, GEval), Inspect AI dataset plus `task.py` (Inspect's default `model_graded_qa` grader), plain JSONL. The manifest lists file hashes and which judges passed. | |
 | `report` | `report.md` and `report.json`: sources with sha256, counts, redactions, selection reasons, the judge table, and the limits. | |
 
 The verdict thresholds are explicit flags with defaults: flip rate at most 20% of cases,
 position consistency at least 80%, padding helps at most 10% of cases, kappa at least
 0.4 on at least 20 human-labeled cases. A judge without human labels is never called
-trustworthy.
+trustworthy. When at least 80% of the selected cases or of the human labels share one
+outcome, `select` and `judge-check` say so with the real proportions, because a judge
+that always gives that answer would look accurate on them.
 
 ## Setup for agents
 
@@ -183,14 +233,22 @@ file it edits and does nothing on a second run. The workflow the agent follows i
 [`skills/eval-builder/SKILL.md`](skills/eval-builder/SKILL.md).
 
 MCP tools: `ingest`, `select`, `draft`, `list_cases`, `update_case`, `set_rubric`,
-`validate`, `judge_plan`, `judge_check`, `export`, `report`, `status`. The judge runner
+`validate`, `judge_plan`, `label`, `label_import`, `judge_check`, `export`, `report`,
+`status`. The judge runner
 is CLI only, because it executes a command.
 
 ## What it can't do
 
 - It does not write expected behavior or human labels. The agent drafts expected
   behavior with you; labels must come from people. Without labels, judge-check can
-  tell you a judge is unstable or biased, but not that it is right.
+  tell you a judge is unstable or biased, but not that it is right. `label` makes the
+  labeling quick, but someone still has to read each case.
+- Picking labels where judges disagree makes each label more informative, but those
+  cases are harder than average, so accuracy measured on them leans pessimistic.
+  `--uncertain-share 0` picks by outcome and topic only.
+- The labeling sheet is a static page, so it cannot save files: the person has to
+  click Export (or copy the text) and import it. Unexported progress lives only in
+  that browser's local storage.
 - Selection is lexical. Two requests that mean the same thing in different words can
   land in different clusters, and near-duplicate detection only catches close textual
   matches.
@@ -202,16 +260,21 @@ is CLI only, because it executes a command.
 - It does not run your app, your judges or your eval. The agent (or a script you name
   with `judge-run`) calls the judge model; the exported files run the eval in promptfoo,
   DeepEval or Inspect AI.
-- Only the promptfoo export wires a checked judge in as the grader, and only a
-  pointwise judge whose prompt answers in JSON (`{"pass": ..., "reason": ...}`), because
-  promptfoo's llm-rubric cannot parse a bare "pass". DeepEval and Inspect exports use
-  their own default graders.
+- A checked judge is wired in as the grader in promptfoo and DeepEval, and only a
+  pointwise one. promptfoo needs its prompt to answer in JSON (`{"pass": ...,
+  "reason": ...}`), because llm-rubric cannot parse a bare "pass". The DeepEval test
+  calls Ollama judges itself; for any other provider you fill in `call_judge`. The
+  Inspect AI export still uses Inspect's default `model_graded_qa` grader, not your
+  checked judge.
 
 ## Privacy and safety
 
 Everything runs locally. eval-builder makes no network calls and sends nothing
 anywhere. The only process it starts is the judge command you name with
-`judge-run --enable-judge-plugin`. Redaction is on unless you pass `--no-redact`, and
+`judge-run --enable-judge-plugin`. The labeling sheet is one HTML file with a content
+security policy that blocks every network request; it keeps unexported labels in the
+browser's local storage on that machine. Exported test files can call a model when
+you run them (the DeepEval test calls the judge it was given). Redaction is on unless you pass `--no-redact`, and
 the report lists what was replaced (counts and kinds, never the values).
 
 ## License

{eval_builder-0.1.1 → eval_builder-0.1.2}/README.md

@@ -10,18 +10,26 @@ uvx eval-builder ingest sample_logs.jsonl     # or your own logs: OpenAI, Anthro
 uvx eval-builder select -n 10 --stratify category && uvx eval-builder draft
 ```
 
-What you get (real output, eval-builder 0.1.1 on the sample file):
+![The quickstart running against eval-builder 0.1.1 from PyPI](docs/demo.gif)
+
+What you get (real output of the commands above, eval-builder 0.1.1 from PyPI; the
+curl and the three uvx calls took 35 s in total here; the very first uvx run also
+downloads about 38 MiB, mostly numpy, scipy and scikit-learn, which took 6 s here):
 
 ```
 ingested 48 traces into evalset/traces.jsonl
   sample_logs.jsonl: format=openai records=48 traces=48 skipped=0 sha256=2d5dd2295784988b
 redactions: 0 {}
+next: eval-builder select -n 30 (add --stratify <metadata keys> to cover them)
 48 traces -> 16 unique (32 exact dupes, 0 near dupes) -> selected 10 (8 failures) across 4 clusters
   q121-llama-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
+  q81-alpaca-13b: failure (negative user feedback); 69% of unique traces are failures and at least 30% of picks are reserved for them
   q102-alpaca-13b: covers category=reasoning (2 unique traces, 12%)
   q111-alpaca-13b: adds variety within cluster 1 (8 traces, 50%; triangle, response, person); least similar to cases already picked there
-  ...
+  ... (6 more picks)
+next: run draft to turn the selection into cases.yaml
 10 case(s) added, 10 total in evalset/cases.yaml
+next: read the cases (list_cases, or cases.yaml), define criteria and judges (set_rubric, or rubric.yaml), fill expected_behavior and criteria per case and set status: ready (update_case), then run validate
 ```
 
 `evalset/cases.yaml` now holds 10 real conversations with a TODO where the expected
@@ -71,7 +79,10 @@ export and report in 32 turns and about 10 minutes. From its final answer:
 > Its verdicts will be unverified.
 
 It wrote every expected behavior itself because nobody was there to confirm them, and
-said so on each case. It did not invent human labels; it asked for them.
+said so on each case. It did not invent human labels; it asked for them. That is where
+this session and the earlier one both stopped, so 0.1.2 adds the missing step: `label`
+writes a sheet the person labels in their browser, and `label import` brings the
+labels back for judge-check ([Human labels, end to end](#human-labels-end-to-end)).
 
 An earlier session against 0.1.0 is where the promptfoo context bug fixed in 0.1.1 came
 from. The agent read the export and told the user: "Right now it only sends the final
@@ -131,6 +142,42 @@ The same check on the suite's own pass/fail judge (qwen2.5 7B, 24 cases, no huma
 labels) gives `unstable`: its verdict changed across 5 identical calls on 7 of 24
 cases (29%, interval 15% to 49%).
 
+### Human labels, end to end
+
+```sh
+uvx eval-builder label                                   # picks 24 cases, writes label_sheet.html
+uvx eval-builder label import ~/Downloads/labels.jsonl   # what the sheet's Export button saved
+uvx eval-builder judge-check
+```
+
+![The labeling sheet: one case per screen, pass and fail buttons with keyboard shortcuts](docs/label-sheet.png)
+
+Run on the 48 sample conversations with every model's answer kept as a case (47
+cases), three local judges (1,128 calls, 0 errors), and 24 labels entered through the
+sheet in headless Chrome. The labels were made by the developer (Claude Code reading
+each case for him), not by an independent annotator, so read this as a demonstration
+of the flow. Details and every file: [`examples/sample-labeling/`](examples/sample-labeling/).
+
+- `label` split the 24 picks 12/12 between cases the judges called pass and fail; 15
+  are cases where the judges disagree, flip or move under padding.
+- The sheet made no network requests, survived a reload mid-way, and its download was
+  byte-identical to the copy box. `label import` took 24 rows, rejected 0.
+- `judge-check` against those labels (fail 18, pass 6):
+
+| judge | verdict | flip rate | accuracy vs labels | kappa |
+|---|---|---|---|---|
+| qwen2.5:7b-instruct, temp 0 | trustworthy | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
+| qwen2.5:7b-instruct, temp 0.8 | trustworthy | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
+| llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] |
+
+The qwen judges pass the default thresholds, but judge-check also warns that their
+accuracy is no better than always answering "fail" (75% of the labels), and every
+miss went the same way: both passed a reply that said "Here is an allegorical poem"
+and then wrote no poem. With 24 labels the kappa interval runs from about 0.1 to 0.8.
+The DeepEval export wired the 0.8-temperature qwen judge in with its exact prompt;
+DeepEval 4.2.8 ran three logged cases through it and it made the same call on the
+missing poem.
+
 ## How it works
 
 | step | what it does | what it uses |
@@ -140,14 +187,17 @@ cases (29%, interval 15% to 49%).
 | `draft`, `validate` | Writes `cases.yaml` and `rubric.yaml` with TODO markers. The agent fills in expected behavior and criteria with you. Validation refuses ready cases that still contain TODO or reference unknown criteria. | PyYAML |
 | `judge-plan` | Lists every judge call to make: each case N times, plus probes that swap the answer order (pairwise judges) and pad an answer with an irrelevant paragraph. | |
 | `judge-run` | Optional and off by default. Sends each request as a JSON line to a command you name (your script, your provider, your keys) and records the verdicts. eval-builder ships no API keys and no provider code. | your command |
+| `label`, `label import` | Picks the ready cases a person should label (default 24): the budget is split across outcomes (the judges' consensus, or the logged failure flag before judges ran) and up to half of each share goes to cases where judges disagree, flip across repeats or move under padding. Writes `label_sheet.html`, one offline file (one case per screen, pass/fail or A/B buttons, a note, keyboard shortcuts, judge verdicts hidden, progress kept in the browser) whose Export button downloads `labels.jsonl` in the format judge-check reads, and `label_sheet.csv` for spreadsheet users. `label import` checks the file against `cases.yaml` and merges it into the workspace. | Python standard library; the sheet is plain HTML and JavaScript |
 | `judge-check` | Per judge: flip rate across repeated calls (with a Wilson interval), self-agreement, majority-of-3 vote stability, accuracy and Cohen's kappa against your human labels (with intervals), position consistency and first-shown preference, and how often padding moved the verdict toward the padded answer. Verdict: `trustworthy`, `unstable`, `biased`, `misaligned`, or `not_enough_data`, with the numbers behind it. | |
-| `export` | promptfoo `promptfooconfig.yaml` (the full conversation as chat messages, llm-rubric asserts, and a judge that passed judge-check wired in as the grader when its rubric entry names a promptfoo `provider`), DeepEval dataset plus a `deepeval test run` file, Inspect AI dataset plus `task.py`, plain JSONL. The manifest lists file hashes and which judges passed. | |
+| `export` | promptfoo `promptfooconfig.yaml` (the full conversation as chat messages, llm-rubric asserts, and a judge that passed judge-check wired in as the grader when its rubric entry names a promptfoo `provider`), DeepEval dataset plus a `deepeval test run` file (the same checked judge grades every case with its exact prompt; without one, GEval), Inspect AI dataset plus `task.py` (Inspect's default `model_graded_qa` grader), plain JSONL. The manifest lists file hashes and which judges passed. | |
 | `report` | `report.md` and `report.json`: sources with sha256, counts, redactions, selection reasons, the judge table, and the limits. | |
 
 The verdict thresholds are explicit flags with defaults: flip rate at most 20% of cases,
 position consistency at least 80%, padding helps at most 10% of cases, kappa at least
 0.4 on at least 20 human-labeled cases. A judge without human labels is never called
-trustworthy.
+trustworthy. When at least 80% of the selected cases or of the human labels share one
+outcome, `select` and `judge-check` say so with the real proportions, because a judge
+that always gives that answer would look accurate on them.
 
 ## Setup for agents
 
@@ -164,14 +214,22 @@ file it edits and does nothing on a second run. The workflow the agent follows i
 [`skills/eval-builder/SKILL.md`](skills/eval-builder/SKILL.md).
 
 MCP tools: `ingest`, `select`, `draft`, `list_cases`, `update_case`, `set_rubric`,
-`validate`, `judge_plan`, `judge_check`, `export`, `report`, `status`. The judge runner
+`validate`, `judge_plan`, `label`, `label_import`, `judge_check`, `export`, `report`,
+`status`. The judge runner
 is CLI only, because it executes a command.
 
 ## What it can't do
 
 - It does not write expected behavior or human labels. The agent drafts expected
   behavior with you; labels must come from people. Without labels, judge-check can
-  tell you a judge is unstable or biased, but not that it is right.
+  tell you a judge is unstable or biased, but not that it is right. `label` makes the
+  labeling quick, but someone still has to read each case.
+- Picking labels where judges disagree makes each label more informative, but those
+  cases are harder than average, so accuracy measured on them leans pessimistic.
+  `--uncertain-share 0` picks by outcome and topic only.
+- The labeling sheet is a static page, so it cannot save files: the person has to
+  click Export (or copy the text) and import it. Unexported progress lives only in
+  that browser's local storage.
 - Selection is lexical. Two requests that mean the same thing in different words can
   land in different clusters, and near-duplicate detection only catches close textual
   matches.
@@ -183,16 +241,21 @@ is CLI only, because it executes a command.
 - It does not run your app, your judges or your eval. The agent (or a script you name
   with `judge-run`) calls the judge model; the exported files run the eval in promptfoo,
   DeepEval or Inspect AI.
-- Only the promptfoo export wires a checked judge in as the grader, and only a
-  pointwise judge whose prompt answers in JSON (`{"pass": ..., "reason": ...}`), because
-  promptfoo's llm-rubric cannot parse a bare "pass". DeepEval and Inspect exports use
-  their own default graders.
+- A checked judge is wired in as the grader in promptfoo and DeepEval, and only a
+  pointwise one. promptfoo needs its prompt to answer in JSON (`{"pass": ...,
+  "reason": ...}`), because llm-rubric cannot parse a bare "pass". The DeepEval test
+  calls Ollama judges itself; for any other provider you fill in `call_judge`. The
+  Inspect AI export still uses Inspect's default `model_graded_qa` grader, not your
+  checked judge.
 
 ## Privacy and safety
 
 Everything runs locally. eval-builder makes no network calls and sends nothing
 anywhere. The only process it starts is the judge command you name with
-`judge-run --enable-judge-plugin`. Redaction is on unless you pass `--no-redact`, and
+`judge-run --enable-judge-plugin`. The labeling sheet is one HTML file with a content
+security policy that blocks every network request; it keeps unexported labels in the
+browser's local storage on that machine. Exported test files can call a model when
+you run them (the DeepEval test calls the judge it was given). Redaction is on unless you pass `--no-redact`, and
 the report lists what was replaced (counts and kinds, never the values).
 
 ## License

eval_builder-0.1.2/docs/demo.tape

@@ -0,0 +1,25 @@
+Output docs/demo.gif
+Set FontSize 14
+Set Width 1600
+Set Height 900
+Set Theme "Builtin Dark"
+Set TypingSpeed 30ms
+Hide
+Type "cd \"$(mktemp -d)\" && clear"
+Enter
+Show
+Type "curl -sLO https://raw.githubusercontent.com/Abelo9996/eval-builder/main/examples/sample_logs.jsonl"
+Enter
+Wait
+Type "uvx eval-builder ingest sample_logs.jsonl"
+Enter
+Wait
+Sleep 1.5s
+Type "uvx eval-builder select -n 10 --stratify category"
+Enter
+Wait+Screen@20s /next: run draft/
+Sleep 2s
+Type "uvx eval-builder draft && grep -m1 -A10 'id: case-002' evalset/cases.yaml | cut -c1-120"
+Enter
+Wait+Screen@20s /role: user/
+Sleep 5s

eval_builder-0.1.2/examples/sample-labeling/README.md

@@ -0,0 +1,79 @@
+# Labeling example
+
+The human-labeling step, end to end, on the 48 conversations in
+[`../sample_logs.jsonl`](../sample_logs.jsonl) (16 MT-Bench questions, each answered
+by gpt-4, alpaca-13b and llama-13b; CC BY 4.0). Everything here was produced by
+[`run.sh`](run.sh) on an Apple M4 MacBook (16 GB) on October 8 and 9, 2026, with
+eval-builder 0.1.2 from this repository.
+
+**Who labeled.** The labels in [`evalset/labels.jsonl`](evalset/labels.jsonl) were made
+by the developer, not by an independent annotator: Claude Code, working for the
+developer, read each picked case in full (earlier turns, reply, expected behavior) and
+recorded a decision with a note for the close calls ([`decisions.json`](decisions.json)).
+[`drive_sheet.py`](drive_sheet.py) then entered those decisions through the real
+`label_sheet.html` in headless Chrome, using the keyboard shortcuts, and clicked Export.
+The expected behavior in [`fill.py`](fill.py) was written the same way. Treat the
+numbers below as a demonstration of the flow, not as ground truth about these judges.
+
+## Steps and what they printed
+
+1. `select --dedupe-on input+output --near-dup 0.99` kept each model's answer: 48
+   traces, 47 unique (two answers to one question were near-identical). All 47 became
+   ready cases with one expected behavior per question.
+2. Three local judges through Ollama (`examples/judges/ollama_judge.py`), one pass/fail
+   prompt that answers in JSON, 5 trials per case plus 3 with an irrelevant paragraph
+   appended to the reply: 1,128 calls, 0 errors (qwen2.5:7b-instruct at temperature
+   0.8: 1,114 s; the same model at temperature 0: 1,130 s; llama3.2:3b: 589 s; the
+   machine was shared with other jobs).
+3. `judge-check` without labels: llama3.2-3b `unstable` (verdict changed across repeats
+   on 47% of cases), both qwen judges `not_enough_data` ("stable, but only 0 case(s)
+   have human labels").
+4. `label` picked 24 of the 47 cases: 12 where the judges' consensus was fail and 12
+   where it was pass; 15 of the 24 are cases where judges disagree, flip or move under
+   padding ([`evalset/label_plan.json`](evalset/label_plan.json) has every reason).
+5. The sheet in headless Chrome ([`sheet-run/drive_log.txt`](sheet-run/drive_log.txt)):
+   24 cases, a reload after 13 labels kept "13 of 24 labeled", the download
+   (`labels.jsonl`, 4,292 bytes) was identical to the copy box, 2 requests in total,
+   none of them to anything but `file:`, `blob:` or `data:` URLs, no console errors. Screenshots:
+   [first case](sheet-run/sheet-first-case.png), [phone width](sheet-run/sheet-phone.png),
+   [export](sheet-run/sheet-export.png).
+6. `label import`: 24 imported, 0 rejected; labels fail 18, pass 6.
+7. `judge-check` with the labels ([`evalset/report.md`](evalset/report.md)):
+
+| judge | verdict | flip rate | accuracy vs labels | kappa | padding helped |
+|---|---|---|---|---|---|
+| qwen2.5:7b-instruct, temp 0 | trustworthy | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] | 0% [0%, 8%] |
+| qwen2.5:7b-instruct, temp 0.8 | trustworthy | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] | 0% [0%, 8%] |
+| llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] | 4% [1%, 14%] |
+
+n = 47 cases for flip rate and padding, 24 labeled cases for accuracy and kappa;
+brackets are 95% intervals.
+
+## What it shows, read plainly
+
+- Both qwen judges clear the default thresholds (kappa at least 0.4 on at least 20
+  labels), so the export wires `qwen2.5-7b` (the first that passed) into promptfoo and
+  DeepEval. The kappa intervals are wide, from about 0.1 to 0.8: 24 labels cannot pin
+  the agreement down.
+- Their accuracy (71% and 75%) is no better than always answering "fail" (75% of the
+  labels are fail), and judge-check now says so in a warning. Every miss went the same
+  way: the judge passed a reply the labels failed. Both qwen judges passed case-018
+  (llama-13b answered "Here is an allegorical poem that illustrates the above:" and
+  no poem) and case-019 (alpaca-13b's rewrite in which most sentences do not start with
+  "A").
+- The labels came out 18 fail to 6 pass even though `label` split the picks 12/12 by
+  the judges' consensus: the judges said pass far more often than the labels did.
+- llama3.2:3b changes its verdict on almost half the cases between identical calls.
+
+## DeepEval with the checked judge
+
+`export` wrote `evalset/exports/deepeval/judge.json` (judge `qwen2.5-7b`, its exact
+prompt, provider `ollama:chat:qwen2.5:7b-instruct` at temperature 0.8). DeepEval 4.2.8
+ran three of the cases on the logged outputs
+(`EVAL_BUILDER_USE_OBSERVED=1 pytest test_eval_builder.py -k "case-018 or case-020 or case-040"`):
+case-020 passed, case-040 failed with the judge's reason
+`{"pass": false, "reason": "Incorrect circumradius calculation."}`, and case-018 (the
+missing poem) passed, the same mistake judge-check found.
+
+`evalset/judge_requests.jsonl` (4.7 MB of rendered prompts) is not committed; `judge-plan`
+rebuilds it from `cases.yaml` and `rubric.yaml`.

eval_builder-0.1.2/examples/sample-labeling/decisions.json

@@ -0,0 +1,98 @@
+{
+ "case-006": {
+  "label": "fail",
+  "note": ""
+ },
+ "case-007": {
+  "label": "fail",
+  "note": "same textbook definitions, not for a five-year-old"
+ },
+ "case-008": {
+  "label": "pass",
+  "note": ""
+ },
+ "case-009": {
+  "label": "fail",
+  "note": "repeats one definition, not simplified"
+ },
+ "case-010": {
+  "label": "fail",
+  "note": "one sentence, no program"
+ },
+ "case-012": {
+  "label": "fail",
+  "note": ""
+ },
+ "case-013": {
+  "label": "fail",
+  "note": "wrong start values and sums two terms, not three"
+ },
+ "case-015": {
+  "label": "fail",
+  "note": "Fibonacci-style, wrong recurrence"
+ },
+ "case-016": {
+  "label": "fail",
+  "note": "names the stages literally; not an allegory"
+ },
+ "case-018": {
+  "label": "fail",
+  "note": "no poem"
+ },
+ "case-019": {
+  "label": "fail",
+  "note": "most sentences do not start with A"
+ },
+ "case-023": {
+  "label": "pass",
+  "note": "names one improvement; misses that the email is not short"
+ },
+ "case-024": {
+  "label": "fail",
+  "note": "answers a different message"
+ },
+ "case-025": {
+  "label": "fail",
+  "note": "'US President' and 'Lewis' are not specific people"
+ },
+ "case-027": {
+  "label": "fail",
+  "note": "three lines, Washington instead of Roosevelt, da Vinci under science"
+ },
+ "case-029": {
+  "label": "pass",
+  "note": "explains why it is a contradiction"
+ },
+ "case-031": {
+  "label": "pass",
+  "note": "says no, without explaining why"
+ },
+ "case-032": {
+  "label": "pass",
+  "note": ""
+ },
+ "case-034": {
+  "label": "fail",
+  "note": "says $4000, should be $2000"
+ },
+ "case-036": {
+  "label": "fail",
+  "note": "ratings reordered and only one date"
+ },
+ "case-039": {
+  "label": "fail",
+  "note": "12 is wrong; 5*pi"
+ },
+ "case-042": {
+  "label": "fail",
+  "note": "restates definitions, no real assumptions examined"
+ },
+ "case-043": {
+  "label": "pass",
+  "note": ""
+ },
+ "case-047": {
+  "label": "fail",
+  "note": "repeats a wrong claim"
+ }
+}