eval-builder 0.1.2__tar.gz → 0.1.4__tar.gz

{eval_builder-0.1.2 → eval_builder-0.1.4}/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 0.1.4
+
+### Fixed
+
+- Require mcp 1.30 or newer. The old floor (`mcp>=1.2`) allowed releases that no longer import with current pydantic; the MCP server's tests pass on 1.30.0 and 2.3.0.
+
+## 0.1.3
+
+### Fixed
+
+- `judge-check` no longer calls a judge `trustworthy` when its accuracy against the human labels is no better than always giving the most common label. Kappa alone let a lenient judge pass: on the labeling example (18 of 24 labels "fail") both qwen2.5 judges cleared kappa 0.4 at 71% and 75% accuracy, the same as always answering "fail". They are now `misaligned`, with the baseline in the reason. Set `Thresholds(beat_majority_baseline=False)` to get the old behavior.
+- Because no judge passes on that example any more, `export` no longer wires one into promptfoo or DeepEval there; the example README says what changed.
+
 ## 0.1.2 (2026-10-08)
 
 Human labels. No judge can be called trustworthy without them, and both real agent

{eval_builder-0.1.2 → eval_builder-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: eval-builder
-Version: 0.1.2
+Version: 0.1.4
 Summary: Turn real LLM app logs into an eval suite and measure which LLM judges you can trust.
 Author: Abel Yagubyan
 License-Expression: MIT
@@ -10,7 +10,7 @@ Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.11
-Requires-Dist: mcp>=1.2
+Requires-Dist: mcp>=1.30
 Requires-Dist: numpy>=1.24
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: scikit-learn>=1.3
@@ -173,8 +173,8 @@ uvx eval-builder judge-check
 
 Run on the 48 sample conversations with every model's answer kept as a case (47
 cases), three local judges (1,128 calls, 0 errors), and 24 labels entered through the
-sheet in headless Chrome. The labels were made by the developer (Claude Code reading
-each case for him), not by an independent annotator, so read this as a demonstration
+sheet in headless Chrome. The labels were made on the developer's side (Claude Code
+reading each case), not by an independent annotator, so read this as a demonstration
 of the flow. Details and every file: [`examples/sample-labeling/`](examples/sample-labeling/).
 
 - `label` split the 24 picks 12/12 between cases the judges called pass and fail; 15
@@ -185,17 +185,19 @@ of the flow. Details and every file: [`examples/sample-labeling/`](examples/samp
 
 | judge | verdict | flip rate | accuracy vs labels | kappa |
 |---|---|---|---|---|
-| qwen2.5:7b-instruct, temp 0 | trustworthy | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
-| qwen2.5:7b-instruct, temp 0.8 | trustworthy | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
+| qwen2.5:7b-instruct, temp 0 | misaligned | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
+| qwen2.5:7b-instruct, temp 0.8 | misaligned | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
 | llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] |
 
-The qwen judges pass the default thresholds, but judge-check also warns that their
-accuracy is no better than always answering "fail" (75% of the labels), and every
-miss went the same way: both passed a reply that said "Here is an allegorical poem"
-and then wrote no poem. With 24 labels the kappa interval runs from about 0.1 to 0.8.
-The DeepEval export wired the 0.8-temperature qwen judge in with its exact prompt;
-DeepEval 4.2.8 ran three logged cases through it and it made the same call on the
-missing poem.
+The qwen judges clear the kappa threshold (0.4), but their accuracy is no better than
+always answering "fail" (75% of the labels), so since 0.1.3 judge-check calls them
+`misaligned` instead of trustworthy (0.1.2 called them trustworthy; that was the bug).
+Every miss went the same way: both passed a reply that said "Here is an allegorical
+poem" and then wrote no poem. With 24 labels the kappa interval runs from about 0.1
+to 0.8. Under 0.1.2 the DeepEval export wired the 0.8-temperature qwen judge in with
+its exact prompt; DeepEval 4.2.8 ran three logged cases through it and it made the
+same call on the missing poem. Now no judge is wired in unless you force one with
+`export --judge <id>`, which warns.
 
 ## How it works
 

{eval_builder-0.1.2 → eval_builder-0.1.4}/README.md

@@ -154,8 +154,8 @@ uvx eval-builder judge-check
 
 Run on the 48 sample conversations with every model's answer kept as a case (47
 cases), three local judges (1,128 calls, 0 errors), and 24 labels entered through the
-sheet in headless Chrome. The labels were made by the developer (Claude Code reading
-each case for him), not by an independent annotator, so read this as a demonstration
+sheet in headless Chrome. The labels were made on the developer's side (Claude Code
+reading each case), not by an independent annotator, so read this as a demonstration
 of the flow. Details and every file: [`examples/sample-labeling/`](examples/sample-labeling/).
 
 - `label` split the 24 picks 12/12 between cases the judges called pass and fail; 15
@@ -166,17 +166,19 @@ of the flow. Details and every file: [`examples/sample-labeling/`](examples/samp
 
 | judge | verdict | flip rate | accuracy vs labels | kappa |
 |---|---|---|---|---|
-| qwen2.5:7b-instruct, temp 0 | trustworthy | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
-| qwen2.5:7b-instruct, temp 0.8 | trustworthy | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
+| qwen2.5:7b-instruct, temp 0 | misaligned | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] |
+| qwen2.5:7b-instruct, temp 0.8 | misaligned | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] |
 | llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] |
 
-The qwen judges pass the default thresholds, but judge-check also warns that their
-accuracy is no better than always answering "fail" (75% of the labels), and every
-miss went the same way: both passed a reply that said "Here is an allegorical poem"
-and then wrote no poem. With 24 labels the kappa interval runs from about 0.1 to 0.8.
-The DeepEval export wired the 0.8-temperature qwen judge in with its exact prompt;
-DeepEval 4.2.8 ran three logged cases through it and it made the same call on the
-missing poem.
+The qwen judges clear the kappa threshold (0.4), but their accuracy is no better than
+always answering "fail" (75% of the labels), so since 0.1.3 judge-check calls them
+`misaligned` instead of trustworthy (0.1.2 called them trustworthy; that was the bug).
+Every miss went the same way: both passed a reply that said "Here is an allegorical
+poem" and then wrote no poem. With 24 labels the kappa interval runs from about 0.1
+to 0.8. Under 0.1.2 the DeepEval export wired the 0.8-temperature qwen judge in with
+its exact prompt; DeepEval 4.2.8 ran three logged cases through it and it made the
+same call on the missing poem. Now no judge is wired in unless you force one with
+`export --judge <id>`, which warns.
 
 ## How it works
 

{eval_builder-0.1.2 → eval_builder-0.1.4}/examples/sample-labeling/README.md

@@ -42,8 +42,8 @@ numbers below as a demonstration of the flow, not as ground truth about these ju
 
 | judge | verdict | flip rate | accuracy vs labels | kappa | padding helped |
 |---|---|---|---|---|---|
-| qwen2.5:7b-instruct, temp 0 | trustworthy | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] | 0% [0%, 8%] |
-| qwen2.5:7b-instruct, temp 0.8 | trustworthy | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] | 0% [0%, 8%] |
+| qwen2.5:7b-instruct, temp 0 | misaligned | 0% [0%, 8%] | 75% [55%, 88%] | 0.50 [0.15, 0.85] | 0% [0%, 8%] |
+| qwen2.5:7b-instruct, temp 0.8 | misaligned | 9% [3%, 20%] | 71% [51%, 85%] | 0.44 [0.09, 0.79] | 0% [0%, 8%] |
 | llama3.2:3b | unstable | 47% [33%, 61%] | 54% [35%, 72%] | 0.12 [-0.26, 0.50] | 4% [1%, 14%] |
 
 n = 47 cases for flip rate and padding, 24 labeled cases for accuracy and kappa;
@@ -51,12 +51,12 @@ brackets are 95% intervals.
 
 ## What it shows, read plainly
 
-- Both qwen judges clear the default thresholds (kappa at least 0.4 on at least 20
-  labels), so the export wires `qwen2.5-7b` (the first that passed) into promptfoo and
-  DeepEval. The kappa intervals are wide, from about 0.1 to 0.8: 24 labels cannot pin
-  the agreement down.
-- Their accuracy (71% and 75%) is no better than always answering "fail" (75% of the
-  labels are fail), and judge-check now says so in a warning. Every miss went the same
+- Both qwen judges clear the kappa threshold (at least 0.4 on at least 20 labels), but
+  their accuracy (71% and 75%) is no better than always answering "fail" (75% of the
+  labels are fail). eval-builder 0.1.2 still called them trustworthy and wired
+  `qwen2.5-7b` into promptfoo and DeepEval; 0.1.3 requires a judge to beat that
+  baseline, so both are now `misaligned` and no judge is wired in. The kappa intervals
+  are wide, from about 0.1 to 0.8: 24 labels cannot pin the agreement down. Every miss went the same
   way: the judge passed a reply the labels failed. Both qwen judges passed case-018
   (llama-13b answered "Here is an allegorical poem that illustrates the above:" and
   no poem) and case-019 (alpaca-13b's rewrite in which most sentences do not start with
@@ -65,15 +65,17 @@ brackets are 95% intervals.
   the judges' consensus: the judges said pass far more often than the labels did.
 - llama3.2:3b changes its verdict on almost half the cases between identical calls.
 
-## DeepEval with the checked judge
+## DeepEval with the checked judge (run under 0.1.2)
 
-`export` wrote `evalset/exports/deepeval/judge.json` (judge `qwen2.5-7b`, its exact
+Under 0.1.2, `export` wrote `evalset/exports/deepeval/judge.json` (removed by the 0.1.3 re-export) (judge `qwen2.5-7b`, its exact
 prompt, provider `ollama:chat:qwen2.5:7b-instruct` at temperature 0.8). DeepEval 4.2.8
 ran three of the cases on the logged outputs
 (`EVAL_BUILDER_USE_OBSERVED=1 pytest test_eval_builder.py -k "case-018 or case-020 or case-040"`):
 case-020 passed, case-040 failed with the judge's reason
 `{"pass": false, "reason": "Incorrect circumradius calculation."}`, and case-018 (the
-missing poem) passed, the same mistake judge-check found.
+missing poem) passed, the same mistake judge-check found. With 0.1.3 the same export
+falls back to GEval unless you pass `export --judge qwen2.5-7b`, which warns that the
+judge did not pass.
 
 `evalset/judge_requests.jsonl` (4.7 MB of rendered prompts) is not committed; `judge-plan`
 rebuilds it from `cases.yaml` and `rubric.yaml`.

eval_builder-0.1.4/examples/sample-labeling/evalset/exports/manifest.json

@@ -0,0 +1,60 @@
+{
+  "cases": 47,
+  "formats": [
+    "promptfoo",
+    "deepeval",
+    "inspect",
+    "jsonl"
+  ],
+  "files": [
+    {
+      "path": "exports/promptfoo/promptfooconfig.yaml",
+      "sha256": "07a874ff9a93431609865407d7b5759e39fbe50d9eb5a993f6ba8e3a8b852cbd"
+    },
+    {
+      "path": "exports/deepeval/dataset.json",
+      "sha256": "f6cf982eeef58e135e8b20bd9bd2a39f6965e1c2bc8e8ceb39e38f07b8c17536"
+    },
+    {
+      "path": "exports/deepeval/test_eval_builder.py",
+      "sha256": "05f445309c2983afd8a9cfb675dc7fd2a97f32f35ea2d918442a7bb391c270f7"
+    },
+    {
+      "path": "exports/inspect/dataset.jsonl",
+      "sha256": "63fb2547e1e810076647ebbd7c8a23cadd290ed8ad28fd1fc886b5b17b2b9215"
+    },
+    {
+      "path": "exports/inspect/task.py",
+      "sha256": "df284b6b6dedeb833e283cd9226241c17a84bed70924d097e9a6c45cba69c36e"
+    },
+    {
+      "path": "exports/jsonl/cases.jsonl",
+      "sha256": "d10eff47dcc67a113871b634bdbf8a705ce71ab6d485f0b109a06934d9c2bdef"
+    }
+  ],
+  "judge_check": {
+    "trustworthy": [],
+    "verdicts": {
+      "llama3.2-3b": "unstable",
+      "qwen2.5-7b": "misaligned",
+      "qwen2.5-7b-temp0": "misaligned"
+    },
+    "thresholds": {
+      "min_trials": 3,
+      "min_cases": 10,
+      "max_flip_rate": 0.2,
+      "min_position_consistency": 0.8,
+      "max_toward_padded_rate": 0.1,
+      "min_kappa": 0.4,
+      "min_labeled": 20,
+      "beat_majority_baseline": true
+    }
+  },
+  "trusted_judge_prompts": {},
+  "promptfoo_grader": null,
+  "deepeval_grader": null,
+  "notes": [
+    "DeepEval: no judge passed judge-check, so test_eval_builder.py grades with GEval and the judge model configured in DeepEval (not a judge checked here)",
+    "Inspect AI: task.py grades with Inspect's default model_graded_qa scorer, not a judge checked here"
+  ]
+}

{eval_builder-0.1.2 → eval_builder-0.1.4}/examples/sample-labeling/evalset/exports/promptfoo/promptfooconfig.yaml

@@ -4,39 +4,14 @@
 # The prompt is the whole conversation as a JSON list of chat messages (vars.messages), which
 # chat providers such as openai:chat or ollama:chat accept as is. If your provider takes plain
 # text, use {{input}} (the latest user message) and {{context}} (the earlier turns) instead.
-# Grader: judge qwen2.5-7b from rubric.yaml (judge-check verdict: trustworthy),
-# with its exact prompt and model. promptfoo's llm-rubric needs the grader to answer in JSON,
-# {"pass": true|false, "reason": "..."}; a judge that answers with a bare word fails with
-# "Could not extract JSON".
+# No grader is wired in: llm-rubric uses promptfoo's default grader, which
+# sees only the output and the rubric. To grade with a judge you checked, give it a `provider`
+# in rubric.yaml (a promptfoo provider id) and export again.
 description: MT-Bench sample, labeling example
 prompts:
 - '{{ messages | dump }}'
 providers:
 - echo
-defaultTest:
-  options:
-    provider: ollama:chat:qwen2.5:7b-instruct
-    rubricPrompt: |-
-      {% raw %}You are grading one reply from an AI assistant.
-
-      Earlier conversation:
-      {% endraw %}{{ context }}{% raw %}
-
-      Latest user message:
-      {% endraw %}{{ input }}{% raw %}
-
-      Assistant reply to grade:
-      {% endraw %}{{ output }}{% raw %}
-
-      What a good reply does:
-      {% endraw %}{{ expected_behavior }}{% raw %}
-
-      Criteria:
-      {% endraw %}{{ criteria }}{% raw %}
-
-      Does the reply do what a good reply does and meet the criteria? Answer with JSON only:
-      {"pass": true or false, "reason": "<one short sentence>"}
-      {% endraw %}
 tests:
 - description: 'case-001: How do you like dancing? Can you teach me?'
   vars:

{eval_builder-0.1.2 → eval_builder-0.1.4}/examples/sample-labeling/evalset/judge_check.json

@@ -8,9 +8,9 @@
     "pass": 6
   },
   "warnings": [
-    "judge llama3.2-3b's accuracy (54%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%)); kappa 0.12 [-0.26, 0.50] is the number that corrects for this. Look at the cases it got wrong before relying on it",
-    "judge qwen2.5-7b's accuracy (71%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%)); kappa 0.44 [0.09, 0.79] is the number that corrects for this. Look at the cases it got wrong before relying on it",
-    "judge qwen2.5-7b-temp0's accuracy (75%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said fail 12 of 24 (50%), pass 12 (50%)); kappa 0.50 [0.15, 0.85] is the number that corrects for this. Look at the cases it got wrong before relying on it"
+    "judge llama3.2-3b's accuracy (54%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%); kappa 0.12 [-0.26, 0.50]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong",
+    "judge qwen2.5-7b's accuracy (71%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%); kappa 0.44 [0.09, 0.79]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong",
+    "judge qwen2.5-7b-temp0's accuracy (75%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said fail 12 of 24 (50%), pass 12 (50%); kappa 0.50 [0.15, 0.85]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong"
   ],
   "thresholds": {
     "min_trials": 3,
@@ -19,7 +19,8 @@
     "min_position_consistency": 0.8,
     "max_toward_padded_rate": 0.1,
     "min_kappa": 0.4,
-    "min_labeled": 20
+    "min_labeled": 20,
+    "beat_majority_baseline": true
   },
   "summary": [
     {
@@ -61,7 +62,7 @@
     {
       "judge": "qwen2.5-7b",
       "mode": "pointwise",
-      "verdict": "trustworthy",
+      "verdict": "misaligned",
       "cases": 47,
       "calls": 376,
       "invalid_outputs": 0,
@@ -90,13 +91,13 @@
         0.0756
       ],
       "reasons": [
-        "stable (flip rate 9%), kappa 0.44 with humans"
+        "accuracy 71% is no better than always answering 'fail' (75% on these labels), so kappa 0.44 alone doesn't show the judge adds anything"
       ]
     },
     {
       "judge": "qwen2.5-7b-temp0",
       "mode": "pointwise",
-      "verdict": "trustworthy",
+      "verdict": "misaligned",
       "cases": 47,
       "calls": 376,
       "invalid_outputs": 0,
@@ -125,7 +126,7 @@
         0.0756
       ],
       "reasons": [
-        "stable (flip rate 0%), kappa 0.50 with humans"
+        "accuracy 75% is no better than always answering 'fail' (75% on these labels), so kappa 0.50 alone doesn't show the judge adds anything"
       ]
     }
   ],
@@ -829,15 +830,15 @@
     },
     "qwen2.5-7b": {
       "mode": "pointwise",
-      "verdict": "trustworthy",
+      "verdict": "misaligned",
       "reasons": [
-        "stable (flip rate 9%), kappa 0.44 with humans"
+        "accuracy 71% is no better than always answering 'fail' (75% on these labels), so kappa 0.44 alone doesn't show the judge adds anything"
       ],
       "checks": {
         "stability": "pass",
         "position": "not run",
         "verbosity": "pass",
-        "human_agreement": "pass"
+        "human_agreement": "fail"
       },
       "stability": {
         "cases": 47,
@@ -1507,15 +1508,15 @@
     },
     "qwen2.5-7b-temp0": {
       "mode": "pointwise",
-      "verdict": "trustworthy",
+      "verdict": "misaligned",
       "reasons": [
-        "stable (flip rate 0%), kappa 0.50 with humans"
+        "accuracy 75% is no better than always answering 'fail' (75% on these labels), so kappa 0.50 alone doesn't show the judge adds anything"
       ],
       "checks": {
         "stability": "pass",
         "position": "not run",
         "verbosity": "pass",
-        "human_agreement": "pass"
+        "human_agreement": "fail"
       },
       "stability": {
         "cases": 47,
@@ -2180,9 +2181,6 @@
       ]
     }
   },
-  "trustworthy": [
-    "qwen2.5-7b",
-    "qwen2.5-7b-temp0"
-  ],
-  "next": "export: judges ['qwen2.5-7b', 'qwen2.5-7b-temp0'] passed; export wires a passing pointwise judge into promptfoo when its rubric.yaml entry has a `provider`"
+  "trustworthy": [],
+  "next": "no judge passed; read each judge's reasons. Typical fixes: majority vote over 3 calls, run both answer orders and keep agreements, tighten the rubric wording, or try a stronger judge model, then run judge_plan and judge_check again"
 }

{eval_builder-0.1.2 → eval_builder-0.1.4}/examples/sample-labeling/evalset/report.json

@@ -1,5 +1,5 @@
 {
-  "generated_at": "2026-10-09T08:21:43+00:00",
+  "generated_at": "2026-10-09T08:26:01+00:00",
   "eval_builder_version": "0.1.2",
   "workspace": "examples/sample-labeling/evalset",
   "ingest": {
@@ -1280,9 +1280,9 @@
       "pass": 6
     },
     "warnings": [
-      "judge llama3.2-3b's accuracy (54%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%)); kappa 0.12 [-0.26, 0.50] is the number that corrects for this. Look at the cases it got wrong before relying on it",
-      "judge qwen2.5-7b's accuracy (71%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%)); kappa 0.44 [0.09, 0.79] is the number that corrects for this. Look at the cases it got wrong before relying on it",
-      "judge qwen2.5-7b-temp0's accuracy (75%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said fail 12 of 24 (50%), pass 12 (50%)); kappa 0.50 [0.15, 0.85] is the number that corrects for this. Look at the cases it got wrong before relying on it"
+      "judge llama3.2-3b's accuracy (54%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%); kappa 0.12 [-0.26, 0.50]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong",
+      "judge qwen2.5-7b's accuracy (71%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%); kappa 0.44 [0.09, 0.79]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong",
+      "judge qwen2.5-7b-temp0's accuracy (75%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said fail 12 of 24 (50%), pass 12 (50%); kappa 0.50 [0.15, 0.85]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong"
     ],
     "thresholds": {
       "min_trials": 3,
@@ -1291,7 +1291,8 @@
       "min_position_consistency": 0.8,
       "max_toward_padded_rate": 0.1,
       "min_kappa": 0.4,
-      "min_labeled": 20
+      "min_labeled": 20,
+      "beat_majority_baseline": true
     },
     "summary": [
       {
@@ -1333,7 +1334,7 @@
       {
         "judge": "qwen2.5-7b",
         "mode": "pointwise",
-        "verdict": "trustworthy",
+        "verdict": "misaligned",
         "cases": 47,
         "calls": 376,
         "invalid_outputs": 0,
@@ -1362,13 +1363,13 @@
           0.0756
         ],
         "reasons": [
-          "stable (flip rate 9%), kappa 0.44 with humans"
+          "accuracy 71% is no better than always answering 'fail' (75% on these labels), so kappa 0.44 alone doesn't show the judge adds anything"
         ]
       },
       {
         "judge": "qwen2.5-7b-temp0",
         "mode": "pointwise",
-        "verdict": "trustworthy",
+        "verdict": "misaligned",
         "cases": 47,
         "calls": 376,
         "invalid_outputs": 0,
@@ -1397,7 +1398,7 @@
           0.0756
         ],
         "reasons": [
-          "stable (flip rate 0%), kappa 0.50 with humans"
+          "accuracy 75% is no better than always answering 'fail' (75% on these labels), so kappa 0.50 alone doesn't show the judge adds anything"
         ]
       }
     ],
@@ -2101,15 +2102,15 @@
       },
       "qwen2.5-7b": {
         "mode": "pointwise",
-        "verdict": "trustworthy",
+        "verdict": "misaligned",
         "reasons": [
-          "stable (flip rate 9%), kappa 0.44 with humans"
+          "accuracy 71% is no better than always answering 'fail' (75% on these labels), so kappa 0.44 alone doesn't show the judge adds anything"
         ],
         "checks": {
           "stability": "pass",
           "position": "not run",
           "verbosity": "pass",
-          "human_agreement": "pass"
+          "human_agreement": "fail"
         },
         "stability": {
           "cases": 47,
@@ -2779,15 +2780,15 @@
       },
       "qwen2.5-7b-temp0": {
         "mode": "pointwise",
-        "verdict": "trustworthy",
+        "verdict": "misaligned",
         "reasons": [
-          "stable (flip rate 0%), kappa 0.50 with humans"
+          "accuracy 75% is no better than always answering 'fail' (75% on these labels), so kappa 0.50 alone doesn't show the judge adds anything"
         ],
         "checks": {
           "stability": "pass",
           "position": "not run",
           "verbosity": "pass",
-          "human_agreement": "pass"
+          "human_agreement": "fail"
         },
         "stability": {
           "cases": 47,
@@ -3452,11 +3453,8 @@
         ]
       }
     },
-    "trustworthy": [
-      "qwen2.5-7b",
-      "qwen2.5-7b-temp0"
-    ],
-    "next": "export: judges ['qwen2.5-7b', 'qwen2.5-7b-temp0'] passed; export wires a passing pointwise judge into promptfoo when its rubric.yaml entry has a `provider`"
+    "trustworthy": [],
+    "next": "no judge passed; read each judge's reasons. Typical fixes: majority vote over 3 calls, run both answer orders and keep agreements, tighten the rubric wording, or try a stronger judge model, then run judge_plan and judge_check again"
   },
   "exports": {
     "cases": 47,
@@ -3469,7 +3467,7 @@
     "files": [
       {
         "path": "exports/promptfoo/promptfooconfig.yaml",
-        "sha256": "23c43efb656ee17cf190fffd1100eb18986917593f6854dcdf36973ae4a6b6f3"
+        "sha256": "07a874ff9a93431609865407d7b5759e39fbe50d9eb5a993f6ba8e3a8b852cbd"
       },
       {
         "path": "exports/deepeval/dataset.json",
@@ -3479,10 +3477,6 @@
         "path": "exports/deepeval/test_eval_builder.py",
         "sha256": "05f445309c2983afd8a9cfb675dc7fd2a97f32f35ea2d918442a7bb391c270f7"
       },
-      {
-        "path": "exports/deepeval/judge.json",
-        "sha256": "dd632220790c10ab8c788b1407cc08c0db17a28b58d4911b373b32b32bcf2fdd"
-      },
       {
         "path": "exports/inspect/dataset.jsonl",
         "sha256": "63fb2547e1e810076647ebbd7c8a23cadd290ed8ad28fd1fc886b5b17b2b9215"
@@ -3497,14 +3491,11 @@
       }
     ],
     "judge_check": {
-      "trustworthy": [
-        "qwen2.5-7b",
-        "qwen2.5-7b-temp0"
-      ],
+      "trustworthy": [],
       "verdicts": {
         "llama3.2-3b": "unstable",
-        "qwen2.5-7b": "trustworthy",
-        "qwen2.5-7b-temp0": "trustworthy"
+        "qwen2.5-7b": "misaligned",
+        "qwen2.5-7b-temp0": "misaligned"
       },
       "thresholds": {
         "min_trials": 3,
@@ -3513,38 +3504,15 @@
         "min_position_consistency": 0.8,
         "max_toward_padded_rate": 0.1,
         "min_kappa": 0.4,
-        "min_labeled": 20
-      }
-    },
-    "trusted_judge_prompts": {
-      "qwen2.5-7b": {
-        "mode": "pointwise",
-        "labels": [
-          "pass",
-          "fail"
-        ],
-        "prompt": "You are grading one reply from an AI assistant.\n\nEarlier conversation:\n{context}\n\nLatest user message:\n{input}\n\nAssistant reply to grade:\n{output}\n\nWhat a good reply does:\n{expected_behavior}\n\nCriteria:\n{criteria}\n\nDoes the reply do what a good reply does and meet the criteria? Answer with JSON only:\n{\"pass\": true or false, \"reason\": \"<one short sentence>\"}\n"
-      },
-      "qwen2.5-7b-temp0": {
-        "mode": "pointwise",
-        "labels": [
-          "pass",
-          "fail"
-        ],
-        "prompt": "You are grading one reply from an AI assistant.\n\nEarlier conversation:\n{context}\n\nLatest user message:\n{input}\n\nAssistant reply to grade:\n{output}\n\nWhat a good reply does:\n{expected_behavior}\n\nCriteria:\n{criteria}\n\nDoes the reply do what a good reply does and meet the criteria? Answer with JSON only:\n{\"pass\": true or false, \"reason\": \"<one short sentence>\"}\n"
+        "min_labeled": 20,
+        "beat_majority_baseline": true
       }
     },
-    "promptfoo_grader": {
-      "judge": "qwen2.5-7b",
-      "provider": "ollama:chat:qwen2.5:7b-instruct",
-      "verdict": "trustworthy"
-    },
-    "deepeval_grader": {
-      "judge": "qwen2.5-7b",
-      "provider": "ollama:chat:qwen2.5:7b-instruct",
-      "verdict": "trustworthy"
-    },
+    "trusted_judge_prompts": {},
+    "promptfoo_grader": null,
+    "deepeval_grader": null,
     "notes": [
+      "DeepEval: no judge passed judge-check, so test_eval_builder.py grades with GEval and the judge model configured in DeepEval (not a judge checked here)",
       "Inspect AI: task.py grades with Inspect's default model_graded_qa scorer, not a judge checked here"
     ]
   },

{eval_builder-0.1.2 → eval_builder-0.1.4}/examples/sample-labeling/evalset/report.md

@@ -1,6 +1,6 @@
 # Labeling example on the 48-conversation sample
 
-Generated 2026-10-09T08:21:43+00:00 by eval-builder 0.1.2.
+Generated 2026-10-09T08:26:01+00:00 by eval-builder 0.1.2.
 
 ## Sources
 
@@ -106,18 +106,18 @@ Thresholds: flip rate <= 20% (cases with >= 3 trials, >= 10 cases), position con
 | judge | mode | verdict | flip rate | self-agreement | majority-of-3 stable | accuracy vs humans | kappa | position consistency | first-shown picked | padding helped |
 |---|---|---|---|---|---|---|---|---|---|---|
 | llama3.2-3b | pointwise | **unstable** | 47% [33%, 61%] (n=47) | 83% | 88% | 54% [35%, 72%] (n=24) | 0.12 [-0.26, 0.50] | n/a | n/a | 4% [1%, 14%] (n=47) |
-| qwen2.5-7b | pointwise | **trustworthy** | 9% [3%, 20%] (n=47) | 97% | 97% | 71% [51%, 85%] (n=24) | 0.44 [0.09, 0.79] | n/a | n/a | 0% [0%, 8%] (n=47) |
-| qwen2.5-7b-temp0 | pointwise | **trustworthy** | 0% [0%, 8%] (n=47) | 100% | 100% | 75% [55%, 88%] (n=24) | 0.50 [0.15, 0.85] | n/a | n/a | 0% [0%, 8%] (n=47) |
+| qwen2.5-7b | pointwise | **misaligned** | 9% [3%, 20%] (n=47) | 97% | 97% | 71% [51%, 85%] (n=24) | 0.44 [0.09, 0.79] | n/a | n/a | 0% [0%, 8%] (n=47) |
+| qwen2.5-7b-temp0 | pointwise | **misaligned** | 0% [0%, 8%] (n=47) | 100% | 100% | 75% [55%, 88%] (n=24) | 0.50 [0.15, 0.85] | n/a | n/a | 0% [0%, 8%] (n=47) |
 
 - **llama3.2-3b** (unstable): verdict changed across repeated trials on 47% of cases (limit 20%). agreement with human labels is low: kappa 0.12 (need 0.4), accuracy 54%.
-- **qwen2.5-7b** (trustworthy): stable (flip rate 9%), kappa 0.44 with humans.
-- **qwen2.5-7b-temp0** (trustworthy): stable (flip rate 0%), kappa 0.50 with humans.
+- **qwen2.5-7b** (misaligned): accuracy 71% is no better than always answering 'fail' (75% on these labels), so kappa 0.44 alone doesn't show the judge adds anything.
+- **qwen2.5-7b-temp0** (misaligned): accuracy 75% is no better than always answering 'fail' (75% on these labels), so kappa 0.50 alone doesn't show the judge adds anything.
 
 Human labels on judged cases: fail 18 of 24 (75%), pass 6 (25%). A judge that always gave the most common label would score 75% accuracy, which is the bar accuracy has to clear; kappa already corrects for it.
 
-- Warning: judge llama3.2-3b's accuracy (54%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%)); kappa 0.12 [-0.26, 0.50] is the number that corrects for this. Look at the cases it got wrong before relying on it
-- Warning: judge qwen2.5-7b's accuracy (71%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%)); kappa 0.44 [0.09, 0.79] is the number that corrects for this. Look at the cases it got wrong before relying on it
-- Warning: judge qwen2.5-7b-temp0's accuracy (75%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said fail 12 of 24 (50%), pass 12 (50%)); kappa 0.50 [0.15, 0.85] is the number that corrects for this. Look at the cases it got wrong before relying on it
+- Warning: judge llama3.2-3b's accuracy (54%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%); kappa 0.12 [-0.26, 0.50]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong
+- Warning: judge qwen2.5-7b's accuracy (71%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said pass 13 of 24 (54%), fail 11 (46%); kappa 0.44 [0.09, 0.79]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong
+- Warning: judge qwen2.5-7b-temp0's accuracy (75%) is no better than always answering 'fail' (75%) on these 24 labeled cases (the judge said fail 12 of 24 (50%), pass 12 (50%); kappa 0.50 [0.15, 0.85]). A judge has to beat that baseline to be called trustworthy; look at the cases it got wrong
 
 Judge calls made through the opt-in judge plugin (`judge-run`):
 
@@ -131,10 +131,9 @@ Judge calls made through the opt-in judge plugin (`judge-run`):
 
 | file | sha256 |
 |---|---|
-| exports/promptfoo/promptfooconfig.yaml | `23c43efb656ee17c` |
+| exports/promptfoo/promptfooconfig.yaml | `07a874ff9a934316` |
 | exports/deepeval/dataset.json | `f6cf982eeef58e13` |
 | exports/deepeval/test_eval_builder.py | `05f445309c2983af` |
-| exports/deepeval/judge.json | `dd632220790c10ab` |
 | exports/inspect/dataset.jsonl | `63fb2547e1e81007` |
 | exports/inspect/task.py | `df284b6b6dedeb83` |
 | exports/jsonl/cases.jsonl | `d10eff47dcc67a11` |