hemp-eval 0.1.0__tar.gz

hemp_eval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nithilan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hemp_eval-0.1.0/PKG-INFO

@@ -0,0 +1,233 @@
+Metadata-Version: 2.4
+Name: hemp-eval
+Version: 0.1.0
+Summary: Help Evaluate My Prompt: work out how to test an LLM prompt from the prompt and your data, run it, and compare prompt versions.
+Keywords: llm,evaluation,prompt,eval,dspy,llm-as-a-judge,prompt-engineering
+Author: NithilanVishvanath
+Author-email: NithilanVishvanath <nithilanvishvanath@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: dspy>=3.4.1
+Requires-Dist: litellm>=1.106.6
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: rich>=13.7
+Requires-Dist: typer>=0.12
+Requires-Dist: pillow>=10 ; extra == 'images'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/NithilanVishvanath/hemp
+Project-URL: Repository, https://github.com/NithilanVishvanath/hemp
+Project-URL: Issues, https://github.com/NithilanVishvanath/hemp/issues
+Provides-Extra: images
+Description-Content-Type: text/markdown
+
+# HEMP: Help Evaluate My Prompt
+
+You have a prompt and some example data. HEMP works out **how that prompt should be tested**, runs the test, and tells you whether a new version of the prompt is better or worse, down to the examples that got fixed or broke.
+
+```
+hemp plan  prompt.txt data.jsonl   # decide how to test the prompt (you review the plan)
+hemp run   prompt.txt data.jsonl   # run it, score it, compare with simple baselines
+hemp diff                          # after editing the prompt: what got better, what broke
+```
+
+- **It reads your data, not just your prompt.** An answer column with labels gets an exact-match check, JSON answers get field-by-field checks, free text gets meaning-based checks.
+- **Yes/no questions for an LLM judge** cover what code can't check ("Is the answer polite?"). No 1–10 scores.
+- **You can check the judge.** Label a few outputs yourself and HEMP reports how often the judge agrees with you, and tunes it to agree more.
+- **Baselines give scores context.** "Always answer the most common label" and "no instructions at all" run alongside your prompt.
+- **Local and private by default.** It runs on your machine with Ollama, or with any model LiteLLM supports (OpenAI, Gemini, Anthropic, ...). There's no account and no server.
+- **Text and images.** A column of image paths is sent to the model as real images.
+
+## Install
+
+```
+pip install hemp-eval
+pip install "hemp-eval[images]"   # also scale down large images before sending them
+```
+
+The command is `hemp` and the Python package is `hemp`. Python 3.11+ is required.
+
+You also need a model. Either:
+
+- **Local (default):** install [Ollama](https://ollama.com), then `ollama pull qwen2.5:7b` and `ollama pull nomic-embed-text`. Keep Ollama running.
+- **Hosted:** set `HEMP_MODEL` to a [LiteLLM model name](https://docs.litellm.ai/docs/providers) and its API key, e.g. `HEMP_MODEL=openai/gpt-4o-mini` with `OPENAI_API_KEY`, or `HEMP_MODEL=gemini/gemini-2.0-flash` with `GEMINI_API_KEY`.
+
+Then check your setup with `hemp doctor`. It confirms each model answers, and says exactly what to fix if one doesn't. Use `--offline` to check without making any calls.
+
+## Quick start
+
+**1. Write your prompt** with `{column}` where each row's data goes, in `prompt.txt`:
+
+```
+Classify the sentiment of this product review as exactly one of: positive, negative, neutral.
+Reply with the label only.
+
+Review: {review}
+```
+
+**2. Collect examples** in `data.jsonl` (CSV and JSON work too), ideally with the right answer. 20–100 rows is a good start.
+
+```
+{"review": "Great product, arrived early.", "label": "positive"}
+{"review": "Broke after two days.", "label": "negative"}
+```
+
+**3. Plan.** Run `hemp plan prompt.txt data.jsonl`. HEMP profiles the data, asks the planner model how to test the prompt, and writes `prompt.plan.yaml`:
+
+```yaml
+columns:
+  input: [review]
+  answer: label
+
+checks:  # automatic, no LLM needed
+  - check: allowed_values
+    values: [negative, positive, neutral]
+  - check: exact_match
+    compare_to: label
+
+questions:  # an LLM judge answers each; "yes" = pass
+  - id: q1
+    ask: Does the label fit the review's overall tone?
+
+baselines:  # simple strategies your prompt should beat
+  - majority_class
+  - zero_shot
+```
+
+Read it, and delete or add checks and questions as you like. `hemp validate prompt.plan.yaml` checks your edits, and `hemp metrics` lists every available check.
+
+**4. Run.** `hemp run prompt.txt data.jsonl` runs your prompt on every row exactly as written and scores it:
+
+```
+| Check             | Your prompt | majority_class (baseline) | zero_shot (baseline) |
+| allowed_values    | 100% (8/8)  | 100% (8/8)                |   0% (0/8)           |
+| exact_match_label | 100% (8/8)  |  38% (3/8)                |   0% (0/8)           |
+```
+
+Then it lists the examples your prompt got wrong. Results are kept in `.hemp/hemp.db`; `hemp show` lists past runs.
+
+**5. Improve and compare.** Edit the prompt, run again, then `hemp diff`:
+
+```
+Changed: prompt   Same: plan, dataset, target model, judge model
+| Check             | Run #1 | Run #2 | Change  | Fixed | Broke |
+| exact_match_label | 75%    | 100%   | +25 pts |   3   |   0   |
+Fixed (3): "It's ok"  before: positive  ->  now: neutral
+```
+
+Baselines are reused between runs when nothing they depend on changed, so each round only pays for your prompt.
+
+## Images
+
+A column of image paths (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.bmp`), image URLs, or `data:image/...` URIs is an image column. Use it in the prompt like any other column:
+
+```
+Classify this photo as cat, dog or bird. Reply with the label only.
+{image}
+```
+
+```
+image,label
+photos/cat_01.jpg,cat
+photos/dog_01.jpg,dog
+```
+
+- **What the model gets:** the image itself, at the spot where `{image}` is, not the path.
+- **Paths:** relative paths are looked up next to the data file.
+- **Size:** with `hemp-eval[images]`, images bigger than 1536 px are scaled down first. Set the limit with `HEMP_IMAGE_MAX_SIDE`; `0` turns it off.
+- **Baselines and judge:** the zero_shot baseline sends just the image, and the judge sees the images too.
+- **Model:** use one that can see images, such as a Gemini or GPT-4o model, or a local `ollama_chat/gemma3:4b`.
+
+## Check and tune the judge
+
+Judge scores are only useful if the judge answers the way you would. After a run with questions:
+
+```
+hemp calibrate prompt.plan.yaml
+```
+
+HEMP shows outputs one at a time, with the judge's answer hidden. You answer each question yourself: `y`, `n`, `s` to skip, `q` to stop. It then reports how often the judge agrees with you, as accuracy and Cohen's kappa (which corrects for lucky agreement), per question.
+
+With 8 or more labels, HEMP also tunes the judge with [DSPy](https://dspy.ai): it picks worked examples from half your labels and tests on the other half. If the tuned judge agrees with you more, it's saved next to the plan and `hemp run` uses it from then on.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `hemp doctor` | Check that your models are reachable, with a fix for each problem. |
+| `hemp profile data.jsonl` | Column types and roles (input, answer, id) and value stats. No model calls. |
+| `hemp plan prompt.txt data.jsonl` | Write an editable evaluation plan. `--no-llm` uses rules only. |
+| `hemp validate plan.yaml` | Check an edited plan. |
+| `hemp metrics` | List the checks a plan can use and their settings. |
+| `hemp run prompt.txt data.jsonl` | Run and score. Useful options: `--target-model`, `--judge-model`, `--no-judge`, `--no-baselines`. |
+| `hemp show [run]` | List runs, or show one with its failed examples. |
+| `hemp diff [A] [B]` | Compare two runs (default: the latest two). `--fail-on-regression` for CI. |
+| `hemp calibrate plan.yaml` | Label outputs, measure and tune the judge. |
+
+## Models
+
+| Used for | Default | Change with |
+|---|---|---|
+| Planning, running your prompt, judging | `ollama_chat/qwen2.5:7b` | `HEMP_MODEL` for all three, or `HEMP_PLANNER_MODEL`, `HEMP_TARGET_MODEL`, `HEMP_JUDGE_MODEL` |
+| Comparing free-text answers by meaning | `ollama/nomic-embed-text` | `HEMP_EMBED_MODEL` |
+
+Any [LiteLLM model name](https://docs.litellm.ai/docs/providers) works. A judge at least as strong as the model being tested gives better scores.
+
+## Your own checks
+
+Write a check in a Python file:
+
+```python
+# my_metrics.py
+import hemp
+
+
+@hemp.metric("polite")
+def polite(ctx: hemp.MetricContext, params: hemp.MetricParams) -> hemp.MetricResult:
+    """The output says please or thanks."""
+    text = ctx.output.lower()
+    return hemp.MetricResult.check("please" in text or "thank" in text)
+```
+
+Pass it with `--plugin my_metrics.py` to `hemp plan` or `hemp metrics`. The planner can choose it, and the plan remembers the file.
+
+## Examples
+
+The [examples folder](https://github.com/NithilanVishvanath/hemp/tree/main/examples) has ready-to-run projects:
+
+| Example | What it shows |
+|---|---|
+| `sentiment` | Labels: exact match and baselines |
+| `extraction` | JSON output checked field by field |
+| `qa` | Free-text answers, a judge question, and a custom check |
+| `colors` | Images: name the color of each swatch (needs a vision model) |
+
+For example: `hemp plan examples/sentiment/prompt.txt examples/sentiment/data.jsonl`.
+
+## Development
+
+```
+git clone https://github.com/NithilanVishvanath/hemp
+cd hemp
+uv sync
+uv run pytest                   # unit tests, no models needed
+uv run pytest -m integration    # tests that call a real model
+uv run ruff check . && uv run ruff format .
+uv run mypy
+```
+
+## License
+
+MIT

hemp_eval-0.1.0/README.md

@@ -0,0 +1,198 @@
+# HEMP: Help Evaluate My Prompt
+
+You have a prompt and some example data. HEMP works out **how that prompt should be tested**, runs the test, and tells you whether a new version of the prompt is better or worse, down to the examples that got fixed or broke.
+
+```
+hemp plan  prompt.txt data.jsonl   # decide how to test the prompt (you review the plan)
+hemp run   prompt.txt data.jsonl   # run it, score it, compare with simple baselines
+hemp diff                          # after editing the prompt: what got better, what broke
+```
+
+- **It reads your data, not just your prompt.** An answer column with labels gets an exact-match check, JSON answers get field-by-field checks, free text gets meaning-based checks.
+- **Yes/no questions for an LLM judge** cover what code can't check ("Is the answer polite?"). No 1–10 scores.
+- **You can check the judge.** Label a few outputs yourself and HEMP reports how often the judge agrees with you, and tunes it to agree more.
+- **Baselines give scores context.** "Always answer the most common label" and "no instructions at all" run alongside your prompt.
+- **Local and private by default.** It runs on your machine with Ollama, or with any model LiteLLM supports (OpenAI, Gemini, Anthropic, ...). There's no account and no server.
+- **Text and images.** A column of image paths is sent to the model as real images.
+
+## Install
+
+```
+pip install hemp-eval
+pip install "hemp-eval[images]"   # also scale down large images before sending them
+```
+
+The command is `hemp` and the Python package is `hemp`. Python 3.11+ is required.
+
+You also need a model. Either:
+
+- **Local (default):** install [Ollama](https://ollama.com), then `ollama pull qwen2.5:7b` and `ollama pull nomic-embed-text`. Keep Ollama running.
+- **Hosted:** set `HEMP_MODEL` to a [LiteLLM model name](https://docs.litellm.ai/docs/providers) and its API key, e.g. `HEMP_MODEL=openai/gpt-4o-mini` with `OPENAI_API_KEY`, or `HEMP_MODEL=gemini/gemini-2.0-flash` with `GEMINI_API_KEY`.
+
+Then check your setup with `hemp doctor`. It confirms each model answers, and says exactly what to fix if one doesn't. Use `--offline` to check without making any calls.
+
+## Quick start
+
+**1. Write your prompt** with `{column}` where each row's data goes, in `prompt.txt`:
+
+```
+Classify the sentiment of this product review as exactly one of: positive, negative, neutral.
+Reply with the label only.
+
+Review: {review}
+```
+
+**2. Collect examples** in `data.jsonl` (CSV and JSON work too), ideally with the right answer. 20–100 rows is a good start.
+
+```
+{"review": "Great product, arrived early.", "label": "positive"}
+{"review": "Broke after two days.", "label": "negative"}
+```
+
+**3. Plan.** Run `hemp plan prompt.txt data.jsonl`. HEMP profiles the data, asks the planner model how to test the prompt, and writes `prompt.plan.yaml`:
+
+```yaml
+columns:
+  input: [review]
+  answer: label
+
+checks:  # automatic, no LLM needed
+  - check: allowed_values
+    values: [negative, positive, neutral]
+  - check: exact_match
+    compare_to: label
+
+questions:  # an LLM judge answers each; "yes" = pass
+  - id: q1
+    ask: Does the label fit the review's overall tone?
+
+baselines:  # simple strategies your prompt should beat
+  - majority_class
+  - zero_shot
+```
+
+Read it, and delete or add checks and questions as you like. `hemp validate prompt.plan.yaml` checks your edits, and `hemp metrics` lists every available check.
+
+**4. Run.** `hemp run prompt.txt data.jsonl` runs your prompt on every row exactly as written and scores it:
+
+```
+| Check             | Your prompt | majority_class (baseline) | zero_shot (baseline) |
+| allowed_values    | 100% (8/8)  | 100% (8/8)                |   0% (0/8)           |
+| exact_match_label | 100% (8/8)  |  38% (3/8)                |   0% (0/8)           |
+```
+
+Then it lists the examples your prompt got wrong. Results are kept in `.hemp/hemp.db`; `hemp show` lists past runs.
+
+**5. Improve and compare.** Edit the prompt, run again, then `hemp diff`:
+
+```
+Changed: prompt   Same: plan, dataset, target model, judge model
+| Check             | Run #1 | Run #2 | Change  | Fixed | Broke |
+| exact_match_label | 75%    | 100%   | +25 pts |   3   |   0   |
+Fixed (3): "It's ok"  before: positive  ->  now: neutral
+```
+
+Baselines are reused between runs when nothing they depend on changed, so each round only pays for your prompt.
+
+## Images
+
+A column of image paths (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.bmp`), image URLs, or `data:image/...` URIs is an image column. Use it in the prompt like any other column:
+
+```
+Classify this photo as cat, dog or bird. Reply with the label only.
+{image}
+```
+
+```
+image,label
+photos/cat_01.jpg,cat
+photos/dog_01.jpg,dog
+```
+
+- **What the model gets:** the image itself, at the spot where `{image}` is, not the path.
+- **Paths:** relative paths are looked up next to the data file.
+- **Size:** with `hemp-eval[images]`, images bigger than 1536 px are scaled down first. Set the limit with `HEMP_IMAGE_MAX_SIDE`; `0` turns it off.
+- **Baselines and judge:** the zero_shot baseline sends just the image, and the judge sees the images too.
+- **Model:** use one that can see images, such as a Gemini or GPT-4o model, or a local `ollama_chat/gemma3:4b`.
+
+## Check and tune the judge
+
+Judge scores are only useful if the judge answers the way you would. After a run with questions:
+
+```
+hemp calibrate prompt.plan.yaml
+```
+
+HEMP shows outputs one at a time, with the judge's answer hidden. You answer each question yourself: `y`, `n`, `s` to skip, `q` to stop. It then reports how often the judge agrees with you, as accuracy and Cohen's kappa (which corrects for lucky agreement), per question.
+
+With 8 or more labels, HEMP also tunes the judge with [DSPy](https://dspy.ai): it picks worked examples from half your labels and tests on the other half. If the tuned judge agrees with you more, it's saved next to the plan and `hemp run` uses it from then on.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `hemp doctor` | Check that your models are reachable, with a fix for each problem. |
+| `hemp profile data.jsonl` | Column types and roles (input, answer, id) and value stats. No model calls. |
+| `hemp plan prompt.txt data.jsonl` | Write an editable evaluation plan. `--no-llm` uses rules only. |
+| `hemp validate plan.yaml` | Check an edited plan. |
+| `hemp metrics` | List the checks a plan can use and their settings. |
+| `hemp run prompt.txt data.jsonl` | Run and score. Useful options: `--target-model`, `--judge-model`, `--no-judge`, `--no-baselines`. |
+| `hemp show [run]` | List runs, or show one with its failed examples. |
+| `hemp diff [A] [B]` | Compare two runs (default: the latest two). `--fail-on-regression` for CI. |
+| `hemp calibrate plan.yaml` | Label outputs, measure and tune the judge. |
+
+## Models
+
+| Used for | Default | Change with |
+|---|---|---|
+| Planning, running your prompt, judging | `ollama_chat/qwen2.5:7b` | `HEMP_MODEL` for all three, or `HEMP_PLANNER_MODEL`, `HEMP_TARGET_MODEL`, `HEMP_JUDGE_MODEL` |
+| Comparing free-text answers by meaning | `ollama/nomic-embed-text` | `HEMP_EMBED_MODEL` |
+
+Any [LiteLLM model name](https://docs.litellm.ai/docs/providers) works. A judge at least as strong as the model being tested gives better scores.
+
+## Your own checks
+
+Write a check in a Python file:
+
+```python
+# my_metrics.py
+import hemp
+
+
+@hemp.metric("polite")
+def polite(ctx: hemp.MetricContext, params: hemp.MetricParams) -> hemp.MetricResult:
+    """The output says please or thanks."""
+    text = ctx.output.lower()
+    return hemp.MetricResult.check("please" in text or "thank" in text)
+```
+
+Pass it with `--plugin my_metrics.py` to `hemp plan` or `hemp metrics`. The planner can choose it, and the plan remembers the file.
+
+## Examples
+
+The [examples folder](https://github.com/NithilanVishvanath/hemp/tree/main/examples) has ready-to-run projects:
+
+| Example | What it shows |
+|---|---|
+| `sentiment` | Labels: exact match and baselines |
+| `extraction` | JSON output checked field by field |
+| `qa` | Free-text answers, a judge question, and a custom check |
+| `colors` | Images: name the color of each swatch (needs a vision model) |
+
+For example: `hemp plan examples/sentiment/prompt.txt examples/sentiment/data.jsonl`.
+
+## Development
+
+```
+git clone https://github.com/NithilanVishvanath/hemp
+cd hemp
+uv sync
+uv run pytest                   # unit tests, no models needed
+uv run pytest -m integration    # tests that call a real model
+uv run ruff check . && uv run ruff format .
+uv run mypy
+```
+
+## License
+
+MIT

hemp_eval-0.1.0/pyproject.toml

@@ -0,0 +1,93 @@
+[project]
+name = "hemp-eval"
+version = "0.1.0"
+description = "Help Evaluate My Prompt: work out how to test an LLM prompt from the prompt and your data, run it, and compare prompt versions."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "NithilanVishvanath", email = "nithilanvishvanath@gmail.com" }
+]
+requires-python = ">=3.11"
+keywords = ["llm", "evaluation", "prompt", "eval", "dspy", "llm-as-a-judge", "prompt-engineering"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Software Development :: Testing",
+    "Typing :: Typed",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+]
+dependencies = [
+    "dspy>=3.4.1",
+    "litellm>=1.106.6",
+    "pydantic>=2.6",
+    "pyyaml>=6.0.3",
+    "rich>=13.7",
+    "typer>=0.12",
+]
+
+[project.urls]
+Homepage = "https://github.com/NithilanVishvanath/hemp"
+Repository = "https://github.com/NithilanVishvanath/hemp"
+Issues = "https://github.com/NithilanVishvanath/hemp/issues"
+
+[project.scripts]
+hemp = "hemp.cli:app"
+
+[project.optional-dependencies]
+images = [
+    "pillow>=10",
+]
+
+[dependency-groups]
+dev = [
+    "mypy>=1.10",
+    "pillow>=10",
+    "pytest>=8.0",
+    "ruff>=0.6",
+    "types-pyyaml>=6.0.12.20260906",
+]
+
+[build-system]
+requires = ["uv_build>=0.11.14,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-name = "hemp"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = ["-ra", "--strict-markers", "-m", "not integration"]
+markers = [
+    "integration: tests that call a real model (run with `uv run pytest -m integration`)",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "SIM", "RUF"]
+
+[tool.mypy]
+python_version = "3.11"
+strict = true
+files = ["src"]
+plugins = ["pydantic.mypy"]
+
+[[tool.mypy.overrides]]
+# dspy and litellm ship without type information.
+module = ["dspy", "dspy.*", "litellm", "litellm.*"]
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+# DSPy signatures and modules are declared by subclassing dspy classes.
+module = ["hemp.plan.planner", "hemp.runner.target", "hemp.metrics.judge"]
+disallow_subclassing_any = false

hemp_eval-0.1.0/src/hemp/__init__.py

@@ -0,0 +1,32 @@
+"""HEMP: Help Evaluate My Prompt."""
+
+from hemp._version import __version__
+from hemp.dataset import DatasetError, load_dataset
+from hemp.metrics import MetricContext, MetricParams, MetricResult, metric
+from hemp.plan import EvalPlan, PlanError, create_plan, read_plan, write_plan
+from hemp.profile import DatasetProfile, profile_dataset, profile_rows
+from hemp.runner import RunResult, run_eval
+from hemp.store import RunDiff, Store, diff_runs
+
+__all__ = [
+    "DatasetError",
+    "DatasetProfile",
+    "EvalPlan",
+    "MetricContext",
+    "MetricParams",
+    "MetricResult",
+    "PlanError",
+    "RunDiff",
+    "RunResult",
+    "Store",
+    "__version__",
+    "create_plan",
+    "diff_runs",
+    "load_dataset",
+    "metric",
+    "profile_dataset",
+    "profile_rows",
+    "read_plan",
+    "run_eval",
+    "write_plan",
+]

hemp_eval-0.1.0/src/hemp/_version.py

@@ -0,0 +1,6 @@
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("hemp-eval")
+except PackageNotFoundError:  # pragma: no cover - running from a source tree without install
+    __version__ = "0.0.0"

hemp_eval-0.1.0/src/hemp/calibrate/__init__.py

@@ -0,0 +1,24 @@
+"""Human labeling, judge tuning, and judge-human agreement."""
+
+from hemp.calibrate.agreement import Agreement, agreement
+from hemp.calibrate.labels import Label, LabelItem, label_candidates
+from hemp.calibrate.optimize import (
+    MIN_LABELS_TO_TUNE,
+    CalibrationFailed,
+    CalibrationReport,
+    QuestionAgreement,
+    calibrate,
+)
+
+__all__ = [
+    "MIN_LABELS_TO_TUNE",
+    "Agreement",
+    "CalibrationFailed",
+    "CalibrationReport",
+    "Label",
+    "LabelItem",
+    "QuestionAgreement",
+    "agreement",
+    "calibrate",
+    "label_candidates",
+]

hemp_eval-0.1.0/src/hemp/calibrate/agreement.py

@@ -0,0 +1,33 @@
+"""How well the judge agrees with human labels: accuracy and Cohen's kappa.
+
+Accuracy alone flatters a judge on lopsided labels (if 90% of outputs are good, a
+judge that always says yes agrees 90% of the time). Kappa corrects for agreement
+expected by chance: 1 is perfect, 0 is no better than chance.
+"""
+
+from collections.abc import Sequence
+
+from pydantic import BaseModel, ConfigDict
+
+
+class Agreement(BaseModel):
+    model_config = ConfigDict(frozen=True, extra="forbid")
+
+    n: int
+    n_agree: int
+    accuracy: float | None
+    kappa: float | None  # None when chance agreement is total (one answer only)
+
+
+def agreement(pairs: Sequence[tuple[bool, bool]]) -> Agreement:
+    """Agreement over (human, judge) verdict pairs."""
+    n = len(pairs)
+    if n == 0:
+        return Agreement(n=0, n_agree=0, accuracy=None, kappa=None)
+    n_agree = sum(1 for human, judge in pairs if human == judge)
+    observed = n_agree / n
+    human_yes = sum(1 for human, _ in pairs if human) / n
+    judge_yes = sum(1 for _, judge in pairs if judge) / n
+    chance = human_yes * judge_yes + (1 - human_yes) * (1 - judge_yes)
+    kappa = None if chance >= 1 else (observed - chance) / (1 - chance)
+    return Agreement(n=n, n_agree=n_agree, accuracy=observed, kappa=kappa)