synthbench-eval 0.4.0__tar.gz

synthbench_eval-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DataViking Tech
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

synthbench_eval-0.4.0/PKG-INFO

@@ -0,0 +1,279 @@
+Metadata-Version: 2.4
+Name: synthbench-eval
+Version: 0.4.0
+Summary: Open benchmark harness for synthetic survey respondent quality
+License-Expression: MIT
+Keywords: benchmark,survey,synthetic-data,llm,evaluation,opinionsqa,globalopinionqa,gss,convergence,synthpanel
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0
+Requires-Dist: httpx>=0.24
+Requires-Dist: numpy>=1.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: scipy>=1.10
+Requires-Dist: synthpanel>=0.2.0
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.18; extra == "anthropic"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == "openai"
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.7; extra == "viz"
+Provides-Extra: hf
+Requires-Dist: datasets>=2.14; extra == "hf"
+Provides-Extra: subpop
+Requires-Dist: datasets>=2.14; extra == "subpop"
+Provides-Extra: r2
+Requires-Dist: boto3>=1.34; extra == "r2"
+Provides-Extra: all
+Requires-Dist: anthropic>=0.18; extra == "all"
+Requires-Dist: openai>=1.0; extra == "all"
+Requires-Dist: matplotlib>=3.7; extra == "all"
+Requires-Dist: datasets>=2.14; extra == "all"
+Requires-Dist: boto3>=1.34; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
+Dynamic: license-file
+
+# SynthBench
+
+[![CI](https://github.com/DataViking-Tech/synthbench/actions/workflows/ci.yml/badge.svg)](https://github.com/DataViking-Tech/synthbench/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue.svg)](pyproject.toml)
+[![Leaderboard](https://img.shields.io/badge/leaderboard-live-success)](https://synthbench.org/)
+
+Open benchmark harness for synthetic survey respondent quality.
+
+**The MLPerf of synthetic UXR.**
+
+SynthBench measures how well synthetic respondent systems (like [synthpanel](https://github.com/DataViking-Tech/SynthPanel), Ditto, Synthetic Users, or raw ChatGPT prompting) reproduce real human survey response patterns against real Pew American Trends Panel and GlobalOpinionQA ground truth — so "it sounds plausible" gets replaced with a measurable similarity score.
+
+## Quick Start
+
+Run your first benchmark in 3 commands:
+
+```bash
+pip install synthbench-eval
+synthbench run --provider random --suite smoke --output results/
+synthbench leaderboard --results-dir results/
+```
+
+> **Note:** the distribution is named `synthbench-eval` — the bare `synthbench`
+> name on PyPI belongs to an unrelated project. The import package and CLI are
+> still `synthbench`. For development, clone this repo and `pip install -e .`.
+> API-backed providers need an extra, e.g. `pip install "synthbench-eval[openai]"`
+> for the `openrouter` / `raw-openai` / `raw-gemini` / `ollama` providers.
+
+Try with a real model (requires API key):
+
+```bash
+export OPENROUTER_API_KEY=your-key
+synthbench run --provider openrouter --model openai/gpt-4o-mini --suite core --samples 50
+```
+
+See [`notebooks/getting_started.ipynb`](notebooks/getting_started.ipynb) for a guided walkthrough.
+
+## Leaderboard
+
+**[View the live leaderboard](https://synthbench.org/)** — see also the
+[methodology](https://synthbench.org/methodology/) and
+[findings](https://synthbench.org/findings/) pages.
+
+Regenerate leaderboard data for the Astro site:
+```bash
+synthbench publish-data --results-dir ./leaderboard-results --output site/src/data/leaderboard.json
+```
+
+### Contributor note: gated data publishing
+
+Most contributors do **not** need to publish gated artifacts. If you're running
+benchmarks locally or contributing via PR, focus on `synthbench run`,
+`synthbench validate`, and result submission.
+
+The gated data publication path is maintainer infrastructure and is handled by
+project deployment workflows.
+
+## Development
+
+After cloning, enable the repo-tracked git hooks so pushes that would fail CI's
+`ruff format --check` are caught locally:
+
+```bash
+./scripts/install-hooks.sh   # one-time: wires .githooks/ via core.hooksPath
+```
+
+The `pre-push` hook only checks Python files changed in the commits being
+pushed, so already-formatted branches add no meaningful overhead. Run
+`./scripts/format-check.sh` anytime to mirror the full CI lint job. Emergency
+bypass: `git push --no-verify`.
+
+For contribution workflow and PR expectations, see
+[`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## Submit Results
+
+Three ways to land a run on the leaderboard, in order of friction:
+
+### 1. CLI (recommended for repeat submissions)
+
+Mint an API key at [synthbench.org/account](https://synthbench.org/account/),
+then:
+
+```bash
+export SYNTHBENCH_API_KEY=sb_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+synthbench run --provider openrouter --model gpt-4o-mini --suite full -o results/
+synthbench submit results/openrouter_gpt-4o-mini_opinionsqa.json
+```
+
+#### End-to-end: run + submit in one command (`--submit`)
+
+Collapse the two steps above into a single invocation. The CLI saves the
+result JSON locally (so a validation rejection doesn't lose your run) and
+then POSTs it to the leaderboard. With `--wait`, the process blocks until
+the validator reaches a terminal state and the exit code mirrors the
+outcome — suitable for dropping into a CI pipeline that gates on
+leaderboard publication:
+
+```bash
+export SYNTHBENCH_API_KEY=sb_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+synthbench run \
+  --provider raw-anthropic --model claude-haiku-4-5 \
+  --dataset globalopinionqa --samples 30 -n 100 \
+  --submit --wait \
+  --submit-message "first pass with new prompt template"
+```
+
+Exit codes with `--wait`:
+
+| Code | Meaning |
+|-----:|---------|
+| 0    | Published — result is live on the leaderboard |
+| 1    | Rejected by the validator OR hard error (bad key, 5xx, etc.) |
+| 2    | Poll deadline exceeded (validation is still running server-side; check [/account/submissions/](https://synthbench.org/account/submissions/)) |
+
+Without `--wait`, the upload exits 0 as soon as the Worker accepts the
+submission (status = `validating`) — useful for fire-and-forget runs
+where you'll check the web dashboard later.
+
+`--submit-message` is optional; it's stored alongside the uploaded JSON
+so you can label experiments (e.g. "v2 prompt", "temp=1.0 sweep") without
+touching your `config` fields and perturbing the `config_id` hash.
+
+The Worker validates the submission, stages it to R2, and dispatches the
+GitHub Actions pipeline. Successful runs publish within ~5 minutes. Keys
+are rate-limited to 60 submissions/hour. See
+[SUBMISSIONS.md → API key flow](SUBMISSIONS.md#api-key-flow-cli-submission).
+
+### 2. Web upload
+
+Sign in at [/account](https://synthbench.org/account/) and drag your result
+JSON into [/submit/upload](https://synthbench.org/submit/upload/). Same
+validation pipeline, no key required.
+
+### 3. GitHub PR (power-user path)
+
+1. **Fork** this repo.
+2. **Run** SynthBench with your provider:
+   ```bash
+   synthbench run --provider <your-provider> --model <your-model> --suite full --output results/
+   ```
+3. **Validate locally** before opening a PR:
+   ```bash
+   synthbench validate results/<your-result>.json
+   ```
+4. **Copy** the result JSON into `leaderboard-results/`.
+5. **Open a PR** against this repo.
+6. **CI validates** schema, bounds, distributions, and recomputes every metric against the per-question data. Fabricated or inconsistent submissions are rejected.
+7. **Maintainers review and merge** — your results appear on the leaderboard.
+
+See [SUBMISSIONS.md](SUBMISSIONS.md) for the full list of integrity checks and common failure modes.
+
+## Key Research Findings
+
+Our benchmarking experiments across 3 models, 3 datasets, and 200+ runs reveal:
+
+| Finding | Impact |
+|---------|--------|
+| **3-model ensemble hits SPS 0.90** | Equal-weight average of Haiku + Gemini + GPT-4o-mini beats any single model by +5-7 pts |
+| **Temperature is model-specific** | Gemini benefits from high temp (+4.5%), Haiku is insensitive, GPT-4o-mini mild |
+| **Demographic conditioning quantifies LLM bias** | Republican conditioning 2.4x stronger than Democrat — model defaults approximate liberal responses |
+| **Persona template matters** | Default template beats stripped/broken templates by +11 SPS pts |
+
+See [FINDINGS.md](FINDINGS.md) for the full experimental report with methodology, replications, and per-metric breakdowns.
+
+## Status
+
+Phase 2 complete: Multi-model benchmarking, ensemble blending, temperature sweeps, and demographic conditioning analysis across OpinionsQA, SubPOP, and GlobalOpinionQA.
+
+## Ground Truth
+
+Built on nine registered survey datasets. Each adapter declares a
+redistribution policy; `full` ships `human_distribution` publicly, `gated`
+routes per-question artifacts to a JWT-authenticated Cloudflare R2 origin,
+and `aggregates_only` / `citation_only` contribute to leaderboard aggregates
+only. Canonical source of truth is the `redistribution_policy` attribute on
+each adapter in `src/synthbench/datasets/` (see
+[`src/synthbench/datasets/policy.py`](src/synthbench/datasets/policy.py)).
+
+| Dataset | Tier | Source |
+|---------|------|--------|
+| [OpinionsQA](https://github.com/tatsu-lab/opinions_qa) (Santurkar et al., ICML 2023) | gated | Pew American Trends Panel, 1,498 questions |
+| [GlobalOpinionQA](https://arxiv.org/abs/2306.16388) (Durmus et al., 2024) | gated | Pew Global Attitudes, 138 countries |
+| GSS (General Social Survey) | full | NORC, microdata-capable |
+| NTIA Internet Use Supplement | full | US Census / NTIA |
+| SubPOP | gated | 22 US subpopulations, 3,362 questions |
+| WVS (World Values Survey) | gated | WVSA, cross-national |
+| Eurobarometer | gated | European Commission |
+| Michigan (Surveys of Consumers) | gated | U. of Michigan |
+| Pew Technology | gated | Pew Research |
+
+GSS and NTIA ship with full per-question distributions; the remaining seven
+require a signed-in account to reach per-question payloads.
+
+## Cost tracking
+
+The leaderboard JSON carries per-row cost fields and a top-level
+`pricing_snapshot` object:
+
+- Each row exposes `cost_usd`, `cost_per_100q`, `cost_per_sps_point`, and
+  `is_cost_estimated`. Ensemble rows sum `cost_usd` across constituent
+  runs listed in `config.ensemble_sources`.
+- `pricing_snapshot` records the per-model `input_per_1m` / `output_per_1m`
+  rates used for this publish run, the `snapshot_date` anchor comment, and
+  the installed `synth_panel_version` that produced the rates.
+
+This lets downstream consumers audit which pricing table produced which
+`cost_usd` and reconcile against provider-reported billing without guessing
+at rate drift. See [sb-x8t] and `src/synthbench/publish.py::_build_pricing_snapshot`.
+
+## Convergence analysis
+
+`synthbench convergence bootstrap` computes theoretical ~1/√n convergence
+curves for every question in a dataset by multinomial resampling from the
+aggregate `human_distribution`. `synthbench convergence real` runs the same
+curve shape over individual-level microdata (GSS today; WVS / Eurobarometer
+microdata adapters are a follow-on). `synthbench convergence compare` emits
+both curves side-by-side.
+
+See [`docs/convergence-analysis.md`](docs/convergence-analysis.md) for the
+JSON schema, CLI flags, and the `load_convergence_baseline` integration
+surface that synthpanel's `--calibrate-against DATASET:QUESTION` flag
+consumes.
+
+## Citation
+
+If you use SynthBench in your research, please cite:
+
+```bibtex
+@misc{synthbench2026,
+  title={SynthBench: Open Benchmark for Synthetic Survey Respondent Quality},
+  author={DataViking-Tech},
+  year={2026},
+  url={https://github.com/DataViking-Tech/synthbench}
+}
+```
+
+## License
+
+MIT

synthbench_eval-0.4.0/README.md

@@ -0,0 +1,241 @@
+# SynthBench
+
+[![CI](https://github.com/DataViking-Tech/synthbench/actions/workflows/ci.yml/badge.svg)](https://github.com/DataViking-Tech/synthbench/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue.svg)](pyproject.toml)
+[![Leaderboard](https://img.shields.io/badge/leaderboard-live-success)](https://synthbench.org/)
+
+Open benchmark harness for synthetic survey respondent quality.
+
+**The MLPerf of synthetic UXR.**
+
+SynthBench measures how well synthetic respondent systems (like [synthpanel](https://github.com/DataViking-Tech/SynthPanel), Ditto, Synthetic Users, or raw ChatGPT prompting) reproduce real human survey response patterns against real Pew American Trends Panel and GlobalOpinionQA ground truth — so "it sounds plausible" gets replaced with a measurable similarity score.
+
+## Quick Start
+
+Run your first benchmark in 3 commands:
+
+```bash
+pip install synthbench-eval
+synthbench run --provider random --suite smoke --output results/
+synthbench leaderboard --results-dir results/
+```
+
+> **Note:** the distribution is named `synthbench-eval` — the bare `synthbench`
+> name on PyPI belongs to an unrelated project. The import package and CLI are
+> still `synthbench`. For development, clone this repo and `pip install -e .`.
+> API-backed providers need an extra, e.g. `pip install "synthbench-eval[openai]"`
+> for the `openrouter` / `raw-openai` / `raw-gemini` / `ollama` providers.
+
+Try with a real model (requires API key):
+
+```bash
+export OPENROUTER_API_KEY=your-key
+synthbench run --provider openrouter --model openai/gpt-4o-mini --suite core --samples 50
+```
+
+See [`notebooks/getting_started.ipynb`](notebooks/getting_started.ipynb) for a guided walkthrough.
+
+## Leaderboard
+
+**[View the live leaderboard](https://synthbench.org/)** — see also the
+[methodology](https://synthbench.org/methodology/) and
+[findings](https://synthbench.org/findings/) pages.
+
+Regenerate leaderboard data for the Astro site:
+```bash
+synthbench publish-data --results-dir ./leaderboard-results --output site/src/data/leaderboard.json
+```
+
+### Contributor note: gated data publishing
+
+Most contributors do **not** need to publish gated artifacts. If you're running
+benchmarks locally or contributing via PR, focus on `synthbench run`,
+`synthbench validate`, and result submission.
+
+The gated data publication path is maintainer infrastructure and is handled by
+project deployment workflows.
+
+## Development
+
+After cloning, enable the repo-tracked git hooks so pushes that would fail CI's
+`ruff format --check` are caught locally:
+
+```bash
+./scripts/install-hooks.sh   # one-time: wires .githooks/ via core.hooksPath
+```
+
+The `pre-push` hook only checks Python files changed in the commits being
+pushed, so already-formatted branches add no meaningful overhead. Run
+`./scripts/format-check.sh` anytime to mirror the full CI lint job. Emergency
+bypass: `git push --no-verify`.
+
+For contribution workflow and PR expectations, see
+[`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## Submit Results
+
+Three ways to land a run on the leaderboard, in order of friction:
+
+### 1. CLI (recommended for repeat submissions)
+
+Mint an API key at [synthbench.org/account](https://synthbench.org/account/),
+then:
+
+```bash
+export SYNTHBENCH_API_KEY=sb_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+synthbench run --provider openrouter --model gpt-4o-mini --suite full -o results/
+synthbench submit results/openrouter_gpt-4o-mini_opinionsqa.json
+```
+
+#### End-to-end: run + submit in one command (`--submit`)
+
+Collapse the two steps above into a single invocation. The CLI saves the
+result JSON locally (so a validation rejection doesn't lose your run) and
+then POSTs it to the leaderboard. With `--wait`, the process blocks until
+the validator reaches a terminal state and the exit code mirrors the
+outcome — suitable for dropping into a CI pipeline that gates on
+leaderboard publication:
+
+```bash
+export SYNTHBENCH_API_KEY=sb_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+synthbench run \
+  --provider raw-anthropic --model claude-haiku-4-5 \
+  --dataset globalopinionqa --samples 30 -n 100 \
+  --submit --wait \
+  --submit-message "first pass with new prompt template"
+```
+
+Exit codes with `--wait`:
+
+| Code | Meaning |
+|-----:|---------|
+| 0    | Published — result is live on the leaderboard |
+| 1    | Rejected by the validator OR hard error (bad key, 5xx, etc.) |
+| 2    | Poll deadline exceeded (validation is still running server-side; check [/account/submissions/](https://synthbench.org/account/submissions/)) |
+
+Without `--wait`, the upload exits 0 as soon as the Worker accepts the
+submission (status = `validating`) — useful for fire-and-forget runs
+where you'll check the web dashboard later.
+
+`--submit-message` is optional; it's stored alongside the uploaded JSON
+so you can label experiments (e.g. "v2 prompt", "temp=1.0 sweep") without
+touching your `config` fields and perturbing the `config_id` hash.
+
+The Worker validates the submission, stages it to R2, and dispatches the
+GitHub Actions pipeline. Successful runs publish within ~5 minutes. Keys
+are rate-limited to 60 submissions/hour. See
+[SUBMISSIONS.md → API key flow](SUBMISSIONS.md#api-key-flow-cli-submission).
+
+### 2. Web upload
+
+Sign in at [/account](https://synthbench.org/account/) and drag your result
+JSON into [/submit/upload](https://synthbench.org/submit/upload/). Same
+validation pipeline, no key required.
+
+### 3. GitHub PR (power-user path)
+
+1. **Fork** this repo.
+2. **Run** SynthBench with your provider:
+   ```bash
+   synthbench run --provider <your-provider> --model <your-model> --suite full --output results/
+   ```
+3. **Validate locally** before opening a PR:
+   ```bash
+   synthbench validate results/<your-result>.json
+   ```
+4. **Copy** the result JSON into `leaderboard-results/`.
+5. **Open a PR** against this repo.
+6. **CI validates** schema, bounds, distributions, and recomputes every metric against the per-question data. Fabricated or inconsistent submissions are rejected.
+7. **Maintainers review and merge** — your results appear on the leaderboard.
+
+See [SUBMISSIONS.md](SUBMISSIONS.md) for the full list of integrity checks and common failure modes.
+
+## Key Research Findings
+
+Our benchmarking experiments across 3 models, 3 datasets, and 200+ runs reveal:
+
+| Finding | Impact |
+|---------|--------|
+| **3-model ensemble hits SPS 0.90** | Equal-weight average of Haiku + Gemini + GPT-4o-mini beats any single model by +5-7 pts |
+| **Temperature is model-specific** | Gemini benefits from high temp (+4.5%), Haiku is insensitive, GPT-4o-mini mild |
+| **Demographic conditioning quantifies LLM bias** | Republican conditioning 2.4x stronger than Democrat — model defaults approximate liberal responses |
+| **Persona template matters** | Default template beats stripped/broken templates by +11 SPS pts |
+
+See [FINDINGS.md](FINDINGS.md) for the full experimental report with methodology, replications, and per-metric breakdowns.
+
+## Status
+
+Phase 2 complete: Multi-model benchmarking, ensemble blending, temperature sweeps, and demographic conditioning analysis across OpinionsQA, SubPOP, and GlobalOpinionQA.
+
+## Ground Truth
+
+Built on nine registered survey datasets. Each adapter declares a
+redistribution policy; `full` ships `human_distribution` publicly, `gated`
+routes per-question artifacts to a JWT-authenticated Cloudflare R2 origin,
+and `aggregates_only` / `citation_only` contribute to leaderboard aggregates
+only. Canonical source of truth is the `redistribution_policy` attribute on
+each adapter in `src/synthbench/datasets/` (see
+[`src/synthbench/datasets/policy.py`](src/synthbench/datasets/policy.py)).
+
+| Dataset | Tier | Source |
+|---------|------|--------|
+| [OpinionsQA](https://github.com/tatsu-lab/opinions_qa) (Santurkar et al., ICML 2023) | gated | Pew American Trends Panel, 1,498 questions |
+| [GlobalOpinionQA](https://arxiv.org/abs/2306.16388) (Durmus et al., 2024) | gated | Pew Global Attitudes, 138 countries |
+| GSS (General Social Survey) | full | NORC, microdata-capable |
+| NTIA Internet Use Supplement | full | US Census / NTIA |
+| SubPOP | gated | 22 US subpopulations, 3,362 questions |
+| WVS (World Values Survey) | gated | WVSA, cross-national |
+| Eurobarometer | gated | European Commission |
+| Michigan (Surveys of Consumers) | gated | U. of Michigan |
+| Pew Technology | gated | Pew Research |
+
+GSS and NTIA ship with full per-question distributions; the remaining seven
+require a signed-in account to reach per-question payloads.
+
+## Cost tracking
+
+The leaderboard JSON carries per-row cost fields and a top-level
+`pricing_snapshot` object:
+
+- Each row exposes `cost_usd`, `cost_per_100q`, `cost_per_sps_point`, and
+  `is_cost_estimated`. Ensemble rows sum `cost_usd` across constituent
+  runs listed in `config.ensemble_sources`.
+- `pricing_snapshot` records the per-model `input_per_1m` / `output_per_1m`
+  rates used for this publish run, the `snapshot_date` anchor comment, and
+  the installed `synth_panel_version` that produced the rates.
+
+This lets downstream consumers audit which pricing table produced which
+`cost_usd` and reconcile against provider-reported billing without guessing
+at rate drift. See [sb-x8t] and `src/synthbench/publish.py::_build_pricing_snapshot`.
+
+## Convergence analysis
+
+`synthbench convergence bootstrap` computes theoretical ~1/√n convergence
+curves for every question in a dataset by multinomial resampling from the
+aggregate `human_distribution`. `synthbench convergence real` runs the same
+curve shape over individual-level microdata (GSS today; WVS / Eurobarometer
+microdata adapters are a follow-on). `synthbench convergence compare` emits
+both curves side-by-side.
+
+See [`docs/convergence-analysis.md`](docs/convergence-analysis.md) for the
+JSON schema, CLI flags, and the `load_convergence_baseline` integration
+surface that synthpanel's `--calibrate-against DATASET:QUESTION` flag
+consumes.
+
+## Citation
+
+If you use SynthBench in your research, please cite:
+
+```bibtex
+@misc{synthbench2026,
+  title={SynthBench: Open Benchmark for Synthetic Survey Respondent Quality},
+  author={DataViking-Tech},
+  year={2026},
+  url={https://github.com/DataViking-Tech/synthbench}
+}
+```
+
+## License
+
+MIT

synthbench_eval-0.4.0/pyproject.toml

@@ -0,0 +1,79 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+# Distribution name is synthbench-eval: the bare "synthbench" name on PyPI
+# belongs to an unrelated project. Import package and CLI remain `synthbench`.
+name = "synthbench-eval"
+# Placeholder — the publish job stamps the real version from the release tag
+# at build time (see publish-pypi in .github/workflows/auto-tag.yml).
+version = "0.4.0"
+description = "Open benchmark harness for synthetic survey respondent quality"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.9"
+keywords = [
+    "benchmark",
+    "survey",
+    "synthetic-data",
+    "llm",
+    "evaluation",
+    "opinionsqa",
+    "globalopinionqa",
+    "gss",
+    "convergence",
+    "synthpanel",
+]
+dependencies = [
+    "click>=8.0",
+    "httpx>=0.24",
+    "numpy>=1.24",
+    "pyyaml>=6.0",
+    "scipy>=1.10",
+    "synthpanel>=0.2.0",
+]
+
+[project.optional-dependencies]
+anthropic = ["anthropic>=0.18"]
+openai = ["openai>=1.0"]
+viz = ["matplotlib>=3.7"]
+hf = ["datasets>=2.14"]
+subpop = ["datasets>=2.14"]
+# R2 publish target for gated-tier per-question/run/config artifacts (sb-sjs).
+# Only required in environments that actually upload (CI publish step); local
+# dev defaults to writing to site/public/data without R2.
+r2 = ["boto3>=1.34"]
+all = [
+    "anthropic>=0.18",
+    "openai>=1.0",
+    "matplotlib>=3.7",
+    "datasets>=2.14",
+    "boto3>=1.34",
+]
+dev = ["pytest>=7.0", "pytest-asyncio>=0.21"]
+
+[project.scripts]
+synthbench = "synthbench.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+# Holdout answer-key data (issue #259) is intentionally NOT included in
+# the wheel. Raw upstream survey items are downloaded at runtime by the
+# dataset adapters into the user's data directory, never from the package
+# install. The public / private-holdout partition is a logical filter over
+# question keys (see ``synthbench.private_holdout``), and the canonical
+# private *answer key* (per-question human distributions) lives only in
+# the gated R2 origin + credentialed maintainer caches. We therefore
+# deliberately leave ``[tool.setuptools.package-data]`` unset for
+# synthbench — the only files shipped are the Python sources resolved by
+# ``packages.find``. Adding raw data globs here would break the holdout
+# integrity model. See ``docs/held-out.md``.
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]
+# Prepend local src/ to sys.path so the in-tree synthbench package shadows any
+# stale `pip install -e .` entries from sibling worktrees (sb-3bn).
+pythonpath = ["src"]

synthbench_eval-0.4.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

synthbench_eval-0.4.0/src/synthbench/__init__.py

@@ -0,0 +1,16 @@
+"""SynthBench — open benchmark harness for synthetic survey respondent quality."""
+
+from synthbench.convergence.baseline import (
+    BaselineGatedError,
+    BaselineUnavailable,
+    load_convergence_baseline,
+)
+
+__version__ = "0.4.0"
+
+__all__ = [
+    "BaselineGatedError",
+    "BaselineUnavailable",
+    "load_convergence_baseline",
+    "__version__",
+]

synthbench_eval-0.4.0/src/synthbench/__main__.py

@@ -0,0 +1,5 @@
+"""Allow running as `python -m synthbench`."""
+
+from synthbench.cli import main
+
+main()