debatebench 0.1.0__tar.gz

debatebench-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,107 @@
+# ADR-029: publish to PyPI with no PyPI credential anywhere. The job proves it
+# is this repository to PyPI over OIDC, so there is no token to store, rotate,
+# leak or revoke — which is the specific failure this project has already had.
+#
+# Before the first run, the account owner must create a *pending publisher* at
+# https://pypi.org/manage/account/publishing/ binding:
+#     project      debatebench
+#     owner        robert-e-roy
+#     repository   debatebench
+#     workflow     release.yml
+#     environment  pypi
+# Until that exists the upload step fails to authenticate, which is the intended
+# failure mode: nothing can publish by accident.
+name: release
+
+# A *published GitHub Release*, not a tag push (ADR-029 §3): pushing a tag by
+# mistake must not publish anything, and a version on PyPI cannot be replaced.
+# No workflow_dispatch: a manual run would publish from whatever main happens to
+# be, which is the same accident §3 rules out for a stray tag push, by another
+# route. Re-running a failed release means a new version, because PyPI will not
+# take 0.1.0 twice however the first attempt went.
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      # Pinned exactly: setup-uv publishes no floating major tag (see tests.yml).
+      - uses: astral-sh/setup-uv@v10.1.0
+        with:
+          python-version: "3.11"
+
+      # The suite runs here too. The release artifact and the tested tree are
+      # then the same checkout, rather than two things assumed to match.
+      - run: uv sync --locked
+      - run: uv run --locked pytest -q
+
+      - name: Build the wheel and sdist from the tagged commit
+        run: uv build
+
+      - name: Refuse metadata PyPI would reject
+        run: uvx twine check dist/*
+
+      - uses: actions/upload-artifact@v7
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    # Named so the trusted-publisher binding can require it, and so a reviewer
+    # can be added later without changing this file.
+    environment:
+      name: pypi
+      url: https://pypi.org/p/debatebench
+    permissions:
+      id-token: write  # the OIDC token; this is the whole authentication story
+    steps:
+      - uses: actions/download-artifact@v8
+        with:
+          name: dist
+          path: dist/
+
+      - uses: pypa/gh-action-pypi-publish@v1.14.2
+
+  # twine check validates metadata, not that the thing installs and runs. B7's
+  # gate clause was "install from an index and run", and that is what proved
+  # determinism across a packaging boundary — so the release repeats it against
+  # real PyPI. This cannot prevent a bad upload; it tells us within a minute
+  # whether to yank, which is the only remedy ADR-029 §6 leaves.
+  verify:
+    needs: publish
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.11"
+
+      - name: Install from real PyPI into a venv that has never seen the source
+        run: |
+          version="$(python -c 'import tomllib;print(tomllib.load(open("pyproject.toml","rb"))["project"]["version"])')"
+          echo "waiting for debatebench==$version to be installable"
+          cd "$(mktemp -d)"           # out of the checkout, so nothing resolves locally
+          python -m venv venv
+          # PyPI's index can lag the upload by under a minute; retry rather than
+          # fail a release on CDN timing.
+          for attempt in 1 2 3 4 5 6 7 8 9 10; do
+            if ./venv/bin/pip install --no-cache-dir "debatebench==$version"; then
+              installed=yes; break
+            fi
+            echo "attempt $attempt failed; retrying in 15s"
+            sleep 15
+          done
+          [ "${installed:-no}" = yes ] || { echo "::error::debatebench==$version never became installable"; exit 1; }
+
+          ./venv/bin/debate --help > /dev/null
+          ./venv/bin/judge --help > /dev/null
+          ./venv/bin/python -c "from debatebench.api import debate, judge, load_run; from debatebench.transcript import package_version; print('installed version:', package_version())"
+          echo "debatebench==$version installs from PyPI and both commands run"

debatebench-0.1.0/.github/workflows/tests.yml

@@ -0,0 +1,44 @@
+# CLAUDE.md said "CI is deferred to B7" from B0 until 2026-09-16. B7 finished on
+# 2026-09-14 without a workflow because there was no remote to run one; the repo
+# now exists, so this closes it.
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # pyproject declares >=3.11 and classifies 3.11 and 3.12. Both run.
+        python-version: ["3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v7
+
+      # Pinned exactly: astral-sh/setup-uv publishes no floating `v10` tag —
+      # only v10.0.0, v10.0.1, v10.1.0 — so `@v10` fails to resolve at all.
+      # actions/checkout does publish `v7`, hence the difference above.
+      - uses: astral-sh/setup-uv@v10.1.0
+        with:
+          python-version: ${{ matrix.python-version }}
+          enable-cache: true
+
+      # --locked, not --frozen: a uv.lock that has drifted from pyproject should
+      # fail here rather than be silently re-resolved into something untested.
+      - run: uv sync --locked
+
+      # DEBATEBENCH_LIVE_TESTS is deliberately unset, so every test runs against
+      # the scripted FakeBackend (ADR-004) and the live-model tests skip. CI has
+      # no Ollama, no AFM and no GPU, and a suite that needed one would be a
+      # suite nobody could run.
+      - name: pytest (fake backend only; live-model tests skip)
+        run: uv run --locked pytest -q

debatebench-0.1.0/.gitignore

@@ -0,0 +1,27 @@
+# Python
+__pycache__/
+*.pyc
+.venv/
+.pytest_cache/
+
+# Build artifacts (`uv build` writes here). Ignored by a global excludes file on
+# the machine this was developed on, which does not survive a clone.
+dist/
+build/
+*.egg-info/
+
+# Editors / macOS
+*.swp
+.DS_Store
+
+# Run artifacts. The README tells people to run `debate examples/run.yaml`,
+# whose `output:` resolves next to the run.yaml — so a transcript, its rotated
+# `.1` backup, and the score file all land inside examples/ on a plain clone.
+examples/transcript.json
+examples/transcript.json.1
+examples/scores.json
+examples/scores.json.1
+examples/transcript-prep.json
+examples/transcript-prep.json.1
+examples/scores-prep.json
+examples/scores-prep.json.1

debatebench-0.1.0/.python-version

@@ -0,0 +1 @@
+3.11

debatebench-0.1.0/ADR-001-orchestration-build-vs-reuse.md

@@ -0,0 +1,126 @@
+# ADR-001: Debate Orchestration Layer — Build vs. Reuse
+
+**Status:** Accepted
+**Date:** 2026-09-10
+**Amended:** 2026-09-11 — rationale wording and cross-references corrected against
+the R0 results; the decision itself is unchanged.
+**Context session:** R0 repo review (aragora-debate, arbgjr/multi-agent-debate,
+paolodalprato/agent-discussion-arena, rd-serendipity/ai-debate-arena). The brief and
+full results are in this repo, as `R0-repo-review-session.md` and `R0-RESULTS.md`
+(moved from `~/Projects/DebateKit/` on 2026-09-11); scores cited below come from
+the latter.
+
+## Decision
+
+Build our own orchestration layer from scratch (~400–500 lines, zero required
+dependencies). Do not adopt or fork any of the four reviewed projects. Lift three
+narrow, attributed pieces; take three design ideas; design against four observed
+anti-patterns.
+
+"Zero required dependencies" applies to the orchestration layer itself — the phase
+loop, event dispatch and backend `Protocol` that R0 sized at ~400–500 lines. Config
+parsing sits outside that layer and can't be stdlib-only (ADR-002's config is YAML,
+and Python's standard library has no YAML parser). The HTTP adapter can stay
+stdlib-only (`urllib` under `asyncio.to_thread`, which anti-pattern 4 below permits)
+or take an async client; ADR-003 leaves that open.
+
+## Why
+
+None of the four reviewed projects fits, and not for one shared reason. Two
+(`aragora-debate`, `arbgjr/multi-agent-debate`) are built to make agents converge
+toward agreement, and in both that objective is hardcoded into the turn loop
+itself, not exposed as a setting (R0 consensus-separation scores 2/5 and 1/5).
+`rd-serendipity/ai-debate-arena` has no consensus logic at all (5/5) but engineering
+we can't build on. `agent-discussion-arena` has no importable orchestration code at
+all. Our requirement — sustained, non-converging adversarial debate with per-side
+model/budget asymmetry — is not a configuration difference from consensus-seeking;
+it's the opposite objective.
+
+The clearest single data point: the project with the best engineering
+(`aragora-debate` — clean `Agent` ABC, excellent standalone event system, verified
+MIT license) has the worst fit on the one axis that matters most, separation from
+consensus machinery (2/5). The project with the best fit on that axis
+(`rd-serendipity`, true 1v1, no consensus logic anywhere) has the worst engineering
+— LangChain-bound to September 2024 pins, fully synchronous, dead two years, no
+tests. Nothing scored well on both. That is the whole decision.
+
+## What we take (with attribution — all four repos are MIT)
+
+| From | What | Form |
+|---|---|---|
+| `aragora-debate/events.py` | Typed `EventType` enum + `DebateEvent` dataclass, sync+async dispatch, per-callback exception isolation | Design lift, ~120 lines, reimplemented |
+| `arbgjr/multi-agent-debate` `protocols.py` | Single-method `async generate(...)` `Protocol` as the LLM backend seam (not a multi-method ABC) | Design lift, ~20 lines |
+| `aragora-debate/evidence.py` | Stdlib-only, LLM-free, five separately-scored dimensions (citation density, specificity, evidence diversity, temporal relevance, logical chain) | Conditional lift — see the fact-checker bullet under "Decisions this locks in" on naming (R0 flag #5) |
+| `paolodalprato/agent-discussion-arena` | Phase model: `Opening → Round 1..N → Verdict`, verdict as a distinct terminal phase, not a vote folded into the loop | Idea only, no code |
+| `paolodalprato/agent-discussion-arena` | One `openai-compatible` adapter with configurable `base_url` serves MLX (`mlx_lm.server`), Ollama, LM Studio, and OpenAI itself | Idea only, no code |
+| `rd-serendipity/ai-debate-arena` | Alternating initiative — swap who opens each round so neither side keeps first-mover advantage | Idea only, ~2 lines when implemented |
+
+## Anti-patterns to design against (each observed as a real bug in a reviewed repo)
+
+1. **Never let a phase failure be silent.** Aragora's `_run_propose`/`_run_vote` use
+   `return_exceptions=True` and `continue` past a failed side with only a log
+   warning; `_pick_winner` then defaults to `agents[0]`, so a one-sided debate
+   still emits a confident-looking result. **Our rule: every phase must produce a
+   response from every configured side, or the run hard-fails and `--output` is
+   not written.**
+2. **Never key transcript state by side alone.** Aragora's `_proposals` dict is
+   overwritten each round, so `result.proposals` only ever holds the final round.
+   **Our rule: key all transcript state by `(round, side)`.**
+3. **Never blend judge dimensions into one number, and never return 0 on a parse
+   failure.** rd-serendipity's judge asks for a single 0–100 score blending
+   relevance, coherence, and fact-check, parsed by `RegexParser(r"(\d+)")`, which
+   grabs the *first* integer anywhere in the output — a reply starting "Criteria
+   1..." scores 1 — and returns 0 on any parse failure, indistinguishable from a
+   genuinely terrible argument. **Our rule: score rubric dimensions separately;
+   treat a parse failure as an error, never a score.**
+4. **Never call a synchronous SDK client inside `async def`.** All four of
+   Aragora's provider agents construct sync clients (`anthropic.Anthropic`,
+   `openai.OpenAI`, etc.) and call them directly inside `async def` with no
+   `await` and no `asyncio.to_thread` — so `asyncio.gather` across agents
+   delivers zero real concurrency and blocks the event loop. For a tool also
+   driving a live dashboard, this would stall the UI. **Our rule: use a real
+   async client, or wrap sync calls in `asyncio.to_thread`.**
+
+## Decisions this locks in (changes/sharpens the prior plan)
+
+- **Two commands (`debate` / `judge`) stays as designed.** Aragora's fusion of
+  judging into the debate loop is exactly what forces its wasted vote-phase calls.
+  No reviewed project argues for merging the two.
+- **`debate` writes to `--output` file only stays as designed.** No conflicts
+  found. Aragora's `DebateResult.to_dict()` + `DecisionReceipt.to_markdown()` is a
+  reasonable precedent for pairing a machine-readable and human-readable view of
+  one run in the output schema.
+- **Phases must be data, not control flow.** Both Aragora and arbgjr hardcode
+  their phase sequence as literal branches in the loop body — Aragora even has a
+  `Phase` enum its own loop ignores, with two dead members. **Our loop iterates a
+  list of phases; it does not branch on round number.**
+- **Per-side budgets are enforced by the orchestrator, per phase — not by each
+  backend's constructor.** Aragora's `max_tokens`/`temperature` are genuine
+  per-agent constructor args, but every provider's `vote()` hardcodes its own
+  512-token, 0.3-temperature budget regardless of what was configured, so the
+  configured budget silently doesn't hold across every phase. For a benchmarking
+  tool, an unenforced budget claim is worse than no budget claim.
+- **Fast fact-checker vs. slow end-of-debate judge split is vindicated.** No
+  reviewed project implements this separation. Aragora's `evidence.py` is the
+  nearest analogue but is pure regex over surface form (citation patterns, data
+  patterns, vague-vs-specific phrasing) with zero LLM calls and zero verification
+  of truth — a confidently fabricated citation with a recent year and a
+  percentage scores *high*. **If adopted, it is labeled "evidence hygiene," not
+  "fact-check," and sits alongside a real fact-checker, never in place of one.**
+- **New invariant, not previously in the plan:** every phase must produce a
+  response from every configured side, or the run aborts and no output file is
+  written. (See anti-pattern 1.)
+
+## Consequences
+
+- We own and maintain the full orchestration loop (~400–500 lines). No external
+  dependency risk from a project whose roadmap has nothing to do with ours.
+- We do not get Aragora's four cloud-provider integrations for free, but we didn't
+  want them as the primary path anyway (MLX-first).
+- Building the `Agent`/backend seam as a single-method protocol (per arbgjr's
+  shape) keeps an MLX implementation to one method rather than three, avoiding the
+  work an Aragora-based `Agent` subclass would require (structured `Critique`/
+  `Vote` parsing out of a local model).
+- Attribution required in our source/NOTICE file for the two design lifts
+  (`events.py` shape, `LLMProviderProtocol` shape) and the conditional
+  `evidence.py`-derived evidence-hygiene scorer, per each project's MIT license.

debatebench-0.1.0/ADR-002-debatebench-architecture.md

@@ -0,0 +1,356 @@
+# ADR-002: DebateBench — Project Scope and Architecture
+
+**Status:** Accepted
+**Date:** 2026-09-11
+**Supersedes:** nothing. **Depends on:** ADR-001 (orchestration build vs. reuse)
+**Amended:** 2026-09-11 — cross-references, dataset facts and wording corrected. B0's
+hardware findings added to "Hardware", withdrawing the `free` offload option. An
+offline requirement for model servers added to "Backend abstraction".
+
+## What this is
+
+`debatebench` is an open-source Python CLI for running structured, multi-turn,
+adversarial LLM debates and scoring them against a fixed rubric, with a
+post-hoc fact-check pass inside `judge` (ADR-015; real-time per-turn
+fact-checking is the Swift app's feature, not this repo's). It generalizes into a broader pattern (see the MetaTool notes,
+`~/Projects/metatool.md`, outside this repo) but this ADR scopes the concrete first
+implementation.
+
+A Swift port, `DebateKit`, is a later, separate effort once this design stops
+changing session to session (see "Language split" below). Not started yet.
+
+## Why it exists
+
+Built to evaluate MLX model + grounding-dataset combinations for a debate-format
+Mac app, but scoped from the start to be generally useful for comparing LLMs on
+structured, verifiable, multi-turn tasks — not tied to the app.
+
+## Language split: Python first, Swift later
+
+- **Python** owns everything exploratory: model/dataset evaluation, retrieval,
+  rubric iteration, fact-checking logic. Mature ML tooling (`mlx-lm`, `datasets`,
+  dataset ecosystem) makes iteration fast here.
+- **Swift** (`DebateKit`, future) owns the shipped product: privacy-first,
+  on-device, no bundled Python runtime, consistent with the rest of the
+  HomesteadAI portfolio. Ported only once the CLI's design has stabilized —
+  porting a moving target wastes the port.
+- Publish both source repos on GitHub. Python side is a genuine standalone
+  open-source tool (MIT). Swift side's license is a separate, later decision —
+  likely source-available/non-commercial rather than MIT, since its purpose is
+  auditability and trust for the paid app, not general reuse.
+
+## Naming
+
+- Python package: `debatebench` (PyPI name confirmed unclaimed as of this
+  session). `bench` was chosen over `-cli` because it correctly signals
+  "measures and compares models," matching existing conventions in this space
+  (`lm-eval-harness`, similar `*-eval`/`*-bench` tools).
+- Swift library (future): `DebateKit`, consistent with existing `Kit`-suffixed
+  libraries (CorpusKit, JsonHelpKit).
+- Neither collides with any of the ten-plus adjacent "debate"-named projects
+  checked this session (`debate`, `aragora-debate`, `arbiter-debate`,
+  `ai-debate-chat`, `multi-model-debate`, `autodebater`, `debating-machine`,
+  and the four in the R0 review).
+
+## CLI shape: two commands, not one
+
+- **`debate`** — generates a transcript. Writes structured JSON to an explicit
+  output file only — the required `output:` path in `run.yaml`, since `debate`
+  takes no flags (ADR-007 "CLI invocation"). All logging/errors go to stderr, never to the
+  output file — the file must always be valid, parseable JSON, never relying on
+  shell redirection to keep it clean.
+- **`judge`** — scores a transcript (from a file, not a pipe requirement) against
+  the rubric plus fact-check. Kept separate from `debate` deliberately: lets the
+  same transcript be re-judged with a different judge model, or re-scored after
+  a rubric change, without regenerating the debate. (Aragora's fused
+  debate+judge loop is a cautionary example — it burns a mandatory vote-phase
+  call every round even when unwanted, precisely because judging isn't
+  separable from generation in its design.)
+
+## Config: two file types, different lifespans
+
+**`run.yaml`** — one per debate run, ephemeral (schema finalized in ADR-007):
+```yaml
+topic: "..."
+format:
+  phases: [prep, opening:short, rebuttal:long, retort:medium, rebuttal:long, conclusion:short]
+teams:
+  - team: teams/liberal.yaml
+    side: pro
+    model: qwen3-8b
+    base_url: http://127.0.0.1:8080/v1
+    budget: 2000
+    prep_budget: 1500
+  - team: teams/conservative.yaml
+    side: con
+    model: mistral-small
+    base_url: http://127.0.0.1:8081/v1
+    budget: 2000
+    prep_budget: 1500
+sources: [args-me, debatesum]
+seed: 42
+output: transcript.json
+```
+
+**`teams/*.yaml`** — durable, reusable, hand-authored for now (see ADR-006 —
+the earlier "PersonaForge-compatible, zero conversion step" claim here was
+checked against PersonaKit's actual schema and found false; aligning the two
+is real future work, not started):
+```yaml
+id: liberal-climate
+name: "Progressive Climate Advocate"
+voice: "direct, urgency-driven, cites institutional consensus"
+stance: liberal
+corpus: liberal-climate-corpus.jsonl
+values: [collective-action, precaution, equity]
+```
+
+`corpus` is a JSONL file resolved relative to the team file's own directory,
+not `run.yaml`'s — see ADR-014 §1 for the row format and why.
+
+**Deliberately no `model` field in team files.** A team file defines identity
+(voice, corpus, stance, values), not runtime execution. Keeping `model`/`budget`
+in `run.yaml` as per-side overrides means the same team can debate under
+different hardware/model constraints across different runs without editing its
+identity file — that reasoning holds regardless of PersonaForge (see ADR-006).
+
+## Phase structure
+
+Phases are **data, not control flow** — the orchestration loop iterates a
+configured phase list; it does not branch on round number in the loop body (both
+`aragora-debate` and `arbgjr/multi-agent-debate` hardcode their phase sequence as
+control flow — see ADR-001).
+
+Default phase list: `[prep(optional), opening, rebuttal, retort, rebuttal,
+conclusion]`. `prep` is a bounded-budget research phase — analogous to real
+debate prep time — that produces a recorded, per-side evidence set *before* any
+argument is written. This sharpens fact-checking: a claim is checked against what
+a side actually gathered during its own prep, not an open-ended post-hoc search,
+making "cited something you never had" a distinct, catchable failure mode from
+"cited something real but misrepresented it."
+
+Initiative (who goes first each round) alternates by design, so neither side
+keeps a first-mover advantage across the whole debate.
+
+## Asymmetry is a first-class feature, not an edge case
+
+Different teams may run different models under different reasoning/token budgets
+(chess-clock analogy). This is core to the product's value, not a config nicety
+— it's what makes "does a bigger, slower model beat a faster, leaner one" a real,
+answerable, watchable question. Budgets are **enforced by the orchestrator, per
+phase** — never left to a backend's own constructor defaults, which is exactly
+how Aragora's budget claims silently failed to hold across every phase (see
+ADR-001).
+
+## Backend abstraction
+
+Single-method `async` `Protocol` as the LLM seam (lifted design from
+`arbgjr/multi-agent-debate`), not a multi-method ABC. One `openai-compatible`
+adapter with a configurable `base_url` (idea from `agent-discussion-arena`)
+covers MLX (`mlx_lm.server`), Ollama, LM Studio, and OpenAI itself through one
+code path.
+
+**Dev default model is Apple Foundation Models (AFM)**, or another very small
+model — instant, free, no MLX load overhead, no memory pressure. Used while
+building and testing the CLI plumbing itself (config parsing, phase loop,
+output-file discipline, judge scoring format). Real MLX candidate models are
+reserved for once the harness is proven and the actual question becomes "is this
+debate any good," not "does the pipe work."
+
+**Never call a synchronous SDK client inside `async def`** without
+`asyncio.to_thread` — observed as a real, unnoticed bug (fake concurrency) in all
+four of `aragora-debate`'s cloud-provider agents (see ADR-001).
+
+**Model servers run offline — a requirement, not an option.** Every model server
+runs with `HF_HUB_OFFLINE=1`, and anything else that fetches from the Hugging
+Face Hub (such as `datasets` for `sources`, later) uses its equivalent offline
+setting. B0 found `mlx_lm.server` contacts huggingface.co on every start to check
+the model revision. That contradicts the offline-capable, privacy-first
+positioning this tool is built around. In offline mode, already-downloaded
+models resolve from the local cache (verified 2026-09-11). Downloading a model is
+a separate, explicit step.
+
+## Judge design
+
+Five independently-scored rubric dimensions, never blended into one number:
+argument quality & logic (30), evidence grounding (25), steelman fidelity (20),
+rebuttal effectiveness (15), clarity (10). Steelman fidelity is the explicit
+tiebreaker, given the app's core premise.
+
+Rebuttal effectiveness is tracked via a **structured hit-ledger**
+(open/conceded/rebutted/dodged per point — idea from `arbiter-debate`), not a
+subjective number alone.
+
+A narrow, separate **fallacy-detector** check (idea from `autodebater`'s
+"Bullshit Detector" judge) is a candidate fast signal, same tier as the
+fact-checker — not yet decided whether it's folded into fact-checking or kept as
+its own pass.
+
+A candidate **stance-consistency** check (idea surfaced reviewing
+`mlburnham/Political_DEBATE_large_v1.0`, a 0.4B DeBERTa NLI classifier — name
+is a backronym, "DeBERTa Algorithm for Textual Entailment," unrelated to
+adversarial debate) is a second candidate fast signal: cheap enough to run
+after every turn without competing for resources with the debater models,
+verifying a side's turn still entails its assigned stance rather than drifting
+or conceding. Not yet decided whether this is a third fast pass alongside
+fact-checking and the fallacy detector, or folded into one of them.
+
+**Validation data for the rubric itself, before trusting any judge model
+choice:**
+- `ibm-research/argument_quality_ranking_30k` (IBM-Rank-30k; CC-BY-3.0) — 30,497 crowd-written
+  arguments across 71 genuinely contested policy topics (same domain as this
+  app, not academic claims or trivia), each with a human quality score (WA and
+  MACE-P methods) and stance label. Use to correlate a candidate judge's
+  argument-quality scores against real human judgment before trusting it.
+- `ibm-research/debate_speeches` (CDLA-Permissive-2.0) — the HF release
+  (`opening_speeches` config) holds 948 opening speeches across 114 topics from
+  nine sources, human-written and machine-generated, each rated by crowd
+  annotators. **Resolved 2026-09-13 by reading the paper and the data:** the
+  631-speech / 76-topic figure *is* the paper's subset. It removes **78**
+  control speeches — not 81 — plus two topics lacking both Arg-Human sources.
+  Better still, the authors publish that filtered set directly as
+  `noystl/speech-quality-dataset` (CDLA-Permissive-2.0, one `data.csv`), so
+  there is no need to re-derive it from the 948-row release and no chance of
+  filtering it differently than they did.
+
+  Verified locally against that file: 631 speeches, 76 topics, **exactly 15**
+  labelers each (not 5–30), 9,465 individual ratings on a 1–5 Likert scale held
+  per-annotator rather than pre-averaged, mean 605.6 words per speech. Mean
+  score by source forms the gradient a judge has to reproduce: Human expert
+  4.19, Project Debater 4.03, Arg-Human1 3.79, Arg-Human2 3.67, Arg-GPT2 3.40,
+  Speech-GPT2 3.22, Arg-Search 3.12, Summit 2.96. Purpose-built for benchmarking whether an LLM judge's scores track human
+  judgment at the speech/turn level (closer to our unit of judging than
+  Rank-30k's isolated arguments). A recent paper, "Benchmarking LLM Judges via
+  Debate Speech Evaluation," used this exact dataset for this exact task — read
+  it for methodology before designing our own validation pass.
+- `tasksource/logical-fallacy` (Jin et al. 2022) — 3,761 labeled examples
+  across 13-14 fallacy types, real naturalistic argument text, includes a
+  `LogicClimate` challenge subset specifically for climate-policy claims. Real
+  train/test/dev splits; multiple independent models already fine-tuned on it.
+  Candidate training/validation set for the fallacy-detector check above.
+  **License: none found, but the authors' README explicitly says "Feel free to
+  access our data in the `data/` folder."** No LICENSE file, HF lists
+  `unknown`. Treat this as two separate permissions, not one: sufficient for
+  private use (downloading it ourselves to validate a fallacy-detector or judge
+  against) but not for redistribution — nothing grants the right to bundle or
+  republish the data as part of a `debatebench` release. Keep for validation
+  use; do not ship any of it in this repo without a clearer grant.
+
+## Fact-checking is separate from judging, and runs on a different clock
+
+**Superseded for the Python CLI by ADR-015 (2026-09-12).** The per-turn,
+real-time fact-check described below is now a feature of the future Swift
+app (`DebateKit`), attaching to the CLI's existing event seam. In this repo,
+fact-checking is a post-hoc pass inside `judge`, checked against the
+recorded evidence, not "the world" — see ADR-015 §2 for what that means
+concretely. The bullets are kept as the description of the Swift-side
+feature and of the evidence-hygiene rule, which still holds.
+
+- **Fact-check** (Swift app): fast, narrow, per-claim, runs at the end of every
+  turn in real time — verifies a claim against the side's own recorded Prep
+  evidence.
+- **Judge**: slower, holistic, runs once over the complete transcript at the end.
+  In the CLI, the fact-check pass runs here too (ADR-015 §3, a second call).
+- Regex/heuristic "evidence hygiene" (citation density, specificity — as in
+  Aragora's `evidence.py`) is not a substitute for either of the above and, if
+  used at all, must be labeled "evidence hygiene," never "fact-check" — it scores
+  surface form, not truth, and a confidently fabricated citation with a
+  plausible-looking year and percentage would score *high* on it.
+
+## Hard invariant
+
+Every configured phase must produce a response from every side, or the run
+aborts with nothing written to the `output:` path (ADR-007). A silently one-sided debate producing a
+confident-looking verdict is the worst failure mode for a benchmarking tool
+(observed as a real, unflagged bug in a reviewed competitor — see ADR-001).
+
+## Hardware
+
+MLX-first. Running two debater models plus a judge/fact-checker concurrently on
+one Mac (target dev machine: M2 Pro, 32GB unified memory) is a real memory- and
+compute-contention risk, not just a capacity question — Apple Silicon's GPU and
+memory bandwidth are shared, so three "simultaneous" models likely serialize
+rather than truly parallelize. Options on the table, not yet decided: offload the
+judge/fact-checker to a remote host already running inference (`free`, via
+Ollama) — **withdrawn by B0, see below** — or load/unload models per turn rather
+than keeping all three resident.
+**Needs its own M0-style hardware probe (real peak memory + turn latency
+measurement) before any assumption here is trusted.** ("M0" is the MLXProbe
+session-M0 throwaway probe; results at `~/Projects/MLXProbe/RESULTS.md`.)
+
+### B0 findings (2026-09-11)
+
+B0 measured this on the dev machine (`RESULTS.md`). What it changes here:
+
+- **`free` is not a separate host: it's the dev machine itself.** The "offload
+  to `free`" option above offloads nothing, so it's withdrawn unless a genuinely
+  separate host is named. This isn't a minor correction. It invalidates a
+  specific option this ADR proposed, and it would have been silently
+  load-bearing for the rest of the build if nobody had tried to reach the host.
+  Two earlier reachability checks during B0 got it wrong before it was
+  confirmed.
+- **The judge never needs to be co-resident.** The judge runs once, post-hoc,
+  over the finished transcript (see "Fact-checking is separate from judging"),
+  so it can load after the debaters unload. That decision now does real
+  load-bearing work: the "three models at once" concern applies only to a
+  fact-checker that has to run live. This follows from the design; B0 didn't
+  measure it.
+- **The fact-checker hardware problem is resolved for v1 by ADR-015**, not by
+  finding a host: the fact-check pass now runs inside `judge`, post-hoc, so it
+  loads when the judge does, after the debaters have unloaded. The "no
+  designated offload target" finding stands as a fact about this machine, but
+  nothing in this repo needs one anymore. The live per-turn version that
+  would need one is the Swift app's concern (ADR-015 §1).
+- **The debater pair is doubtful and unmeasured, not ruled out.**
+  Mistral-Small-24B alone tripped B0's safety cut-off (critical memory pressure
+  alongside normal desktop use) before the pair and concurrency stages ran. A
+  quiet-machine rerun comes before anything downstream trusts the pair.
+- **Load/unload per turn looks expensive at long contexts**, but that's inferred:
+  a cold 4,000-token prompt took about 25 s on the 8B model, and a full swap cycle
+  wasn't measured.
+
+Prep and argument phases may warrant different inference engines — MLX
+generally wins decode-heavy short-turn work, but long-document Prep ingestion is
+a prefill-heavy workload where llama.cpp/Ollama have shown faster prompt-eval in
+some benchmarks. Worth benchmarking separately, not assumed to inherit the same
+engine choice.
+
+## Scope discipline
+
+Stay narrow: structured multi-turn comparison + rubric scoring + fact-checking.
+No consensus/voting logic anywhere in the design — consensus-seeking built into the
+turn loop is what ruled out `aragora-debate` and `arbgjr/multi-agent-debate` (see
+ADR-001). No formal
+verification (Z3/SymPy-style, as in `arbiter-debate`) — domain-mismatched for
+non-formalizable topics like policy debates. Resist becoming "the debate app's
+backend" — frame and build it as a general-purpose structured-comparison tool
+that the debate app happens to be the first real user of. Aragora's own
+self-documented scope creep (~25% of its codebase admitted by its own authors
+not to serve its core thesis) is the cautionary example to actively design
+against. (The source for the ~25% figure isn't recorded yet — R0's aragora health
+section documents the platform's breadth but not this number. Cite it or drop the
+number.)
+
+## Data sourcing — no scraping user-generated debate platforms
+
+Reviewed several live consumer debate platforms (DebateWise.org, DebateArt.com,
+VersyTalks.com) as competitive-landscape research — genuinely useful for
+understanding the market (all three determine a winner via crowd voting, not a
+rubric judge, which remains a real point of difference for this project) but
+**none of their content is a candidate data source.** Unlike the academic
+datasets used elsewhere in this ADR (IBM-Rank-30k, DebateSum, args.me, etc.,
+all released under explicit open licenses for reuse), these are user-generated
+content platforms whose terms of service almost certainly prohibit scraping and
+bulk reuse, with no indication any of them hold rights to sublicense user
+content for training or grounding purposes. Quality or topical relevance never
+overrides this — a dataset failing the licensing gate is disqualified
+regardless of how well it would otherwise fit. Any future dataset candidate
+gets checked for an explicit reuse license before anything else, same as every
+dataset actually adopted so far.
+
+## Attribution
+
+Two design lifts (`events.py` shape from `aragora-debate`, `Protocol` shape from
+`arbgjr/multi-agent-debate`) and a conditional third (`evidence.py`-derived
+evidence-hygiene scorer, if adopted) require attribution in a `NOTICE` file — all
+source projects are MIT.