sonar-eval 0.3.0__tar.gz

sonar_eval-0.3.0/.gitignore

@@ -0,0 +1,17 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+venv/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+dist/
+build/
+.env
+*.local.yaml
+.DS_Store
+
+# Collector persistent queue (runtime state)
+collector/queue/*
+!collector/queue/.gitkeep

sonar_eval-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Matthew Ahmon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sonar_eval-0.3.0/PKG-INFO

@@ -0,0 +1,209 @@
+Metadata-Version: 2.5
+Name: sonar-eval
+Version: 0.3.0
+Summary: Evaluation harness SDK for agentic AI chatbots, built on sonar-tracing.
+Author-email: Matthew Ahmon <matthew.ahmon@gmail.com>
+Maintainer-email: Matthew Ahmon <matthew.ahmon@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,ai,evaluation,llm,sonar,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Requires-Dist: sonar-tracing>=0.1
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# sonar-eval
+
+Evaluation harness SDK for agentic AI chatbots, built on
+[`sonar-tracing`](https://pypi.org/project/sonar-tracing/).
+
+The harness drives your bot through simulated conversations, optionally injects
+faults into its tool calls, and grades the transcript it pulls back from
+`sonar-ingest`. You write one small adapter for your bot; the trial loop,
+fault injection, assertions, LLM grading, and pass@k / pass^k rollups are generic.
+
+```bash
+pip install sonar-eval
+```
+
+## The mental model
+
+`conversation_id` is the spine of the whole design. For each trial the harness:
+
+1. mints a `conversation_id` via `sonar_tracing.new_conversation()` — this is the
+   **trial id**, the **fault-injection scope key**, and the **join key** all at once;
+2. hands it to your `BotAdapter`, which makes the bot trace under that id (in a
+   real service, by POSTing it to an app-internal endpoint the app threads into
+   `tracing.identity(conversation_id=...)`);
+3. lets a **user simulator** (an LLM playing the human) drive turns until it emits
+   the termination signal or hits `max_turns`. The simulator sees *only* the
+   messages delivered back to it — never the bot's tools or spans;
+4. pulls the whole conversation back out of ingest by that id
+   (`GET /v1/conversations/{id}`) and reconstructs the tool-call transcript;
+5. runs programmatic **assertions** (tier 1) and an optional LLM **rubric grader**
+   (tier 2), then repeats `k` times and rolls the results up per assertion.
+
+The grader scores what actually happened — the span tree in Postgres — not
+whatever the adapter chose to return. That is why `send()` returns only delivered
+messages and the transcript comes from ingest.
+
+## What you provide
+
+Two things are project-specific: an `LLMClient` (the harness never imports a
+vendor SDK) and a `BotAdapter` for your bot.
+
+```python
+from sonar_eval import BaseBotAdapter, Delivered, LLMConfig, Message, Session, TrialContext
+
+
+class MyLLM:
+    """Wrap whatever model client you use. Return the completion text."""
+
+    async def complete(self, config: LLMConfig, system: str, messages: list[Message]) -> str:
+        ...  # call your provider with `system` + `messages`, return the text
+
+
+class MyBotAdapter(BaseBotAdapter):
+    name = "support-bot"
+
+    async def setup_trial(self, ctx: TrialContext) -> Session:
+        # Isolate/seed state for this trial and make the bot trace under
+        # ctx.conversation_id. For a real service: POST ctx.conversation_id
+        # (and, for real fidelity, serialize_faults(ctx.faults)) to your bot's
+        # internal endpoint so it wraps its turns in identity(conversation_id=...).
+        handle = await open_session(ctx.conversation_id, seed=ctx.seed.data)
+        return Session(conversation_id=ctx.conversation_id, handle=handle)
+
+    async def send(self, session: Session, user_msg: str, media=None) -> Delivered:
+        reply = await session.handle.send(user_msg)   # drive one user turn
+        return Delivered(messages=(reply,))            # only what the user sees
+
+    async def teardown_trial(self, session: Session) -> None:
+        await session.handle.close()                   # safe after a failed setup
+```
+
+## Running a scenario
+
+```python
+import asyncio
+
+from sonar_eval import (
+    Assertion, Fidelity, LLMConfig, Orchestrator, ScenarioConfig, Severity,
+    TaskConfig, no_tool_errors, tool_was_called,
+)
+
+task = TaskConfig(
+    ingest_base_url="http://localhost:4319",
+    tenant="acme",
+    user_llm=LLMConfig(model="claude-sonnet-5"),
+    grader_llm=LLMConfig(model="claude-opus-5"),
+    # project defaults to "eval-fullcontent" so the grader sees unredacted tool I/O
+)
+
+scenario = ScenarioConfig(
+    name="order-a-coffee",
+    user_instructions="You want a large oat-milk latte. Order it, then stop.",
+    k=5,
+    assertions=[
+        # capability: passes if it held on ANY of the k trials (pass@k)
+        Assertion("looked-up-menu", Severity.CAPABILITY, tool_was_called("menu_lookup")),
+        # safety-critical: passes only if it held on EVERY trial (pass^k == 1.0)
+        Assertion("no-tool-errors", Severity.SAFETY_CRITICAL, no_tool_errors()),
+    ],
+    grader_rubric="Did the bot confirm the exact order before charging? Score 0..1.",
+)
+
+orch = Orchestrator(task, MyBotAdapter(), MyLLM())
+report = asyncio.run(orch.run_scenario(scenario))
+
+print(report.passed)                       # True only if every assertion passed
+for a in report.assertions:
+    print(a.name, a.severity, a.pass_at_k, a.pass_hat_k)
+print(report.mean_grader_score)            # None unless a grader ran
+```
+
+Built-in checks: `tool_was_called(name)`, `no_tool_errors()`, `max_turns(n)`,
+`max_total_tokens(n)`. A check is any `Callable[[Transcript, ConversationLog], bool]`,
+so you can write your own over the pulled transcript.
+
+Grading is opt-in: pass a `grader_factory` to the orchestrator and set
+`grader_rubric` on the scenario.
+
+```python
+from sonar_eval import Grader
+
+orch = Orchestrator(
+    task, MyBotAdapter(), MyLLM(),
+    grader_factory=lambda s: Grader(MyLLM(), s.grader_llm, s.grader_rubric),
+    store=None,   # defaults to NullStore; use JSONFileStore("results/") to persist
+)
+```
+
+## Fault injection
+
+Faults are declared once and applied by whichever mechanism the scenario's
+fidelity selects. Rules are declarative so trials stay reproducible (pass^k needs
+determinism).
+
+```python
+from sonar_eval import Action, FaultRule, FaultSpec
+
+faults = FaultSpec(rules=(
+    # the 2nd call to payment_api raises, every trial
+    FaultRule(target="payment_api", action=Action.ERROR, when_call_index=1),
+    # a content-aware fault (mocked fidelity only): fail if it tried to overcharge
+    FaultRule(
+        target="payment_api", action=Action.ERROR,
+        when=lambda call: call.args.get("amount", 0) > 100,
+    ),
+))
+scenario = ScenarioConfig(name="payment-outage", user_instructions="...", faults=faults)
+```
+
+- **`Fidelity.MOCKED`** (default): the adapter routes its mocked dependencies
+  through a harness-owned `ProxyInterceptor` built from the spec. Fully in-process;
+  the `when` content predicate works here.
+- **`Fidelity.REAL`**: the adapter posts `serialize_faults(ctx.faults)` to your
+  app's internal endpoint and the app's own tool dispatch honours it. A `when`
+  predicate cannot cross a process boundary, so `serialize_faults()` raises on one
+  rather than silently dropping it — use `when_call_index` for real-fidelity runs.
+
+Actions: `ERROR`, `TIMEOUT`, `REPLACE_OUTPUT` (returns `payload`), `LATENCY`
+(sleeps `payload` seconds).
+
+## Metrics
+
+Results are rolled up **per assertion**, not just per scenario, so a scenario can
+legitimately pass its capability bar while failing a safety gate:
+
+- **capability → pass@k** — passes if the assertion held on at least one of `k` trials;
+- **safety-critical → pass^k** — passes only if it held on every trial.
+
+`ScenarioReport.passed` is true only when every assertion passes under its own rule.
+Records are written through a pluggable `ResultStore` (`JSONFileStore` /
+`NullStore` ship in the box).
+
+## App-side integration
+
+For the harness to drive your bot, the bot must accept a `conversation_id` per
+conversation and thread it straight into `identity(conversation_id=...)` — and, to
+be *drivable*, expose a way for the simulator to hand it that id. See the
+"Grouping runs into a conversation" section of
+[`docs/IMPLEMENTATION_STRANDS.md`](../../docs/IMPLEMENTATION_STRANDS.md) /
+[`docs/IMPLEMENTATION_LANGCHAIN.md`](../../docs/IMPLEMENTATION_LANGCHAIN.md), and
+[`DESIGN.md` §10](../../DESIGN.md) for the full rationale.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

sonar_eval-0.3.0/README.md

@@ -0,0 +1,184 @@
+# sonar-eval
+
+Evaluation harness SDK for agentic AI chatbots, built on
+[`sonar-tracing`](https://pypi.org/project/sonar-tracing/).
+
+The harness drives your bot through simulated conversations, optionally injects
+faults into its tool calls, and grades the transcript it pulls back from
+`sonar-ingest`. You write one small adapter for your bot; the trial loop,
+fault injection, assertions, LLM grading, and pass@k / pass^k rollups are generic.
+
+```bash
+pip install sonar-eval
+```
+
+## The mental model
+
+`conversation_id` is the spine of the whole design. For each trial the harness:
+
+1. mints a `conversation_id` via `sonar_tracing.new_conversation()` — this is the
+   **trial id**, the **fault-injection scope key**, and the **join key** all at once;
+2. hands it to your `BotAdapter`, which makes the bot trace under that id (in a
+   real service, by POSTing it to an app-internal endpoint the app threads into
+   `tracing.identity(conversation_id=...)`);
+3. lets a **user simulator** (an LLM playing the human) drive turns until it emits
+   the termination signal or hits `max_turns`. The simulator sees *only* the
+   messages delivered back to it — never the bot's tools or spans;
+4. pulls the whole conversation back out of ingest by that id
+   (`GET /v1/conversations/{id}`) and reconstructs the tool-call transcript;
+5. runs programmatic **assertions** (tier 1) and an optional LLM **rubric grader**
+   (tier 2), then repeats `k` times and rolls the results up per assertion.
+
+The grader scores what actually happened — the span tree in Postgres — not
+whatever the adapter chose to return. That is why `send()` returns only delivered
+messages and the transcript comes from ingest.
+
+## What you provide
+
+Two things are project-specific: an `LLMClient` (the harness never imports a
+vendor SDK) and a `BotAdapter` for your bot.
+
+```python
+from sonar_eval import BaseBotAdapter, Delivered, LLMConfig, Message, Session, TrialContext
+
+
+class MyLLM:
+    """Wrap whatever model client you use. Return the completion text."""
+
+    async def complete(self, config: LLMConfig, system: str, messages: list[Message]) -> str:
+        ...  # call your provider with `system` + `messages`, return the text
+
+
+class MyBotAdapter(BaseBotAdapter):
+    name = "support-bot"
+
+    async def setup_trial(self, ctx: TrialContext) -> Session:
+        # Isolate/seed state for this trial and make the bot trace under
+        # ctx.conversation_id. For a real service: POST ctx.conversation_id
+        # (and, for real fidelity, serialize_faults(ctx.faults)) to your bot's
+        # internal endpoint so it wraps its turns in identity(conversation_id=...).
+        handle = await open_session(ctx.conversation_id, seed=ctx.seed.data)
+        return Session(conversation_id=ctx.conversation_id, handle=handle)
+
+    async def send(self, session: Session, user_msg: str, media=None) -> Delivered:
+        reply = await session.handle.send(user_msg)   # drive one user turn
+        return Delivered(messages=(reply,))            # only what the user sees
+
+    async def teardown_trial(self, session: Session) -> None:
+        await session.handle.close()                   # safe after a failed setup
+```
+
+## Running a scenario
+
+```python
+import asyncio
+
+from sonar_eval import (
+    Assertion, Fidelity, LLMConfig, Orchestrator, ScenarioConfig, Severity,
+    TaskConfig, no_tool_errors, tool_was_called,
+)
+
+task = TaskConfig(
+    ingest_base_url="http://localhost:4319",
+    tenant="acme",
+    user_llm=LLMConfig(model="claude-sonnet-5"),
+    grader_llm=LLMConfig(model="claude-opus-5"),
+    # project defaults to "eval-fullcontent" so the grader sees unredacted tool I/O
+)
+
+scenario = ScenarioConfig(
+    name="order-a-coffee",
+    user_instructions="You want a large oat-milk latte. Order it, then stop.",
+    k=5,
+    assertions=[
+        # capability: passes if it held on ANY of the k trials (pass@k)
+        Assertion("looked-up-menu", Severity.CAPABILITY, tool_was_called("menu_lookup")),
+        # safety-critical: passes only if it held on EVERY trial (pass^k == 1.0)
+        Assertion("no-tool-errors", Severity.SAFETY_CRITICAL, no_tool_errors()),
+    ],
+    grader_rubric="Did the bot confirm the exact order before charging? Score 0..1.",
+)
+
+orch = Orchestrator(task, MyBotAdapter(), MyLLM())
+report = asyncio.run(orch.run_scenario(scenario))
+
+print(report.passed)                       # True only if every assertion passed
+for a in report.assertions:
+    print(a.name, a.severity, a.pass_at_k, a.pass_hat_k)
+print(report.mean_grader_score)            # None unless a grader ran
+```
+
+Built-in checks: `tool_was_called(name)`, `no_tool_errors()`, `max_turns(n)`,
+`max_total_tokens(n)`. A check is any `Callable[[Transcript, ConversationLog], bool]`,
+so you can write your own over the pulled transcript.
+
+Grading is opt-in: pass a `grader_factory` to the orchestrator and set
+`grader_rubric` on the scenario.
+
+```python
+from sonar_eval import Grader
+
+orch = Orchestrator(
+    task, MyBotAdapter(), MyLLM(),
+    grader_factory=lambda s: Grader(MyLLM(), s.grader_llm, s.grader_rubric),
+    store=None,   # defaults to NullStore; use JSONFileStore("results/") to persist
+)
+```
+
+## Fault injection
+
+Faults are declared once and applied by whichever mechanism the scenario's
+fidelity selects. Rules are declarative so trials stay reproducible (pass^k needs
+determinism).
+
+```python
+from sonar_eval import Action, FaultRule, FaultSpec
+
+faults = FaultSpec(rules=(
+    # the 2nd call to payment_api raises, every trial
+    FaultRule(target="payment_api", action=Action.ERROR, when_call_index=1),
+    # a content-aware fault (mocked fidelity only): fail if it tried to overcharge
+    FaultRule(
+        target="payment_api", action=Action.ERROR,
+        when=lambda call: call.args.get("amount", 0) > 100,
+    ),
+))
+scenario = ScenarioConfig(name="payment-outage", user_instructions="...", faults=faults)
+```
+
+- **`Fidelity.MOCKED`** (default): the adapter routes its mocked dependencies
+  through a harness-owned `ProxyInterceptor` built from the spec. Fully in-process;
+  the `when` content predicate works here.
+- **`Fidelity.REAL`**: the adapter posts `serialize_faults(ctx.faults)` to your
+  app's internal endpoint and the app's own tool dispatch honours it. A `when`
+  predicate cannot cross a process boundary, so `serialize_faults()` raises on one
+  rather than silently dropping it — use `when_call_index` for real-fidelity runs.
+
+Actions: `ERROR`, `TIMEOUT`, `REPLACE_OUTPUT` (returns `payload`), `LATENCY`
+(sleeps `payload` seconds).
+
+## Metrics
+
+Results are rolled up **per assertion**, not just per scenario, so a scenario can
+legitimately pass its capability bar while failing a safety gate:
+
+- **capability → pass@k** — passes if the assertion held on at least one of `k` trials;
+- **safety-critical → pass^k** — passes only if it held on every trial.
+
+`ScenarioReport.passed` is true only when every assertion passes under its own rule.
+Records are written through a pluggable `ResultStore` (`JSONFileStore` /
+`NullStore` ship in the box).
+
+## App-side integration
+
+For the harness to drive your bot, the bot must accept a `conversation_id` per
+conversation and thread it straight into `identity(conversation_id=...)` — and, to
+be *drivable*, expose a way for the simulator to hand it that id. See the
+"Grouping runs into a conversation" section of
+[`docs/IMPLEMENTATION_STRANDS.md`](../../docs/IMPLEMENTATION_STRANDS.md) /
+[`docs/IMPLEMENTATION_LANGCHAIN.md`](../../docs/IMPLEMENTATION_LANGCHAIN.md), and
+[`DESIGN.md` §10](../../DESIGN.md) for the full rationale.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

sonar_eval-0.3.0/pyproject.toml

@@ -0,0 +1,38 @@
+[project]
+name = "sonar-eval"
+dynamic = ["version"]
+description = "Evaluation harness SDK for agentic AI chatbots, built on sonar-tracing."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+authors = [{ name = "Matthew Ahmon", email = "matthew.ahmon@gmail.com" }]
+maintainers = [{ name = "Matthew Ahmon", email = "matthew.ahmon@gmail.com" }]
+keywords = ["evaluation", "ai", "agents", "llm", "testing", "sonar"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Topic :: Software Development :: Testing",
+    "Typing :: Typed",
+]
+dependencies = [
+    "sonar-tracing>=0.1",
+    "httpx>=0.27",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "pytest-asyncio>=0.24"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+path = "src/sonar_eval/version.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/sonar_eval"]

sonar_eval-0.3.0/src/sonar_eval/__init__.py

@@ -0,0 +1,101 @@
+"""sonar-eval - evaluation harness SDK for agentic AI chatbots.
+
+Built on sonar-tracing: the harness drives an agent through simulated conversations,
+injects faults into its tool calls, and grades the transcript it pulls back from
+sonar-ingest by conversation_id.
+
+    from sonar_eval import Orchestrator, TaskConfig, ScenarioConfig
+"""
+
+from .adapter import (
+    BaseBotAdapter,
+    BotAdapter,
+    Delivered,
+    Seed,
+    Session,
+    TrialContext,
+    load_adapter,
+    validate_bot_adapter,
+)
+from .assertions import (
+    Assertion,
+    AssertionEngine,
+    AssertionResult,
+    Severity,
+    max_total_tokens,
+    max_turns,
+    no_tool_errors,
+    tool_was_called,
+)
+from .config import Fidelity, Persona, ScenarioConfig, TaskConfig
+from .grader import Grader, GraderResult
+from .interceptor import (
+    Action,
+    FaultRule,
+    FaultSpec,
+    InjectedFault,
+    InterceptDecision,
+    ProxyInterceptor,
+    ToolCall,
+    ToolInterceptor,
+    serialize_faults,
+)
+from .llm import LLMClient, LLMConfig, Message
+from .metrics import AssertionMetric, ScenarioReport, summarize
+from .orchestrator import Orchestrator, TrialResult
+from .simulator import ConversationLog, Turn, UserSimulator
+from .store import JSONFileStore, NullStore, ResultStore
+from .transcript import IngestClient, ToolInvocation, Transcript
+from .version import __version__
+
+__all__ = [
+    "Action",
+    "Assertion",
+    "AssertionEngine",
+    "AssertionMetric",
+    "AssertionResult",
+    "BaseBotAdapter",
+    "BotAdapter",
+    "ConversationLog",
+    "Delivered",
+    "FaultRule",
+    "FaultSpec",
+    "Fidelity",
+    "Grader",
+    "GraderResult",
+    "IngestClient",
+    "InjectedFault",
+    "InterceptDecision",
+    "JSONFileStore",
+    "LLMClient",
+    "LLMConfig",
+    "Message",
+    "NullStore",
+    "Orchestrator",
+    "Persona",
+    "ProxyInterceptor",
+    "ResultStore",
+    "ScenarioConfig",
+    "ScenarioReport",
+    "Seed",
+    "Session",
+    "Severity",
+    "TaskConfig",
+    "ToolCall",
+    "ToolInterceptor",
+    "ToolInvocation",
+    "Transcript",
+    "TrialContext",
+    "TrialResult",
+    "Turn",
+    "UserSimulator",
+    "__version__",
+    "load_adapter",
+    "max_total_tokens",
+    "max_turns",
+    "no_tool_errors",
+    "serialize_faults",
+    "summarize",
+    "tool_was_called",
+    "validate_bot_adapter",
+]

sonar_eval-0.3.0/src/sonar_eval/adapter.py

@@ -0,0 +1,122 @@
+"""The bot adapter contract.
+
+One thin interface the developer implements once per bot. It absorbs the stack
+differences (auth, DB, transport) so the rest of the harness stays generic, and it is
+where per-trial isolation and seeding happen.
+
+Two things are threaded through every trial:
+
+  conversation_id  minted by the harness via ``sonar_tracing.new_conversation()``. The
+                   adapter must make the bot stamp it on its spans - for a service like
+                   Cafeina that means POSTing it to an app-internal endpoint so the app
+                   wraps its turns in ``tracing.identity(conversation_id=...)``. It is the
+                   join key the grader later uses to pull the transcript back from ingest.
+  faults           a FaultSpec. In mocked fidelity the adapter routes its mocks through a
+                   ProxyInterceptor built from it; in real fidelity it posts
+                   ``serialize_faults(faults)`` to the app alongside conversation_id.
+
+``send()`` returns only what a human user would see. The tool-call transcript is NOT
+returned here - it is pulled from sonar-ingest by conversation_id, which keeps the user
+simulator honest (it can only ever see delivered messages).
+"""
+
+from __future__ import annotations
+
+import importlib
+from dataclasses import dataclass, field
+from typing import Any, Protocol
+
+from .config import Fidelity
+from .interceptor import FaultSpec
+
+
+@dataclass(frozen=True)
+class Seed:
+    """Project-defined seed state: authorizations, DB rows, stored data.
+
+    Opaque to the harness; the adapter is the only thing that interprets ``data``.
+    """
+
+    data: dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class Session:
+    """Opaque per-trial handle the adapter owns. Carries the join key for convenience."""
+
+    conversation_id: str
+    handle: Any = None
+
+
+@dataclass(frozen=True)
+class Delivered:
+    """Exactly what a human user would see for one turn."""
+
+    messages: tuple[str, ...] = ()
+    media: tuple[Any, ...] = ()
+
+
+@dataclass(frozen=True)
+class TrialContext:
+    conversation_id: str
+    seed: Seed
+    faults: FaultSpec
+    fidelity: Fidelity
+
+
+class BotAdapter(Protocol):
+    name: str
+
+    async def setup_trial(self, ctx: TrialContext) -> Session:
+        """Isolate + seed state for one trial, wire fault injection, return a handle."""
+
+    async def send(self, session: Session, user_msg: str, media: Any = None) -> Delivered:
+        """Deliver one user turn; return only what the user would see back."""
+
+    async def teardown_trial(self, session: Session) -> None:
+        """Reset in lockstep. Must be safe to call after a failed setup."""
+
+
+class BaseBotAdapter:
+    """Optional convenience base so a minimal adapter is a few lines."""
+
+    name: str = "unnamed"
+
+    async def setup_trial(self, ctx: TrialContext) -> Session:
+        return Session(conversation_id=ctx.conversation_id)
+
+    async def send(self, session: Session, user_msg: str, media: Any = None) -> Delivered:
+        return Delivered()
+
+    async def teardown_trial(self, session: Session) -> None:
+        return None
+
+
+_REQUIRED_METHODS = ("setup_trial", "send", "teardown_trial")
+
+
+def validate_bot_adapter(candidate: Any) -> None:
+    """Structural check with a useful message.
+
+    ``runtime_checkable`` Protocols cannot verify the ``name`` attribute, and a missing
+    async method would otherwise surface deep inside the trial loop.
+    """
+    if not isinstance(getattr(candidate, "name", None), str):
+        raise TypeError(f"{candidate!r} is missing a string `name` attribute")
+    not_callable = [m for m in _REQUIRED_METHODS if not callable(getattr(candidate, m, None))]
+    if not_callable:
+        raise TypeError(f"{candidate!r} is missing methods: {not_callable}")
+
+
+def load_adapter(path: str) -> BotAdapter:
+    """Load an adapter from a ``module.path:attribute`` string and validate it.
+
+    The attribute may be an instance or a zero-arg class/factory.
+    """
+    module_path, _, attr = path.partition(":")
+    if not attr:
+        raise ValueError(f"adapter path must be 'module:attr', got {path!r}")
+    obj = getattr(importlib.import_module(module_path), attr)
+    candidate = obj() if isinstance(obj, type) else obj
+    validate_bot_adapter(candidate)
+    return candidate