pytest-assay 0.1.0__tar.gz

pytest_assay-0.1.0/.claude/CLAUDE.md

@@ -0,0 +1,120 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Pytest-assay is a framework for the evaluation of Pydantic AI agents. By adding the `@pytest.mark.assay` decorator to a test, you can run an assay resulting in a readout report. The assay compares the current agent responses against previously recorded baseline responses, e.g. from the main branch. The implementation is based on pytest hooks which capture `Agent.run()` responses. 
+
+## Development Commands
+
+### Environment Setup
+
+```bash
+# Install Ollama (required for local models)
+# See https://ollama.com for installation instructions
+
+# Pull the default model
+ollama pull qwen3:8b
+
+# Install dependencies
+uv sync
+```
+
+### Testing
+
+```bash
+# Run all tests
+uv run pytest
+
+# Run tests with verbose output
+uv run pytest -v
+
+# Run specific test file
+uv run pytest tests/test_utils.py
+
+# Run specific test
+uv run pytest tests/test_utils.py::test_duckduckgo_search
+
+# Run tests in parallel
+uv run pytest -n auto
+
+# Run tests with coverage report
+uv run pytest --cov=src/pytest-assay --cov-report=term-missing
+```
+
+### Code Quality
+
+```bash
+# Format code
+uv run ruff format .
+
+# Check and fix linting issues (ALWAYS run with --fix)
+uv run ruff check --fix .
+
+# Type checking (ALWAYS run after code changes)
+uv run pyright .
+```
+
+### Before Committing
+
+Run these checks:
+
+```bash
+# 1. Format code
+uv run ruff format .
+
+# 2. Check and fix linting issues
+uv run ruff check --fix .
+
+# 3. Type checking
+uv run pyright .
+
+# 4. Run tests
+uv run pytest -n auto
+```
+
+## MCP Servers
+
+This project uses Model Context Protocol (MCP) servers to extend AI capabilities. These are automatically invoked when relevant.
+
+### Context7 Documentation Server
+
+**When to use:**
+- Looking up library documentation (e.g., "How do I use pydantic-ai streaming?")
+- Checking API references for dependencies
+- Finding code examples from official docs
+- Verifying correct usage of third-party packages
+
+**Examples:**
+- "What's the latest pydantic-ai agent syntax?"
+- "Show me httpx async client examples"
+- "How do I configure pytest-asyncio?"
+
+### GitHub Repository Server
+
+**When to use:**
+- Checking open/closed issues in this repository
+- Reviewing pull requests and their status
+- Reading issue comments and discussions
+- Finding related issues or PRs
+- Understanding project history and decisions
+
+**Examples:**
+- "What are the open issues about curiosity?"
+- "Show me recent PRs related to PDF support"
+- "Are there any issues about MLX integration?"
+- "What's the status of issue #13?"
+
+### Best Practices
+
+- **Be specific:** "Check issue #15" is better than "check issues"
+- **Context first:** Read codebase before checking issues
+- **Combine sources:** Use Context7 for "how to use X" and GitHub for "what's our approach to X"
+
+## Coding Standards
+
+See the rules files for detailed coding standards:
+
+- `.claude/rules/python.md` - Python coding standards, type hints, async patterns
+- `.claude/rules/python-testing.md` - Testing conventions, markers, coverage requirements

pytest_assay-0.1.0/.claude/README.md

@@ -0,0 +1,4 @@
+If read-write file system access is configured for Claude Code via `/allowed-tools`, it is advised to run [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) in a dev container. For more details, refer to the [Dev Container](../.devcontainer/README.md) documentation.
+
+## MCP servers
+MCP servers are configured in the `./.mcp.json` file. The file cannot be moved to `.claude/mcp.json`.

pytest_assay-0.1.0/.claude/plans/custom_evaluators.plan.md

@@ -0,0 +1,79 @@
+# Custom Evaluators Design Plan
+
+## Problem
+
+A user writing `MyCustomEvaluator` would need to do:
+
+```python
+from pytest_assay.plugin import AGENT_RESPONSES_KEY, BASELINE_DATASET_KEY
+```
+
+These stash keys are **internal implementation details** -- not exported in `__all__`, and they couple any custom evaluator directly to the plugin's storage mechanism. The `Evaluator` protocol receiving a raw `pytest.Item` is the root cause: it forces evaluators to know *how* to extract data rather than just *receiving* it.
+
+## Design: Pass an `EvaluatorInput` Instead of `Item`
+
+Introduce a data class that the plugin constructs and passes to evaluators, replacing the raw `Item`:
+
+```python
+# models.py
+class EvaluatorInput(BaseModel):
+    baseline_dataset: Dataset | None
+    agent_responses: list[AgentRunResult[Any]]
+
+class Evaluator(Protocol):
+    def __call__(self, input: EvaluatorInput) -> Coroutine[Any, Any, Readout]: ...
+```
+
+The plugin's `_run_evaluation` would change from:
+
+```python
+readout = asyncio.run(evaluator(item))
+```
+
+to:
+
+```python
+eval_input = EvaluatorInput(
+    baseline_dataset=item.stash.get(BASELINE_DATASET_KEY, None),
+    agent_responses=item.stash.get(AGENT_RESPONSES_KEY, []),
+)
+readout = asyncio.run(evaluator(eval_input))
+```
+
+The stash keys stay private. Evaluators receive exactly the data they need.
+
+## What Changes
+
+| Component | Change |
+|---|---|
+| `Evaluator` protocol | `__call__(self, input: EvaluatorInput)` instead of `__call__(self, item: Item)` |
+| `_run_evaluation` in plugin.py | Build `EvaluatorInput` from stash, pass it |
+| `PairwiseEvaluator.__call__` | Accept `EvaluatorInput`, drop the stash key import |
+| `BradleyTerryEvaluator.__call__` | Accept `EvaluatorInput`, drop the stash key import |
+| Public exports | Add `EvaluatorInput`, `Evaluator`, `Readout` to `__all__` |
+
+## What a Custom Evaluator Looks Like
+
+```python
+from pytest_assay.models import Evaluator, EvaluatorInput, Readout
+
+class MyCustomEvaluator:
+    async def __call__(self, input: EvaluatorInput) -> Readout:
+        # input.baseline_dataset has the baseline cases
+        # input.agent_responses has the captured AgentRunResult objects
+        score = my_scoring_logic(input.baseline_dataset, input.agent_responses)
+        return Readout(passed=score > 0.5, details={"score": score})
+```
+
+No internal imports needed. Fully decoupled from pytest stash mechanics.
+
+## Why Not Alternatives
+
+- **Export the stash keys as public API**: Leaks an implementation detail. If you ever change how data flows (e.g. stop using stash), every custom evaluator breaks.
+- **Accessor helper functions** (`get_baseline(item)`, `get_responses(item)`): Better than raw keys, but still couples evaluators to `pytest.Item`. Harder to unit-test evaluators in isolation.
+- **The `EvaluatorInput` approach**: Evaluators become plain async functions on data. Easy to test, no pytest dependency, no internal imports. The plugin owns the extraction logic.
+
+## Open Questions
+
+1. Should `EvaluatorInput` also carry metadata (test name, assay path, mode) so evaluators can customize behavior per-test?
+2. `AgentRunResult` comes from `pydantic-ai` -- is that an acceptable public API surface, or should responses be further abstracted (e.g. to plain strings)?

pytest_assay-0.1.0/.claude/plans/remove_generator.plan.md

@@ -0,0 +1,52 @@
+# Design Analysis: `generator` in `@pytest.mark.assay` vs. External Dataset Creation
+
+## Current Design
+
+The `generator` callable is passed as a marker kwarg. The plugin calls it inside `pytest_runtest_setup` only if no serialized dataset file exists yet. The resulting `Dataset` is wrapped into an `AssayContext` and injected into the test via `item.funcargs["context"]`.
+
+---
+
+## Option A: Keep `generator` in the Marker (current)
+
+**Pros:**
+- **Declarative, self-contained test definition.** The marker is a single source of truth: evaluator *and* data source live together. You can read the decorator and understand the full assay without chasing fixtures.
+- **Lazy, conditional generation.** The plugin only calls the generator when no baseline file exists. This logic is invisible to the test author -- they don't need `if not path.exists()` boilerplate.
+- **Symmetry with `evaluator`.** Both the data source and the evaluation strategy are declared in the same place. Moving one out breaks that symmetry.
+
+**Cons:**
+- **Generator signature is constrained.** It must be a zero-argument callable returning `Dataset`. If a generator needs runtime information (e.g. fixtures, config, parametrize values), there's no clean way to pass it.
+- **Magic injection of `context`.** The test receives `context: AssayContext` as a parameter, but it's not a real pytest fixture -- it's stuffed into `funcargs` by a hook. This is surprising to pytest users who expect fixtures to be defined via `@pytest.fixture` or `conftest.py`.
+- **Harder to share datasets.** Two tests that share the same dataset must both reference the same generator function. With fixtures, you'd just depend on the same fixture name.
+
+---
+
+## Option B: Dataset Created in a Fixture / Test Body
+
+**Pros:**
+- **Standard pytest idiom.** A `@pytest.fixture` returning `AssayContext` is immediately understandable. No hidden `funcargs` injection.
+- **Full access to other fixtures.** A fixture can depend on `tmp_path`, database connections, parametrize values, or any other fixture -- something a zero-arg generator cannot do.
+- **Easier dataset sharing.** Multiple tests reuse the same fixture by name; no need to import and wire the same callable.
+
+**Cons:**
+- **Loses conditional loading.** The plugin currently skips generation when a baseline file already exists. Moving generation to a fixture means either (a) the fixture must replicate that logic, or (b) the plugin still needs a hook to intercept and replace the dataset -- defeating the purpose.
+- **Splits the assay definition.** The marker would declare the evaluator, but the data source would live elsewhere (fixture or test body). You'd need to look in two places to understand a single assay.
+- **Test body pollution.** If the dataset is built inside the test function, the test mixes setup concerns (data generation) with execution concerns (running the agent). Fixtures avoid this, but then you're back to the "two places" problem.
+
+---
+
+## Option C: Hybrid -- Fixture-Based with Marker Override
+
+The marker could accept an *optional* `generator`. If absent, the plugin looks for a `context` fixture (standard pytest resolution). If present, the marker generator takes priority.
+
+**Pros:** Best of both worlds -- simple cases stay declarative, complex cases use fixtures.
+**Cons:** Two code paths to maintain and document. Users must understand when each applies.
+
+---
+
+## Conclusion
+
+The current design is the right call for this project. The key reason: **the plugin needs control over *when* and *whether* to generate the dataset** (skip if baseline file exists, generate otherwise). That conditional logic is core to the assay lifecycle and belongs in the plugin, not in user-written fixtures. Forcing fixture authors to replicate that `if file.exists()` check would be error-prone and leak implementation details.
+
+The zero-arg constraint on the generator is acceptable because dataset generation is inherently a pure function -- it defines *what to evaluate*, not *how to run* the test. If you ever need runtime-dependent datasets (e.g. parametrized topics), the cleaner extension would be to let the generator accept an optional config dict via the marker (`generator_kwargs={"n_cases": 5}`) rather than moving to fixtures.
+
+The one thing worth improving: document that `context` is injected by the plugin, not a regular fixture. A brief note in the marker docstring or a type stub would reduce confusion.

pytest_assay-0.1.0/.claude/rules/general.md

@@ -0,0 +1,5 @@
+## Plan Mode
+
+- Make the plan extremely concise. Sacrifice grammar for the sake of being concise.
+- At the end of each plan, give me a list of unresolved questions to answer, if any. Make the questions extremely concise.
+- When in *plan mode*, you are writing the final plan as markdown file to `~/.claude/plans/` by default. When writing a plan to `~/.claude/plans/` you MUST always copy it to `./.claude/plans/` in the project root. For example, `~/.claude/plans/rosy-dancing-finch.md` should be copied to `./.claude/plans/rosy-dancing-finch.md`.

pytest_assay-0.1.0/.claude/rules/python-testing.md

@@ -0,0 +1,79 @@
+# Testing with Pytest
+
+## Testing Principles
+
+- **Reuse, Don't Replicate**: Tests should reuse as much functional code as possible. Avoid reimplementing application logic within a test. Import and call the actual functions and classes you intend to test.
+- **Mock Fundamental Processes**: When isolating code for a unit test, mock the most fundamental external interaction. For example, mock the `asyncio.create_subprocess_exec` call for a command-line tool, not a higher-level function that wraps it. This ensures you are testing your application's error handling and response parsing logic.
+- **Cover All Failure Modes**: Every test suite should cover not just the "happy path" but also all conceivable failure modes. Use `pytest.raises` to verify that your code correctly handles non-zero return codes, missing commands (`FileNotFoundError`), network errors, and other exceptional conditions.
+
+## Test Structure
+
+```python
+@pytest.mark.asyncio  # For async tests
+async def test_feature() -> None:
+    """Test description."""
+    # Arrange
+    ...
+    
+    # Act
+    result = await some_function()
+    
+    # Assert
+    assert result is not None
+```
+
+## Running Tests
+
+```bash
+# Run all tests
+uv run pytest
+
+# Run tests with verbose output
+uv run pytest -v
+
+# Run specific test file
+uv run pytest tests/test_utils.py
+
+# Run specific test
+uv run pytest tests/test_utils.py::test_fetch_content -v
+
+# Run tests in parallel
+uv run pytest -n auto
+
+# Run tests with coverage report
+uv run pytest --cov=src/pytest-assay --cov-report=term-missing
+```
+
+## Markers
+
+- `@pytest.mark.ollama` - Requires local Ollama (skipped in CI)
+
+## VCR Cassettes
+
+- Location: `tests/cassettes/`
+- Record mode: `none` (playback only by default)
+- Hostname normalization in `conftest.py` handles `host.docker.internal` → `localhost`
+- Deterministic tests: set `temperature=0.0` in `MODEL_SETTINGS`
+
+## Coverage Requirements
+
+**After writing or modifying tests**, verify coverage targets are met:
+
+1. **Read coverage targets** from `.codecov.yaml` to determine:
+   - `coverage.status.project.default.target` - minimum overall project coverage
+   - `coverage.status.project.default.threshold` - allowed coverage drop tolerance
+   - `coverage.status.patch.default.target` - minimum coverage for new/modified code
+
+2. **Run coverage report**:
+```bash
+# Full project coverage
+uv run pytest --cov=src/pytest-assay --cov-report=term-missing
+
+# Specific module coverage
+uv run pytest --cov=src/pytest-assay/MODULE_NAME --cov-report=term-missing tests/test_MODULE_NAME.py
+```
+
+3. **Verify targets are met**:
+   - New code must meet the **patch target** from `.codecov.yaml`
+   - Overall coverage must not drop below **project target minus threshold**
+   - Focus coverage on critical paths: error handling, edge cases, and main functionality

pytest_assay-0.1.0/.claude/rules/python.md

@@ -0,0 +1,305 @@
+# Python Coding Standards
+
+## Python Version
+
+- **Required:** Python 3.12 (no 3.13+ features)
+- **Check:** `requires-python = ">=3.12,<3.13"` in pyproject.toml
+- Avoid features introduced in Python 3.13
+
+## Code Style & Formatting
+
+### Ruff Configuration
+Use Ruff for formatting and linting (configured in `pyproject.toml`):
+
+```bash
+# Format code
+uv run ruff format .
+
+# Check and fix linting issues (ALWAYS run with --fix)
+uv run ruff check --fix .
+```
+
+**Important:** Always use `--fix` to automatically resolve fixable issues. Run this after any code changes.
+
+**Key rules enabled:**
+- `E`, `F` - pycodestyle, pyflakes (essential errors)
+- `I` - isort (import sorting)
+- `UP` - pyupgrade (modern Python syntax)
+- `ANN` - type annotations (required)
+- `B` - bugbear (common bugs)
+- `PL` - pylint rules
+
+**Allowed exceptions:**
+- `SIM108` - Allow if-else blocks instead of forcing ternary operators
+- `PLR2004` - Allow magic values (constants without named variables)
+- `PLR0915` - Allow long functions
+- `PLR0912` - Allow many branches
+- `PLR0913` - Allow many arguments
+
+### Line Length
+- Maximum: 150 characters (configured in ruff)
+- Prefer shorter lines when reasonable
+
+### Import Order
+```python
+# 1. Standard library
+from collections.abc import Generator
+import os
+
+# 2. Third-party packages
+import pytest
+from pydantic_ai import Agent
+
+# 3. Local imports
+from pytest_assay.config import config
+from pytest_assay.models import WebSearchQuery
+```
+
+## Type Hints
+
+### Required
+Type hints are **required** for all functions (enforced by Pyright):
+
+```python
+# Good
+def search_web(query: str, max_results: int = 5) -> list[dict[str, str]]:
+    ...
+
+async def fetch_content(url: str) -> str:
+    ...
+```
+
+### Pyright Configuration
+- `typeCheckingMode: "basic"`
+- Error-level rules:
+  - `reportOptionalMemberAccess`
+  - `reportOptionalSubscript`
+  - `reportOptionalCall`
+  - `reportGeneralTypeIssues`
+  - `reportReturnType`
+
+### Type Checking
+```bash
+# ALWAYS run after code changes
+uv run pyright .
+```
+
+**Important:** Always run pyright to catch type errors. This must pass before committing any Python code.
+
+### Modern Syntax
+Use Python 3.12+ type syntax:
+```python
+# Good (3.12+)
+def process(items: list[str]) -> dict[str, int]:
+    ...
+
+# Avoid (old style)
+from typing import List, Dict
+def process(items: List[str]) -> Dict[str, int]:
+    ...
+```
+
+## Async/Await Patterns
+
+### When to Use Async
+- Network I/O (web searches, API calls)
+- File I/O with async libraries
+- Concurrent operations
+
+```python
+# Good - concurrent operations
+async def fetch_multiple(urls: list[str]) -> list[str]:
+    async with httpx.AsyncClient() as client:
+        tasks = [client.get(url) for url in urls]
+        responses = await asyncio.gather(*tasks)
+        return [r.text for r in responses]
+```
+
+### Pydantic AI Agents
+All agents are async:
+```python
+from pydantic_ai import Agent
+
+agent = Agent(
+    model=model,
+    output_type=WebSearchQuery,
+    system_prompt=QUERY_INSTRUCTIONS,
+    retries=5,
+    instrument=True,
+)
+
+# Usage
+async with agent:
+    result = await agent.run(user_prompt="Generate query")
+    print(result.output)
+```
+
+## Dependencies
+
+### Management
+Use `uv` (not pip or poetry):
+```bash
+# Install dependencies
+uv sync
+
+# Add new dependency
+# Edit pyproject.toml manually, then:
+uv sync
+
+# Run command in venv
+uv run pytest
+```
+
+### Adding Dependencies
+1. Add to `pyproject.toml` under `[project.dependencies]`
+2. Specify minimum version: `"package>=1.2.3"`
+3. Run `uv sync` to update lock file
+4. Test that it works
+
+## Error Handling & Logging
+
+### Logging
+Use `loguru` for structured logging:
+```python
+from pytest_assay.logger import logger
+
+# Info
+logger.info("Starting web search for topic: {}", topic)
+
+# Debug (verbose)
+logger.debug("Received {} results", len(results))
+
+# Error with context
+try:
+    result = await fetch_content(url)
+except Exception as e:
+    logger.error("Failed to fetch {}: {}", url, e)
+    raise
+```
+
+### Error Handling
+```python
+# Good - explicit handling
+try:
+    result = await risky_operation()
+except ValueError as e:
+    logger.error("Invalid input: {}", e)
+    return default_value
+except httpx.HTTPError as e:
+    logger.error("Network error: {}", e)
+    raise
+
+# Avoid silent failures
+try:
+    ...
+except Exception:
+    pass  # Bad - hides errors
+```
+
+## Common Patterns
+
+### Pydantic Models
+Use for structured data:
+```python
+from pydantic import BaseModel, Field
+
+class WebSearchQuery(BaseModel):
+    query: str = Field(max_length=100)
+    aspect: str
+    rationale: str
+```
+
+### Configuration
+Access via centralized config:
+```python
+from pytest_assay.config import config
+
+# Good
+max_loops = config.max_research_loops
+model_name = config.model.value
+
+# Avoid hardcoded values
+max_loops = 5  # Bad
+```
+
+### Agent Definitions
+Pattern used in `agents.py`:
+```python
+AGENT_NAME = Agent(
+    model=model,
+    output_type=OutputType,
+    system_prompt=PROMPT_CONSTANT,
+    retries=5,
+    instrument=True,  # For logfire tracking
+)
+```
+
+## Module and Package Structure
+
+### `__init__.py` Files
+`__init__.py` files should be kept minimal. Their primary purpose is to define a package and expose its public API.
+
+- **Good**: Use for imports, `__all__`, and package-level docstrings.
+  ```python
+  # src/pytest-assay/mcp/__init__.py
+  """MCP server implementations for pytest-assay."""
+  from .server import date_server
+
+  __all__ = ["date_server"]
+  ```
+
+- **Bad**: Avoid defining functions, classes, or complex logic directly in `__init__.py`. This can lead to import side effects and makes the code harder to navigate. Move such code into separate modules (e.g., `server.py`) and import them.
+
+## Docstrings
+
+Use Google-style docstrings (not Sphinx style):
+
+### Formatting Rules
+- **No backticks**: Do not use backticks in docstrings. Write `--assay-mode` as --assay-mode, `None` as None, `True` as True, etc.
+- Google style uses plain text for parameter names, options, and values
+- For single-line docstrings, place initial and final triple quotes on the same line:
+  ```python
+  def add(a: int, b: int) -> int:
+      """Add two integers."""
+      return a + b
+  ```
+- For multi-line docstrings, place initial and final triple quotes on their own lines:
+  ```python
+  def complex_function(param1: str, param2: int = 5) -> dict[str, Any]:
+      """
+      Short one-line description.
+
+      Longer description if needed, explaining the function's purpose,
+      behavior, and any important details.
+
+      Args:
+          param1: Description of param1.
+          param2: Description of param2. Defaults to 5.
+
+      Returns:
+          Description of return value.
+
+      Raises:
+          ValueError: When param2 is negative.
+      """
+      ...
+  ```
+
+## Before Committing
+
+Run these checks:
+```bash
+# 1. Format code
+uv run ruff format .
+
+# 2. Check and fix linting issues
+uv run ruff check --fix .
+
+# 3. Type checking
+uv run pyright .
+
+# 4. Run tests
+uv run pytest -n auto
+```
+
+**Note:** The `--fix` flag automatically resolves fixable linting issues. Always verify the changes it makes are appropriate.