signaltest 0.1.0__tar.gz

signaltest-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,25 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync --extra dev
+      - run: uv run ruff check src tests examples
+      - run: uv run ruff format --check src tests examples
+      - run: uv run mypy
+        if: matrix.python-version == '3.12'
+      - run: uv run coverage run -m pytest -q
+      - run: uv run coverage report

signaltest-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,19 @@
+name: release
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build
+      - uses: pypa/gh-action-pypi-publish@release/v1

signaltest-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.ruff_cache/
+.coverage
+.env
+.env.local
+uv.lock

signaltest-0.1.0/.pre-commit-config.yaml

@@ -0,0 +1,14 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-toml
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.20
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format

signaltest-0.1.0/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## Unreleased
+
+## 0.1.0 - 2026-06-30
+
+### Added
+- Statistical gate: permutation and Fisher significance, bootstrap effect-size
+  interval, Benjamini-Hochberg correction, and a decision that blocks only on a
+  significant regression past a minimum effect size.
+- Underpowered detection so cases with too few samples are flagged, not passed.
+- Metrics: exact match, contains, numeric (configurable polarity), trajectory
+  match, and an LLM-judge metric that wraps any scoring callable.
+- `Metric` protocol so custom metrics need only `name`, `kind`, `polarity`, and
+  `score`.
+- Tool-trajectory model, match score, and a git-style diff renderer.
+- Baseline JSON store with cold-start record-only and corrupt-file detection.
+- Model versioning: baselines re-record on a model change instead of reporting it
+  as a regression.
+- pytest plugin and `assert_no_regression` for single cases.
+- `run_suite` with suite-level correction, plus a text report and CI exit code.
+- Reports show the measured effect size and p-value on every case.
+- `signaltest` CLI for inspecting baselines.
+- Type hints across the package, checked with mypy `strict`.
+- Offline demos (a minimal case and a tool-using agent) and an end-to-end test
+  that run with no API key.

signaltest-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing
+
+Thanks for your interest in signaltest.
+
+## Setup
+
+```sh
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev]"
+pre-commit install   # optional: run the linters on every commit
+```
+
+## Before opening a PR
+
+- Add a test for anything you change.
+- Run the checks (the same ones CI runs):
+
+```sh
+ruff check src tests examples
+ruff format --check src tests examples
+mypy
+coverage run -m pytest && coverage report
+```
+
+- Keep changes small and focused — one idea per PR.
+- Match the existing style: simple, direct code, few comments.
+
+## Reporting bugs
+
+Open an issue with a minimal reproduction: the metric, the inputs, and what you
+expected versus what happened.

signaltest-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hatim
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

signaltest-0.1.0/PKG-INFO

@@ -0,0 +1,277 @@
+Metadata-Version: 2.4
+Name: signaltest
+Version: 0.1.0
+Summary: Flake-proof regression testing for LLM agents
+Project-URL: Homepage, https://github.com/Falcon305/signaltest
+Author: Hatim
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: scipy>=1.11
+Provides-Extra: dev
+Requires-Dist: coverage>=7.0; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pre-commit>=3.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# signaltest
+
+[![ci](https://github.com/Falcon305/signaltest/actions/workflows/ci.yml/badge.svg)](https://github.com/Falcon305/signaltest/actions/workflows/ci.yml)
+
+Regression tests for LLM agents that don't fail your CI on noise.
+
+LLMs are non-deterministic, so naive eval checks flake: a score drifts a point on
+randomness, CI goes red, the team stops trusting it, and the check gets deleted.
+signaltest runs each case several times and blocks a PR only when a regression is
+**statistically real and large enough to matter** — then shows a diff of what
+actually changed in the agent's run.
+
+Local-first. No account, no service, no data leaves your repo.
+
+Status: v0.1.0.
+
+## Contents
+
+- [Why](#why)
+- [Install](#install)
+- [Quick start](#quick-start)
+- [Testing a whole suite](#testing-a-whole-suite)
+- [Metrics](#metrics)
+- [How it works](#how-it-works)
+- [Configuration](#configuration)
+- [Baselines](#baselines)
+- [Using it in CI](#using-it-in-ci)
+- [CLI](#cli)
+- [Development](#development)
+- [FAQ](#faq)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Why
+
+Most eval tools score an agent once and compare against a fixed threshold. With a
+stochastic model, that threshold flakes: the same prompt scores 0.84 one run and
+0.81 the next. CI fails on the bad draw, people stop believing it, and the safety
+net is gone.
+
+signaltest treats the score as a distribution, not a number. It samples the agent
+`n` times for the candidate, compares against `n` recorded baseline samples, and
+only fails when the difference is **statistically significant** *and* clears a
+**minimum effect size**. Noise stays green. Real regressions go red.
+
+## Install
+
+```sh
+pip install signaltest
+```
+
+Or with [uv](https://docs.astral.sh/uv/):
+
+```sh
+uv pip install signaltest   # into the active environment
+uv add signaltest           # into a uv-managed project
+```
+
+## Quick start
+
+Write a normal pytest test. Give signaltest a way to run your agent, the expected
+output, and a metric.
+
+```python
+from signaltest import Case, assert_no_regression, ExactMatch
+
+
+def test_math_agent():
+    case = Case(
+        case_id="math_qa",
+        run=lambda: my_agent("what is 2 + 2?"),
+        expected="4",
+        metric=ExactMatch(),
+    )
+    assert_no_regression(case, "baselines/math_agent.json", n=10)
+```
+
+The first run records a baseline (committed as JSON in your repo). Later runs
+compare against it and fail the test only on a real regression.
+
+## Testing a whole suite
+
+`run_suite` runs many cases and applies a multiple-comparison correction across
+them, so a suite of 50 cases doesn't go red just because one flaked.
+
+```python
+from signaltest import Case, run_suite, format_report, exit_code, ExactMatch
+
+cases = [
+    Case("math", run=lambda: my_agent("2 + 2?"), expected="4", metric=ExactMatch()),
+    Case("geo", run=lambda: my_agent("capital of France?"), expected="Paris", metric=ExactMatch()),
+]
+
+results = run_suite(cases, "baselines/agent.json", n=10)
+print(format_report(results))
+raise SystemExit(exit_code(results))
+```
+
+`format_report` prints a per-case summary; `exit_code` returns `1` if any case
+regressed, `0` otherwise — drop it straight into a CI step.
+
+A failing case reports the measured effect size and p-value, so you see *how
+big* the regression is, not just that one happened:
+
+```
+PASS          geo: no significant regression
+FAIL          math: significant regression past the effect floor (effect=-0.180, p=0.004)
+1 passed, 1 failed, 0 inconclusive
+```
+
+## Metrics
+
+A metric declares its `kind` (numeric or boolean, which picks the significance
+test) and its `polarity` (is higher or lower better).
+
+| Metric | Kind | Polarity | Scores |
+|--------|------|----------|--------|
+| `ExactMatch()` | boolean | higher better | `output == expected` |
+| `Contains()` | boolean | higher better | `expected in output` |
+| `Numeric(name, polarity)` | numeric | configurable | the raw value (latency, cost, judge score) |
+| `TrajectoryMatch(ignore_keys=...)` | numeric | higher better | fraction of matching agent tool-calls |
+
+`Numeric` with `polarity="lower_better"` is how you gate latency or cost — a real
+*increase* becomes the regression.
+
+```python
+from signaltest import Numeric
+from signaltest.metrics.base import LOWER_BETTER
+
+latency = Numeric(name="latency_ms", polarity=LOWER_BETTER)
+```
+
+`TrajectoryMatch` compares the agent's tool-call path against a reference path and
+ignores volatile keys (timestamps, ids):
+
+```python
+from signaltest import TrajectoryMatch, Step
+
+expected_path = [Step("search", {"q": "weather"}), Step("answer", {})]
+metric = TrajectoryMatch(ignore_keys=("request_id",))
+```
+
+## How it works
+
+```
+candidate runs n times ─┐
+                        ├─> significance test ─┐
+stored baseline samples ┘                      ├─> block only if
+                                               │   significant AND
+                          effect size ─────────┘   past the floor
+```
+
+- **Significance** — a permutation test for numeric metrics, Fisher's exact test
+  for boolean metrics. Both are seeded, so the same inputs always give the same
+  result. The gate that kills flakiness is not itself flaky.
+- **Effect floor** — a regression must also clear a minimum effect size, so a
+  statistically significant but meaningless 0.1% drift never blocks the build.
+- **Multiple comparisons** — across a suite, p-values are adjusted with the
+  Benjamini-Hochberg procedure, so flakiness doesn't reappear at the suite level.
+- **Power** — cases with too few samples to detect a real change are flagged
+  `inconclusive`, never passed silently.
+- **Model versioning** — a baseline records the model it was captured under. If you
+  pass a new `model=` and it differs, the baseline is re-recorded instead of
+  reported as a regression, so a provider model swap can't masquerade as one.
+
+## Configuration
+
+Every `assert_no_regression` / `check_case` / `run_suite` call accepts:
+
+| Argument | Default | Meaning |
+|----------|---------|---------|
+| `n` | `10` | samples per run (boolean metrics usually want more) |
+| `alpha` | `0.05` | significance threshold |
+| `min_effect` | `0.03` numeric / `0.10` boolean | minimum effect size to count |
+| `min_valid` | `2` | fewer valid samples than this → `inconclusive` |
+| `model` | `None` | model id recorded with the baseline |
+
+## Baselines
+
+A baseline is a JSON file committed to your repo. Each entry is keyed by
+`case_id::metric_name` and stores the recorded scores and the model.
+
+- **Cold start** — the first run records the baseline and passes.
+- **Updating** — to accept a new baseline on purpose, delete the case's entry and
+  re-run, or edit the JSON. The change is a reviewable diff in the same PR.
+- **Inspecting** — use the CLI (below).
+
+## Using it in CI
+
+Because cases are plain pytest tests, your existing `pytest` step gates them:
+
+```yaml
+- run: pip install -e ".[dev]"
+- run: pytest
+```
+
+A failed case fails the build. Baselines live in the repo, so CI needs no secrets
+and nothing leaves your infrastructure.
+
+## CLI
+
+```sh
+signaltest version
+signaltest baselines baselines/agent.json   # list recorded cases
+signaltest show baselines/agent.json math::exact_match
+```
+
+## Development
+
+```sh
+git clone https://github.com/Falcon305/signaltest
+cd signaltest
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev]"
+pytest
+ruff check src tests examples
+```
+
+With uv the setup is a single command (it creates the environment for you):
+
+```sh
+uv sync --extra dev
+uv run pytest
+```
+
+Try the offline examples (cached responses, no API key):
+
+```sh
+python examples/demo.py        # smallest possible case
+python examples/tool_agent.py  # tool-using agent: trajectory + answer checks
+```
+
+See [docs/architecture.md](docs/architecture.md) for how the pieces fit together
+and how to add your own metric.
+
+## FAQ
+
+**Does it call my LLM?** Only through the `run` function you provide. signaltest
+never talks to a provider itself.
+
+**How many samples do I need?** `n=10` is a sane default for numeric metrics.
+Boolean metrics resolve in coarser steps, so they need more — bump `n` and watch
+for `inconclusive`, which means the test can't yet detect a change of the size you
+care about.
+
+**Why did a case come back `inconclusive`?** Too few valid samples to be
+trustworthy. Increase `n`, or fix whatever made runs error out.
+
+**Does my data leave my machine?** No. Baselines are local JSON; there is no
+service.
+
+## Contributing
+
+Issues and pull requests are welcome. Keep changes small and focused, and add a
+test for anything you change. See `CONTRIBUTING.md`.
+
+## License
+
+MIT

signaltest-0.1.0/README.md

@@ -0,0 +1,259 @@
+# signaltest
+
+[![ci](https://github.com/Falcon305/signaltest/actions/workflows/ci.yml/badge.svg)](https://github.com/Falcon305/signaltest/actions/workflows/ci.yml)
+
+Regression tests for LLM agents that don't fail your CI on noise.
+
+LLMs are non-deterministic, so naive eval checks flake: a score drifts a point on
+randomness, CI goes red, the team stops trusting it, and the check gets deleted.
+signaltest runs each case several times and blocks a PR only when a regression is
+**statistically real and large enough to matter** — then shows a diff of what
+actually changed in the agent's run.
+
+Local-first. No account, no service, no data leaves your repo.
+
+Status: v0.1.0.
+
+## Contents
+
+- [Why](#why)
+- [Install](#install)
+- [Quick start](#quick-start)
+- [Testing a whole suite](#testing-a-whole-suite)
+- [Metrics](#metrics)
+- [How it works](#how-it-works)
+- [Configuration](#configuration)
+- [Baselines](#baselines)
+- [Using it in CI](#using-it-in-ci)
+- [CLI](#cli)
+- [Development](#development)
+- [FAQ](#faq)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Why
+
+Most eval tools score an agent once and compare against a fixed threshold. With a
+stochastic model, that threshold flakes: the same prompt scores 0.84 one run and
+0.81 the next. CI fails on the bad draw, people stop believing it, and the safety
+net is gone.
+
+signaltest treats the score as a distribution, not a number. It samples the agent
+`n` times for the candidate, compares against `n` recorded baseline samples, and
+only fails when the difference is **statistically significant** *and* clears a
+**minimum effect size**. Noise stays green. Real regressions go red.
+
+## Install
+
+```sh
+pip install signaltest
+```
+
+Or with [uv](https://docs.astral.sh/uv/):
+
+```sh
+uv pip install signaltest   # into the active environment
+uv add signaltest           # into a uv-managed project
+```
+
+## Quick start
+
+Write a normal pytest test. Give signaltest a way to run your agent, the expected
+output, and a metric.
+
+```python
+from signaltest import Case, assert_no_regression, ExactMatch
+
+
+def test_math_agent():
+    case = Case(
+        case_id="math_qa",
+        run=lambda: my_agent("what is 2 + 2?"),
+        expected="4",
+        metric=ExactMatch(),
+    )
+    assert_no_regression(case, "baselines/math_agent.json", n=10)
+```
+
+The first run records a baseline (committed as JSON in your repo). Later runs
+compare against it and fail the test only on a real regression.
+
+## Testing a whole suite
+
+`run_suite` runs many cases and applies a multiple-comparison correction across
+them, so a suite of 50 cases doesn't go red just because one flaked.
+
+```python
+from signaltest import Case, run_suite, format_report, exit_code, ExactMatch
+
+cases = [
+    Case("math", run=lambda: my_agent("2 + 2?"), expected="4", metric=ExactMatch()),
+    Case("geo", run=lambda: my_agent("capital of France?"), expected="Paris", metric=ExactMatch()),
+]
+
+results = run_suite(cases, "baselines/agent.json", n=10)
+print(format_report(results))
+raise SystemExit(exit_code(results))
+```
+
+`format_report` prints a per-case summary; `exit_code` returns `1` if any case
+regressed, `0` otherwise — drop it straight into a CI step.
+
+A failing case reports the measured effect size and p-value, so you see *how
+big* the regression is, not just that one happened:
+
+```
+PASS          geo: no significant regression
+FAIL          math: significant regression past the effect floor (effect=-0.180, p=0.004)
+1 passed, 1 failed, 0 inconclusive
+```
+
+## Metrics
+
+A metric declares its `kind` (numeric or boolean, which picks the significance
+test) and its `polarity` (is higher or lower better).
+
+| Metric | Kind | Polarity | Scores |
+|--------|------|----------|--------|
+| `ExactMatch()` | boolean | higher better | `output == expected` |
+| `Contains()` | boolean | higher better | `expected in output` |
+| `Numeric(name, polarity)` | numeric | configurable | the raw value (latency, cost, judge score) |
+| `TrajectoryMatch(ignore_keys=...)` | numeric | higher better | fraction of matching agent tool-calls |
+
+`Numeric` with `polarity="lower_better"` is how you gate latency or cost — a real
+*increase* becomes the regression.
+
+```python
+from signaltest import Numeric
+from signaltest.metrics.base import LOWER_BETTER
+
+latency = Numeric(name="latency_ms", polarity=LOWER_BETTER)
+```
+
+`TrajectoryMatch` compares the agent's tool-call path against a reference path and
+ignores volatile keys (timestamps, ids):
+
+```python
+from signaltest import TrajectoryMatch, Step
+
+expected_path = [Step("search", {"q": "weather"}), Step("answer", {})]
+metric = TrajectoryMatch(ignore_keys=("request_id",))
+```
+
+## How it works
+
+```
+candidate runs n times ─┐
+                        ├─> significance test ─┐
+stored baseline samples ┘                      ├─> block only if
+                                               │   significant AND
+                          effect size ─────────┘   past the floor
+```
+
+- **Significance** — a permutation test for numeric metrics, Fisher's exact test
+  for boolean metrics. Both are seeded, so the same inputs always give the same
+  result. The gate that kills flakiness is not itself flaky.
+- **Effect floor** — a regression must also clear a minimum effect size, so a
+  statistically significant but meaningless 0.1% drift never blocks the build.
+- **Multiple comparisons** — across a suite, p-values are adjusted with the
+  Benjamini-Hochberg procedure, so flakiness doesn't reappear at the suite level.
+- **Power** — cases with too few samples to detect a real change are flagged
+  `inconclusive`, never passed silently.
+- **Model versioning** — a baseline records the model it was captured under. If you
+  pass a new `model=` and it differs, the baseline is re-recorded instead of
+  reported as a regression, so a provider model swap can't masquerade as one.
+
+## Configuration
+
+Every `assert_no_regression` / `check_case` / `run_suite` call accepts:
+
+| Argument | Default | Meaning |
+|----------|---------|---------|
+| `n` | `10` | samples per run (boolean metrics usually want more) |
+| `alpha` | `0.05` | significance threshold |
+| `min_effect` | `0.03` numeric / `0.10` boolean | minimum effect size to count |
+| `min_valid` | `2` | fewer valid samples than this → `inconclusive` |
+| `model` | `None` | model id recorded with the baseline |
+
+## Baselines
+
+A baseline is a JSON file committed to your repo. Each entry is keyed by
+`case_id::metric_name` and stores the recorded scores and the model.
+
+- **Cold start** — the first run records the baseline and passes.
+- **Updating** — to accept a new baseline on purpose, delete the case's entry and
+  re-run, or edit the JSON. The change is a reviewable diff in the same PR.
+- **Inspecting** — use the CLI (below).
+
+## Using it in CI
+
+Because cases are plain pytest tests, your existing `pytest` step gates them:
+
+```yaml
+- run: pip install -e ".[dev]"
+- run: pytest
+```
+
+A failed case fails the build. Baselines live in the repo, so CI needs no secrets
+and nothing leaves your infrastructure.
+
+## CLI
+
+```sh
+signaltest version
+signaltest baselines baselines/agent.json   # list recorded cases
+signaltest show baselines/agent.json math::exact_match
+```
+
+## Development
+
+```sh
+git clone https://github.com/Falcon305/signaltest
+cd signaltest
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev]"
+pytest
+ruff check src tests examples
+```
+
+With uv the setup is a single command (it creates the environment for you):
+
+```sh
+uv sync --extra dev
+uv run pytest
+```
+
+Try the offline examples (cached responses, no API key):
+
+```sh
+python examples/demo.py        # smallest possible case
+python examples/tool_agent.py  # tool-using agent: trajectory + answer checks
+```
+
+See [docs/architecture.md](docs/architecture.md) for how the pieces fit together
+and how to add your own metric.
+
+## FAQ
+
+**Does it call my LLM?** Only through the `run` function you provide. signaltest
+never talks to a provider itself.
+
+**How many samples do I need?** `n=10` is a sane default for numeric metrics.
+Boolean metrics resolve in coarser steps, so they need more — bump `n` and watch
+for `inconclusive`, which means the test can't yet detect a change of the size you
+care about.
+
+**Why did a case come back `inconclusive`?** Too few valid samples to be
+trustworthy. Increase `n`, or fix whatever made runs error out.
+
+**Does my data leave my machine?** No. Baselines are local JSON; there is no
+service.
+
+## Contributing
+
+Issues and pull requests are welcome. Keep changes small and focused, and add a
+test for anything you change. See `CONTRIBUTING.md`.
+
+## License
+
+MIT