meridian-eval 0.2.0__tar.gz

meridian_eval-0.2.0/.gitignore

@@ -0,0 +1,36 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+*.tsbuildinfo
+
+# Secrets — never commit
+.env
+.env.local
+*.pem
+*.key
+
+# Meridian runtime artifacts
+.meridian/
+runs/
+gate-comment.md
+snapshot-ref.json
+
+# OS
+.DS_Store
+
+# Web dashboard
+web/node_modules/
+web/dist/

meridian_eval-0.2.0/CHANGELOG.md

@@ -0,0 +1,46 @@
+# Changelog
+
+## 0.2.0
+
+First release on PyPI, as `meridian-eval` (the command and import are still `meridian`).
+
+### Real models
+- The proxy can translate to any OpenAI-compatible provider (Groq, OpenAI, vLLM, Ollama) as well as
+  Anthropic, and the operator, not the agent, chooses the model.
+- A provider failure is no longer scored as an agent failure. Every `harness_error` carries a cause
+  (`sandbox`, `adapter_config`, `entrypoint`, `grader`, `model_provider`, `internal`); a trial whose
+  last model call failed after the proxy's retries is excluded and retried once.
+- Retries are bounded and jittered, honour `Retry-After` (seconds or HTTP-date; NaN, infinity and
+  negatives rejected), and cover transport errors and 408/429/5xx/529.
+- Time spent backing off from a throttling provider is added to the trial's deadline, capped at twice
+  the deadline. Each run prints how much throttling it absorbed, in the provider's own words.
+
+### Benchmarks
+- `meridian import-harbor` converts tasks in Harbor's format (Terminal-Bench 2.0 and others) and
+  skips, with the reason, every task Meridian cannot run faithfully.
+- `verifier_script` assertion: runs a benchmark's hidden verifier in a network-less sandbox and reads
+  its reward between per-run random markers. Grader-only files are never visible to the agent.
+- `meridian suite build-images` builds and pins one image per task; `--drop-failed` keeps going.
+- A general terminal agent (one `bash` tool) with an append-only transcript; a timeout verdict
+  quotes the agent's last activity.
+
+### Reporting
+- Small-sample Student-t and Wilson intervals (adapted from Inspect, MIT), shown beside the bootstrap.
+- A static React and TypeScript dashboard, published at
+  https://patsypppe.github.io/meridian/dashboard/, whose statistics are tested against the Python ones.
+
+### Packaging
+- `pip install meridian-eval` ships the proxy Dockerfile and the terminal agent inside the wheel;
+  `meridian setup` builds the proxy image from the installed package.
+- Dropped three dependencies nothing imported (`structlog`, `opentelemetry-sdk`, `python-dotenv`), so
+  the install is smaller.
+
+### Contributing
+- A rewritten `CONTRIBUTING.md`, issue and pull-request templates, `SECURITY.md`, a code of conduct,
+  Dependabot for Actions, Python and npm, and unit tests on both supported Pythons in CI.
+
+## 0.1.0
+
+The MVP: isolated trials, host-side grading, pass^k with task-level bootstrap intervals, a paired
+significance gate that reports its minimum detectable effect, record/replay cassettes, manifests and
+replay, and a self-evaluation against a deterministic stand-in model.

meridian_eval-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Pranav T Pattanashetty
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

meridian_eval-0.2.0/PKG-INFO

@@ -0,0 +1,475 @@
+Metadata-Version: 2.5
+Name: meridian-eval
+Version: 0.2.0
+Summary: A CI gate for AI agents: isolated trials, outcome grading, pass^k with honest intervals, and a verdict that says what it could not detect.
+Project-URL: Homepage, https://patsypppe.github.io/meridian/
+Project-URL: Source, https://github.com/patsypppe/meridian
+Project-URL: Dashboard, https://patsypppe.github.io/meridian/dashboard/
+Project-URL: Changelog, https://github.com/patsypppe/meridian/blob/main/CHANGELOG.md
+Author: Pranav T Pattanashetty
+License-Expression: MIT
+License-File: LICENSE
+License-File: THIRD_PARTY_NOTICES.md
+Keywords: agents,benchmark,ci,docker,evals,evaluation,llm,pass^k
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: <3.14,>=3.12
+Requires-Dist: alembic>=1.14
+Requires-Dist: docker>=7.1
+Requires-Dist: httpx>=0.28
+Requires-Dist: jsonpath-ng>=1.7
+Requires-Dist: numpy>=2.1
+Requires-Dist: psycopg[binary]>=3.2
+Requires-Dist: pydantic>=2.10
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: sqlalchemy>=2.0.52
+Requires-Dist: starlette>=0.45
+Requires-Dist: typer>=0.15
+Requires-Dist: uvicorn>=0.34
+Description-Content-Type: text/markdown
+
+# Meridian
+
+[![PyPI](https://img.shields.io/pypi/v/meridian-eval)](https://pypi.org/project/meridian-eval/)
+[![Python](https://img.shields.io/pypi/pyversions/meridian-eval)](https://pypi.org/project/meridian-eval/)
+[![CI](https://github.com/patsypppe/meridian/actions/workflows/ci.yml/badge.svg)](https://github.com/patsypppe/meridian/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/patsypppe/meridian/blob/main/LICENSE)
+[![good first issues](https://img.shields.io/github/issues/patsypppe/meridian/good%20first%20issue)](https://github.com/patsypppe/meridian/labels/good%20first%20issue)
+
+**A CI gate for AI agents: did this change make my agent better or worse, and can I reproduce that answer tomorrow?**
+
+```bash
+pip install meridian-eval     # or: uv tool install meridian-eval
+meridian setup                # one-time: builds the model-proxy image (needs Docker)
+meridian --help
+```
+
+Want to help? Start with a [`good first issue`](https://github.com/patsypppe/meridian/labels/good%20first%20issue)
+and read [CONTRIBUTING.md](https://github.com/patsypppe/meridian/blob/main/CONTRIBUTING.md).
+
+**A harness that cannot measure itself has no standing to measure anything else.**
+
+**[See it on one page → patsypppe.github.io/meridian](https://patsypppe.github.io/meridian/)**: three real gate verdicts, the sensitivity explorer, the isolation model and the contamination probe, all drawn from real runs ([`site/data/`](https://github.com/patsypppe/meridian/blob/main/site/data/)).
+
+[![A real gate verdict: per-task pass^3, the decision trace, the tolerance band and blind zone, and the paired bootstrap](https://raw.githubusercontent.com/patsypppe/meridian/main/docs/showcase.png)](https://patsypppe.github.io/meridian/)
+
+## What has been measured, and on what
+
+Two different things are measured here, and they are kept apart on purpose.
+
+**1. The gate's own machinery, on a simulated model.** The self-eval runs against a
+deterministic **stub provider** ([`stub_provider.py`](https://github.com/patsypppe/meridian/blob/main/src/meridian/runtime/proxy/stub_provider.py)),
+not a real LLM. The stub reproduces the Anthropic wire format and "slips" one trial in five on a
+schedule, so the whole pipeline (record, replay, gate, report) is exercised for free and
+reproducibly. These numbers show the statistics behave as designed on a *controlled* source of
+variance. They say nothing about how a real agent behaves.
+
+| Measure | Result | |
+|---|---|---|
+| **False-regression rate** | **0 / 30** | the *unchanged* simulated agent gated 30× against a fixed baseline. Zero FAILs. |
+| **True-positive rate** | **4 / 5** | seeded regressions caught. The one it missed is named below. |
+| **Replay fidelity** | **5 / 5** | archived runs re-materialized from their manifests alone and reproduced exactly. |
+| **Contamination probe** | **pass / fail** | passes with isolation on, **fails** with `--unsafe-shared-env`. A probe that cannot fail proves nothing. |
+
+<sub>`n=5` · `k=3` · tolerance `0.03` · baseline suite pass^3 `0.486` · mean run 32.4s / 51¢ notional (token usage × default price table, not real spend) · [`benchmarks/self-eval.json`](https://github.com/patsypppe/meridian/blob/main/benchmarks/self-eval.json), regenerate with `make self-eval` (~30 min)</sub>
+
+**The miss, stated plainly.** An arithmetic bug seeded into the tool that writes the invoice dropped
+suite pass^3 by 0.114, a real regression correctly measured, and the gate returned
+`pass_with_warning` rather than FAIL, because a paired significance test over only seven tasks put it
+at `p=0.091`. That is a property of a seven-task suite, not a bug in the gate, and the fix is more
+tasks rather than a looser threshold.
+
+**2. Real models.** The same suite and a hosted model, through the proxy (details in
+[Real models and real benchmarks](#real-models-and-real-benchmarks)). These are small, early and
+honestly noisy: two runs of the same open model on the same seven tasks passed 17 and 9 trials out
+of 70. That spread is itself a finding, and the reason a single run of a small suite should not be
+trusted. See [`benchmarks/real-model-2026-10-23.md`](https://github.com/patsypppe/meridian/blob/main/benchmarks/real-model-2026-10-23.md).
+
+Meridian answers one question: **did this change make my agent better or worse,
+and can I reproduce that answer tomorrow?**
+
+---
+
+## The four rules
+
+Every design decision in this repository loses to one of these.
+
+1. **Every trial starts from a clean, isolated environment.** No shared
+   filesystem, database, network namespace, or process between trials. This is
+   not asserted, it is *tested* — by a probe that must pass with isolation on and
+   **fail** with `--unsafe-shared-env`. A probe that cannot fail proves nothing.
+2. **Grade outcomes, not trajectories.** Assertions run against the final state
+   of the environment. Tool calls, turns, and tokens are recorded as efficiency
+   signals and never decide pass or fail. An agent that reaches the right state
+   by an ugly route passed.
+3. **Report `pass^k`, not a mean score.** `pass^k = C(c,k)/C(n,k)` — the
+   probability that *k* independent attempts all succeed. Never `(c/n)**k`.
+4. **Every run is reproducible from its manifest.** Digests, not tags. Integers,
+   not floats, inside anything hashed.
+
+---
+
+## Why `pass^k`
+
+From a real run of the reference suite:
+
+```
+task                      pass     n   pass@3   pass^3
+────────────────────────────────────────────────────────────
+happy-path                   4     5     1.00     0.40
+```
+
+Both numbers describe the same five trials. `pass@3 = 1.00` says the agent
+solved this task at least once in three attempts. `pass^3 = 0.40` says it would
+solve it three times out of three only 40% of the time.
+
+A mean score reports 80% and reads like a B. `pass^3` reports 0.40 and reads like
+what it is: an agent you cannot ship without a retry loop. Meridian leads with
+`pass^k` because "works if you retry it" is not a claim anybody can ship on.
+
+The metric is not Meridian's invention — it is τ-bench's
+([arXiv:2406.12045](https://arxiv.org/abs/2406.12045)). Implementing it exactly,
+as `C(c,k)/C(n,k)` rather than the `(c/n)^k` that shows up in a lot of code, is
+table stakes rather than a differentiator. Confidence intervals resample **tasks,
+not trials**, because trials within a task are correlated and a trial-level
+interval is far too narrow — see
+[arXiv:2411.00640](https://arxiv.org/abs/2411.00640).
+
+---
+
+## The gate
+
+`meridian gate` is the only command that returns non-zero for a product verdict.
+Everywhere else, non-zero means the tool broke — CI must never confuse "the agent
+got worse" with "the harness crashed", because one is a code review and the other
+is a page.
+
+| Code | Meaning |
+|---|---|
+| `0` | Merge may proceed |
+| `1` | **`gate` only** — the agent regressed |
+| `2` | The harness failed |
+
+The gate declines to draw a conclusion when it should: too many harness errors,
+too many stale cassettes, or a drop that a paired significance test cannot
+distinguish from noise. And with no baseline it returns **PASS** — a gate that
+fails on day one gets disabled on day two.
+
+### It tells you when it could not have seen the regression
+
+Every verdict carries the thing that decides whether it means anything:
+
+> **Sensitivity.** This run could reliably detect a suite-level drop of **0.183**
+> or larger (80% power, one-sided alpha=0.05). A real regression smaller than that
+> would most likely have passed. Resolving the configured tolerance of 0.030 would
+> take about **262 comparable tasks**; this comparison has 7.
+
+A `PASS` from an underpowered suite is not evidence that nothing broke. It is the
+absence of evidence either way, and a gate that reports those identically teaches
+people to trust it exactly when it is least reliable.
+
+This is also the honest explanation of Meridian's own 4/5 above. The seeded
+rounding bug moved 2 of 7 tasks and dropped suite pass^3 by 0.114 — against a
+minimum detectable effect of 0.183. The gate did not miss it because its
+threshold was wrong. It missed it because seven tasks cannot resolve it, and no
+threshold can fix that. The note says so, and says how many tasks would.
+
+### The evaluator never trusts the container
+
+Assertions run on state **extracted** from the trial, on the host, after the
+container is gone. Nothing inside the sandbox decides whether the trial passed.
+
+This is not fastidiousness. In 2026 independent researchers broke several major
+agent benchmarks precisely here: agents on SWE-bench wrote a `conftest.py` that
+rewrote every test result to `passed`; agents on Terminal-Bench replaced
+`/usr/bin/curl` to emit fake test output; one FieldWorkArena validator scored a
+`"{}"` response 1.0 on all 890 tasks. Their first recommendation — *run evaluation
+outside the agent's container, and don't trust files, outputs, or state from
+inside the sandbox* — is this design.
+
+Meridian holds the same line against its **own** inputs. A task file is untrusted:
+in the case this product exists for, a pull request edits `suites/` and CI runs
+the gate on the result. Task paths are contained to the suite directory, grader
+paths to the extracted state, cassette names to the task schema's alphabet, and
+per-trial budgets to slugs the harness recognises.
+
+### And it audits your suite the same way
+
+```bash
+meridian suite audit --suite ./suites/checkout-agent
+```
+
+Runs your tasks against agents known not to have solved anything — one that
+touches nothing, one that creates the output directory and leaves it empty, one
+that writes well-formed JSON full of invented values. Any task that still passes
+is measuring something weaker than it claims:
+
+```
+1 task(s) can be passed without doing the work:
+
+  weak-task passes for the empty-scaffold agent, which creates the output
+  directory and leaves it empty, which passes any assertion checking that a
+  path exists rather than what is in it
+```
+
+Tasks get written by watching an agent fail, so they are only ever checked
+against agents that are *trying*. A do-nothing agent has passed 38% of one widely
+used benchmark's tasks. This is the contamination probe's idea pointed at your
+assertions instead of at Meridian's isolation.
+
+**[See it fail and pass on a real PR →](https://github.com/patsypppe/meridian/blob/main/docs/demo/)**
+
+The change under review was a three-line diff removing one sentence from a
+planner prompt — a sentence that restates a policy document the agent already
+reads. The suite *mean* moved 86% → 80%. `pass^3` moved 0.486 → 0.314, and three
+tasks went from working sometimes to never working at all.
+
+---
+
+## Install
+
+Requires Python 3.12 or 3.13 and Docker.
+
+```bash
+pip install meridian-eval          # or: uv tool install meridian-eval
+meridian setup                     # one-time: builds the model-proxy image
+```
+
+The PyPI name is `meridian-eval` (plain `meridian` is taken); the command and the import are both
+`meridian`. To install the latest unreleased `main` instead:
+`pip install "git+https://github.com/patsypppe/meridian"`.
+
+An installed `meridian` can import benchmarks, build their images and run them with the bundled
+terminal agent, with no repository checkout:
+
+```bash
+meridian import-harbor ./path/to/harbor-tasks --out suites/mine --rewrite-uv-boilerplate
+meridian suite build-images suites/mine --drop-failed
+meridian run --suite suites/mine --proxy-mode record --live-provider --n 5 --k 3
+```
+
+One caveat: two unrelated PyPI packages, `meridian` and `google-meridian`, also import as
+`meridian`, so do not install either into the same environment. `uv tool install` keeps Meridian in
+its own environment, which avoids the clash.
+
+## Quickstart (from a checkout)
+
+Requires Python 3.12+, [uv](https://docs.astral.sh/uv/), and Docker.
+
+```bash
+uv sync
+make db                 # postgres + migrations
+make images             # build the environment snapshot and pin it by digest
+
+# Run the reference suite offline, from committed cassettes
+uv run meridian run --suite ./suites/checkout-agent --n 5 --k 3 --proxy-mode replay
+```
+
+No API key needed. Replay runs from cassettes committed to this repository, which
+is why CI works offline and cannot go red because a provider was slow. A key is
+needed only to *record* a new cassette.
+
+**What replay does and does not test.** Under replay the model's outputs are
+fixed, so a replayed run measures the *scaffold* — prompt assembly, tool routing,
+parsing, state mutation — and not the model. That is a real and useful thing to
+gate on, and it is most of what a pull request actually changes, but it is not a
+measurement of agent capability. Use the two modes for what each is:
+
+| | `--proxy-mode replay` | `--proxy-mode record` |
+|---|---|---|
+| Runs | every PR, offline, free | nightly, or on demand |
+| Catches | scaffold regressions | agent capability regressions |
+| Cannot catch | anything requiring the model to respond differently | — |
+
+`meridian gate` records by default for exactly this reason: a gate measures the
+agent as it is now, and cassettes from the previous version would miss on every
+changed prompt. The gate reports its stale-cassette rate and returns
+**INCONCLUSIVE** rather than FAIL when too many trials missed their recording —
+"re-record" and "your change broke something" look identical otherwise.
+
+```bash
+# Would this change block the merge?
+uv run meridian gate --suite ./suites/checkout-agent --baseline-ref main --n 5 --k 3
+
+# Reproduce any archived run exactly, from its manifest
+uv run meridian replay <run-id>
+```
+
+---
+
+## Real models and real benchmarks
+
+Meridian reaches a model only through its proxy, so the agent never holds a key. The proxy speaks the
+Anthropic messages format to the agent and can translate to any OpenAI-compatible provider, so the
+same suite runs on Claude, or on open models served by Groq, vLLM or Ollama:
+
+```bash
+export GROQ_API_KEY=...                       # or ANTHROPIC_API_KEY
+export MERIDIAN_UPSTREAM_API=openai           # omit for Anthropic
+export MERIDIAN_UPSTREAM_BASE_URL=https://api.groq.com/openai/v1
+export MERIDIAN_UPSTREAM_MODEL=openai/gpt-oss-120b   # chosen by the operator, not the agent
+uv run meridian run --suite ./suites/checkout-agent --n 5 --k 3 --proxy-mode record --live-provider
+```
+
+**A provider failure is not an agent failure.** The proxy retries rate limits and 5xx errors with
+jittered, bounded backoff (honouring `Retry-After`) and records how long it waited. A trial that
+still ends in a provider error is a `harness_error` with a cause (`sandbox`, `adapter_config`,
+`entrypoint`, `grader`, `model_provider`, `internal`), excluded from the denominator and retried
+once, never scored as the agent failing. Time spent backing off is added to the trial's deadline
+(capped at twice the deadline), and the run prints what the provider said, for example the exact
+rate limit it hit. Found the hard way: agents that had finished in one reply were being scored as
+timeouts while their next request sat in a rate-limit backoff.
+
+**Importing established benchmarks.** [Harbor](https://github.com/laude-institute/harbor)'s task
+format is shared by dozens of benchmarks. `meridian import-harbor` reads it and writes a Meridian
+suite:
+
+```bash
+uv run meridian import-harbor <tasks-dir> --out suites/harbor-slice --rewrite-uv-boilerplate
+uv run meridian suite build-images suites/harbor-slice --drop-failed
+uv run meridian run --suite suites/harbor-slice --sut fixtures --proxy-mode record --live-provider
+```
+
+The benchmark's hidden tests ship as grader-only files, never visible to the agent, and its
+verifier script runs in a throwaway container with no network. The reward is read back between
+random per-run markers, so nothing the agent prints can forge a pass. A task Meridian cannot honour
+is **skipped with the reason**, not converted into something that runs under different rules:
+multi-step tasks, GPUs, MCP servers, compose sidecars, Windows, non-apt bases, and verifiers that
+install packages at test time.
+
+What was actually checked: on Terminal-Bench 2.0, **66 of 89 tasks convert** (23 are refused because
+their verifier needs the network); on Harbor's own 32 example tasks, 13 convert. A converted task
+from Harbor's examples ran end to end on a real model and was graded by Harbor's own verifier. **A
+Terminal-Bench slice has not been run yet** (it needs an Anthropic key), so no Terminal-Bench score
+is claimed. Tasks run in a sandbox with no internet, which many Terminal-Bench tasks assume, so
+expect that to cost points.
+
+**Reading results.** `meridian run --out result.json`, then open the
+[dashboard](https://patsypppe.github.io/meridian/dashboard/) (React + TypeScript, runs entirely in
+your browser) and drop the file in: suite pass^k with a Student-t interval, harness errors by cause,
+a per-task table, a trial grid, the pass@k vs pass^k curve, and a comparison against a baseline run.
+With only a handful of tasks a percentile bootstrap undercovers, so the report shows both intervals.
+
+---
+
+## How it works
+
+```
+meridian run
+   ├── manifest      pins suite, config, image digests, adapter, harness version
+   ├── proxy         one container: budget ledger, credential broker, cassettes
+   └── per trial     one container, one volume, torn down in a finally
+          ├── adapter runs the agent inside the container
+          ├── state extracted before the container is removed
+          └── graders assert against that state — never the trajectory
+```
+
+**Isolation.** Every container security option lives in exactly one file,
+`src/meridian/runtime/isolation.py`, so a reviewer audits the isolation claim by
+reading one file rather than grepping for `privileged`.
+
+**The proxy.** The agent gets a `base_url` and no credential. The proxy holds the
+key, enforces per-trial token and cost budgets where the spend actually happens,
+and records or replays every call. One component satisfies secret injection, cost
+control, and reproducibility.
+
+**Cassettes are ordered tapes per trial**, not a request→response map. A map
+would serve two trials of the same task the same response, every recorded run
+would become deterministic, and `pass^k` could only ever be 0 or 1. Replay fails
+closed: an unrecorded or out-of-order request is an error, never a live call.
+
+**The harness never imports the system under test.** Only the adapter does, and it
+runs inside the trial container.
+
+---
+
+## Measuring the harness
+
+```bash
+make self-eval        # ~30 min: 46 real gate runs end to end
+```
+
+Three numbers, and the honest one is the first.
+
+**False-regression rate** gates the *unchanged* agent 30 times against a fixed
+baseline. Every FAIL is noise — the gate crying wolf at a change that did not
+happen. Anything above zero means `n` is too low or the tolerance too tight, and
+reporting that is worth more than a clean number nobody can reproduce.
+
+**True-positive rate** seeds five regressions of known shape into the system under
+test and counts how many the gate catches: the planner's expiry instruction
+removed, a context-window guard that drops the transcript, a tool deleted,
+`max_tokens` starved, and an arithmetic bug in the tool that writes the invoice.
+Anything the gate misses is a change that would have shipped.
+
+**Replay fidelity** re-runs archived runs from their manifests. Drift means an
+input exists that the manifest does not pin.
+
+Every seeded regression changes the system under test — never the harness, never
+the thresholds. A benchmark that tunes the thing it is measuring measures
+nothing. And a seeded edit that silently matches nothing raises rather than
+quietly scoring as a miss the gate never had a chance at.
+
+---
+
+## Development
+
+```bash
+make check         # ruff + mypy --strict + unit tests — green before any commit
+make integration   # Docker-backed tests
+make e2e           # full run, replay, and gate against the fixture agent
+make sweep         # remove leaked trial containers
+cd web && npm ci && npm test && npm run build   # the dashboard
+```
+
+About 570 Python tests (unit, Docker-backed integration, end to end) run under `mypy --strict` in a
+five-job CI pipeline, plus the dashboard's own tests. The dashboard's TypeScript statistics are
+checked against values exported from the Python implementation
+(`uv run python -m benchmarks.export_web_vectors`), and a Python test fails if that file goes stale.
+
+Pure logic — `stats/`, `gate/decide.py`, `manifest/build.py`, `grading/graders/` —
+has no I/O and no Docker imports, and is unit tested exhaustively without a
+daemon. The rule is not a coverage percentage: **every function in those modules
+has both a passing case and a failing case.**
+
+---
+
+## Documentation
+
+| | |
+|---|---|
+| [`CONTRIBUTING.md`](https://github.com/patsypppe/meridian/blob/main/CONTRIBUTING.md) | How to set up, find an issue, run the right tests, and open a pull request |
+| [`CHANGELOG.md`](https://github.com/patsypppe/meridian/blob/main/CHANGELOG.md) | What changed in each release |
+| [`SECURITY.md`](https://github.com/patsypppe/meridian/blob/main/SECURITY.md) | How to report a vulnerability privately |
+| [`docs/runbook.md`](https://github.com/patsypppe/meridian/blob/main/docs/runbook.md) | One entry per failure mode: what you see, what it means, what to do |
+| [`docs/demo/`](https://github.com/patsypppe/meridian/blob/main/docs/demo/) | Captured output from a real gated PR, and how to regenerate it |
+| [`docs/HANDOFF.md`](https://github.com/patsypppe/meridian/blob/main/docs/HANDOFF.md) | The engineering contract this was built against |
+| [`docs/PRD.md`](https://github.com/patsypppe/meridian/blob/main/docs/PRD.md) | Product requirements |
+| [`web/README.md`](https://github.com/patsypppe/meridian/blob/main/web/README.md) | The React + TypeScript dashboard: what it shows, how its statistics are kept in step with Python |
+| [`benchmarks/real-model-2026-10-23.md`](https://github.com/patsypppe/meridian/blob/main/benchmarks/real-model-2026-10-23.md) | The first real-model runs, with their caveats |
+| [`THIRD_PARTY_NOTICES.md`](https://github.com/patsypppe/meridian/blob/main/THIRD_PARTY_NOTICES.md) | Code adapted from other projects, and its licenses |
+
+## Not in this release
+
+LLM-as-judge: it needs a calibration workflow and thirty human labels first, and an uncalibrated
+judge is worse than no judge. Also out: the two-axis result cache (nothing expensive to memoize with
+deterministic graders), drift detection, multi-tenancy, hosted runs, and adapters beyond LangGraph and
+the trivial subprocess one (the terminal agent used for imported benchmarks is a LangGraph graph).
+Known gaps: the cause of a harness error is saved in each run's result file but not yet in the
+Postgres `trial` table; a timed-out trial is scored as a timeout without grading its final state;
+and the suites are small, so intervals are wide.
+
+## Credits
+
+Small-sample interval code is adapted from [Inspect](https://github.com/UKGovernmentBEIS/inspect_ai)
+(MIT); the benchmark task format is [Harbor](https://github.com/laude-institute/harbor)'s
+(Apache-2.0). See [`THIRD_PARTY_NOTICES.md`](https://github.com/patsypppe/meridian/blob/main/THIRD_PARTY_NOTICES.md).