rigorloop 0.1.0__tar.gz → 0.1.1__tar.gz

rigorloop-0.1.1/AGENTS.md

@@ -0,0 +1,121 @@
+# AGENTS.md
+
+Orientation for coding agents working in this repository. Read this first, then
+read **[`CODING_STYLE.md`](CODING_STYLE.md)** before writing or modifying any
+code — its rules are hard constraints, not preferences.
+
+## What this package is
+
+**RigorLoop** is a statistically-sound agentic build framework. You give it a
+task description, a pile of gold-standard input/output examples, and a set of
+checks; it runs agentic loops (a strategy agent directing concurrent executor
+agents) that iteratively build a solution and evaluate it on a strict
+**dev / validation / test** split, so the final score is trustworthy. The
+produced artifact is portable: an executable Python script, an agent skill
+(`SKILL.md`), or a guidance file (`AGENTS.md`/`CLAUDE.md`).
+
+- Pure Python, **stdlib-only** by design (no runtime dependencies), Python ≥ 3.12.
+- Ships a single CLI: `rigorloop` (`init` / `check` / `run` / `report`).
+- Invokes agents headless and tool-less via the `claude` CLI (`claude -p`).
+- User-facing docs: [`README.md`](README.md). Contributing: [`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## Architecture: functional core / imperative shell
+
+This is the single most important thing to understand, and it is enforced by
+[`CODING_STYLE.md`](CODING_STYLE.md):
+
+- **`src/rigorloop/core/`** — the functional core. 100% pure: no I/O, no
+  mutation, no time, no randomness, no network, no environment access. It
+  *decides*; it returns values and plans of effects. Testable with plain inputs
+  and zero mocks. Core coverage is held to a higher bar (≥95%).
+- **`src/rigorloop/shell/`** — the thin imperative shell. Performs all effects
+  (filesystem, subprocess, the `claude` CLI) and hands plain data to the core.
+  Keep it small.
+
+The dev/val/test split is encoded in the type system (`DevExample`,
+`ValExample`, `TestExample`) so leaking holdout data into an agent-context
+prompt is a *type error*, not a runtime bug. Don't defeat this.
+
+## Working in this repo
+
+- Dev commands live in the [`justfile`](justfile), each mirroring a CI job:
+  `just lint`, `just typecheck`, `just test`, `just check` (all three), `just fmt`.
+- Tooling: `uv` for env/build, `ruff` (lint + format), `mypy --strict`, `pytest`.
+  `T20` (print) is banned in the core and allowed in the shell.
+- Every source module has a sibling test in `tests/` (e.g. `scoring_calcs.py` →
+  `test_scoring_calcs.py`); `test_leakage.py` guards the split-type invariant and
+  `test_e2e.py` runs full loops against fake agents.
+
+## Folder tree
+
+```
+RigorLoop/
+├── AGENTS.md                     # this file
+├── CODING_STYLE.md               # MANDATORY coding rules (read before editing)
+├── CONTRIBUTING.md               # contributor workflow
+├── README.md                     # user-facing overview & docs
+├── CHANGELOG.md
+├── SECURITY.md                   # runs generated code locally — read this
+├── LICENSE                       # MIT
+├── justfile                      # dev entry points (mirror CI jobs)
+├── pyproject.toml                # package metadata + tool config
+├── uv.lock
+├── .pre-commit-config.yaml
+├── .gitignore
+│
+├── src/
+│   └── rigorloop/
+│       ├── __init__.py           # package version resolution
+│       ├── py.typed              # PEP 561 typing marker
+│       ├── core/                 # FUNCTIONAL CORE — pure, effect-free
+│       │   ├── __init__.py
+│       │   ├── types.py          # domain model: algebraic data types (products + sums)
+│       │   ├── config_calcs.py   # parse rigorloop.toml → typed RunConfig
+│       │   ├── dataset_calcs.py  # parse examples, dedup, split, manifests, power warnings
+│       │   ├── prompt_calcs.py   # prompt builders (agent-context vs. evaluation channels)
+│       │   ├── scoring_calcs.py  # checks, aggregation, Wilson/bootstrap CIs, McNemar
+│       │   ├── strategy_calcs.py # validation cadence, stopping rules, champion selection
+│       │   └── report_calcs.py   # render report, check summary, budget estimate
+│       └── shell/                # IMPERATIVE SHELL — effects at the edges
+│           ├── __init__.py
+│           ├── cli.py            # argparse entry point + orchestration driver
+│           ├── agent_calls.py    # claude CLI subprocess wrapper, retries, concurrency
+│           └── io_actions.py     # run dir, artifact persist/reload, sandboxed exec
+│
+├── tests/                        # one test module per source module + leakage/e2e
+│   ├── __init__.py
+│   ├── conftest.py
+│   ├── test_config_calcs.py
+│   ├── test_dataset_calcs.py
+│   ├── test_prompt_calcs.py
+│   ├── test_scoring_calcs.py
+│   ├── test_strategy_calcs.py
+│   ├── test_report_calcs.py
+│   ├── test_agent_calls.py
+│   ├── test_io_actions.py
+│   ├── test_cli.py
+│   ├── test_leakage.py           # asserts split types prevent holdout leakage
+│   └── test_e2e.py               # full runs against fake agents
+│
+├── examples/
+│   └── contact-cards/            # toy project; exactly what `rigorloop init` scaffolds
+│       ├── rigorloop.toml
+│       ├── task.md
+│       └── examples.jsonl
+│
+├── scripts/
+│   └── live-smoke.sh             # live smoke test against the real claude CLI
+│
+├── .github/                      # CI/release workflows, issue/PR templates, dependabot
+│   ├── workflows/
+│   │   ├── ci.yml
+│   │   └── release.yml
+│   ├── ISSUE_TEMPLATE/
+│   │   ├── bug_report.md
+│   │   └── feature_request.md
+│   ├── PULL_REQUEST_TEMPLATE.md
+│   └── dependabot.yml
+│
+└── .claude/
+    └── settings.json             # Claude Code project settings & safety hooks
+```

{rigorloop-0.1.0 → rigorloop-0.1.1}/CHANGELOG.md

@@ -4,7 +4,30 @@ All notable changes to this project are documented in this file, following
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
 [SemVer](https://semver.org/) (`0.x`: minor = features/breaking, patch = fixes).
 
-## [Unreleased]
+## [v0.1.1]
+
+Documentation and repository-hygiene only; no changes to the `rigorloop`
+package or its behavior.
+
+### Added
+
+- `AGENTS.md`: an orientation guide for coding agents, describing the package,
+  the functional-core / imperative-shell architecture (pointing at
+  `CODING_STYLE.md` as the binding rules), and a full annotated folder tree.
+
+### Changed
+
+- `CONTRIBUTING.md`: expanded the maintainer *Releases* section to document the
+  tag-driven flow — `hatch-vcs` derives the version from the `vX.Y.Z` tag (no
+  file to bump), the `[Unreleased]` → `[X.Y.Z]` changelog promotion, OIDC
+  Trusted Publishing, prerelease/TestPyPI rehearsal, and the "publish a tag
+  exactly once, fix forward" rule.
+
+### Removed
+
+- Internal planning docs `PLAN.md`, `PACKAGING_PLAN.md`, and `CLAUDE.md` removed
+  from the repository (superseded by the shipped implementation and the
+  README/CONTRIBUTING docs).
 
 ## [0.1.0] - 2026-07-08
 

{rigorloop-0.1.0 → rigorloop-0.1.1}/CONTRIBUTING.md

@@ -72,7 +72,33 @@ Two invariants are non-negotiable and covered by dedicated tests in
 
 ## Releases (maintainer)
 
-1. Ensure `main` is green and `CHANGELOG.md` has the new section.
-2. `git tag vX.Y.Z && git push origin vX.Y.Z`.
-3. Approve the `pypi` environment deployment.
-4. Verify `pip install rigorloop==X.Y.Z` in a scratch venv.
+The git tag **is** the version: `hatch-vcs` derives it from the latest
+`vX.Y.Z` tag, so there is no version number to bump in any file. The only file
+you edit for a release is `CHANGELOG.md`; pushing the tag does the rest.
+
+1. Land the change the normal way — branch, PR, `just check` green, add an
+   entry under `## [Unreleased]` in `CHANGELOG.md`, merge to `main`.
+2. On `main` (`git checkout main && git pull`), promote the changelog: rename
+   `[Unreleased]` to `[X.Y.Z] - YYYY-MM-DD`, add a fresh empty `[Unreleased]`
+   above it, commit, and push. The tag will point at this commit, so `main`
+   must be green first (branch protection requires `ci-ok`).
+3. Tag and push — this triggers `release.yml`:
+   ```bash
+   git tag vX.Y.Z && git push origin vX.Y.Z
+   ```
+4. Approve the `pypi` environment deployment when prompted (the publish job is
+   gated on manual approval). The workflow then rebuilds from the tag,
+   publishes to PyPI via OIDC Trusted Publishing (no token), and cuts a GitHub
+   Release with generated notes and the built artifacts.
+5. Verify `pip install rigorloop==X.Y.Z` in a scratch venv, then
+   `rigorloop --version`.
+
+Version choice while `0.x`: **patch** (`v0.1.1`) = fixes only; **minor**
+(`v0.2.0`) = features or breaking changes (call breakage out in the changelog);
+`v1.0.0` when `rigorloop.toml`, the CLI, and the run-directory format are
+declared stable.
+
+A version publishes to PyPI exactly once — never re-push or edit a tag. If a
+release fails partway, fix forward with a new patch tag rather than reusing the
+old one. To rehearse the full pipeline first, push a prerelease tag
+(e.g. `v0.2.0rc1`) against TestPyPI.

{rigorloop-0.1.0 → rigorloop-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rigorloop
-Version: 0.1.0
+Version: 0.1.1
 Summary: Statistically-sound agentic build framework: dev/val/test-split loops that produce code artifacts without overfitting.
 Project-URL: Homepage, https://github.com/ronikobrosly/RigorLoop
 Project-URL: Issues, https://github.com/ronikobrosly/RigorLoop/issues

{rigorloop-0.1.0 → rigorloop-0.1.1}/src/rigorloop/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.1.0'
-__version_tuple__ = version_tuple = (0, 1, 0)
+__version__ = version = '0.1.1'
+__version_tuple__ = version_tuple = (0, 1, 1)
 
 __commit_id__ = commit_id = None

rigorloop-0.1.0/CLAUDE.md

@@ -1,39 +0,0 @@
-# CLAUDE.md for RigorLoop
-
-## Project overview
-
-This a is statistically-sound agentic build framework that employs agentic loops to create code artifacts (whether a script, a skill markdown file, etc). Crucially, it splits verification data into the classic data science-like dev, validation, and final test sets to avoid overfitting. 
-
-This framework IS NOT meant to be used when someone wants to create an agentic engineering loop to produce a simple, deterministic script that will pass a series of simple unit tests. Instead, it is meant to be used for data science-like efforts like when the user needs to create a coding solution to take a set of inputs (structured or unstructured text data) and convert it into a set of structured outputs. The user would provide this frame with a (hopefully) large set of gold-standard, correct example inputs and outputs, and this agentic framework attempts to create a coding solution through statistically-rigorous loops and verifications. 
-
-The "coding solution" can take a number of forms:
-
-- a set of executable scripts
-- a "skill" for an agentic coding harness (e.g. Claude Skills)
-- a markdown file to guide agents (e.g. AGENTS.md, CLAUDE.md, etc.)
-  
-
-## Requirements
-
-* A strict splitting of a "development", "validation", and "final test" set must be made out of the input data the user provides.
-* The user must provide structured input text with the expected output too. The user must provide a set of either deterministic or probablistic verifications/checks of the proposed solutions. This framework must flexibly allow for whatever the user provides (within realistic bounds and with guidelines). The examples the user provides should be highly-representative of the universe of inputs and outputs they might encounter.
-* The framework will generate a series of solutions that iteratively improve through a series of loops. 
-* The framework will employ a strategy agent (with its own prompt) and a series of execution agents (with their own prompts). The strategy agent will review each iterations results on the "development" set, and periodically spin up an agent to test the best approach on the "validation" set. Each execution agent will only be able to see the current loop's strategy and not know about prior loops. The one sanctioned exception: the strategy agent may embed the current champion solution's *content* in a directive as a refinement starting point — solution content only, never scores, mistakes, or per-example failures from prior loops. 
-* Protection must be taken to prevent data leakage among the agents: e.g. an execution agent sees all mistakes from prior loops. Again, the idea is that the strategy agent understands what works and doesn't work, loop-to-loop, keeps a log for itself, and farms out the pure execution to a set of concurrent executor agents. This leakage guarantee applies to *agent-context* prompts (strategy and executor). *Evaluation* prompts — running a solution-under-test or an LLM judge on a single example — necessarily embed that example (whatever its split) and are a separate, sanctioned channel whose outputs return only to the harness as scores. 
-* The final loop will produce the best performing coding solution, which should be able to be copied by the user, used outside of this framework, and produce good results for them on new input data. 
-* Claude will be called in headless model and without tools (`claude -p --tools ""`) to spin up the above agents.
-
-
-## Comprehensive implementation plan
-
-See the plan in `PLAN.md`
-
-
-## Coding style
-
-See the rules and guidelines described in `CODING_STYLE.md`
-
-
-## Packaging as an open source release
-
-See the `PACKAGING_PLAN.md` file for instructions on how to setup the packaging around this framework. Includes details on dependencies, the `pyproject.toml` file, linting, testing, and deploying to PyPi.

rigorloop-0.1.0/PACKAGING_PLAN.md

@@ -1,292 +0,0 @@
-# RigorLoop Packaging & Release Plan
-
-How RigorLoop will be packaged as an open-source Python package, tested and
-gated in CI via GitHub Actions, and published directly to PyPI. This document
-complements `PLAN.md` (which covers the software itself) and inherits its
-decisions: Python ≥ 3.12, `src/` layout, stdlib-only runtime dependencies,
-console script `rigorloop`, and a `pytest` + `ruff` + `mypy --strict` quality
-gate.
-
-Verified starting facts: the repo is MIT-licensed (LICENSE, 2026, Roni
-Kobrosly) and the name `rigorloop` is currently **unclaimed on PyPI**
-(`pypi.org/pypi/rigorloop/json` returns 404). Claiming the name early — via a
-first `0.0.1` release — is step one of the rollout (§9).
-
----
-
-## 1. Distribution model at a glance
-
-| Concern | Choice | Rationale |
-|---|---|---|
-| Package name | `rigorloop` | Matches repo and CLI; available on PyPI. |
-| Build backend | `hatchling` | Modern, zero-config for `src/` layouts, wide adoption, plays well with dynamic versioning. |
-| Versioning | Dynamic from git tags via `hatch-vcs` | One source of truth (the `vX.Y.Z` tag), no version-bump commits; the tag that triggers a release *is* the version. |
-| Version scheme | SemVer, starting `0.x` | `0.x` signals pre-stable API; `1.0.0` when the config format and CLI stabilize. |
-| Runtime deps | **None** (stdlib only) | Per `PLAN.md`; makes the wheel trivially installable and keeps supply-chain surface minimal. |
-| Dev workflow tool | `uv` with a committed `uv.lock` | Fast, reproducible dev/CI environments; plain `pip` remains supported (§3). |
-| Dev deps | PEP 735 `[dependency-groups]` in `pyproject.toml` | The standard successor to `requirements-dev.txt`; understood by `uv` and `pip ≥ 25.1`. No separate requirements files needed. |
-| Publishing | PyPI **Trusted Publishing** (OIDC) via `pypa/gh-action-pypi-publish` | No long-lived API tokens in repo secrets; publish rights are bound to a specific repo + workflow + environment. |
-| Artifacts | sdist + pure-Python wheel (`py3-none-any`) | No compiled code; a single universal wheel. |
-| Typing | `py.typed` marker shipped in the wheel | The package is strictly typed; downstream users get full type information. |
-| License metadata | SPDX `license = "MIT"` + `license-files` | Current PEP 639 style. |
-
-## 2. `pyproject.toml` (the single packaging file)
-
-Everything — metadata, build config, dev dependencies, and tool configuration —
-lives in `pyproject.toml`. No `setup.py`, `setup.cfg`, `requirements*.txt`,
-`mypy.ini`, `pytest.ini`, or `.ruff.toml`.
-
-```toml
-[build-system]
-requires      = ["hatchling", "hatch-vcs"]
-build-backend = "hatchling.build"
-
-[project]
-name            = "rigorloop"
-dynamic         = ["version"]
-description     = "Statistically-sound agentic build framework: dev/val/test-split loops that produce code artifacts without overfitting."
-readme          = "README.md"
-license         = "MIT"
-license-files   = ["LICENSE"]
-authors         = [{ name = "Roni Kobrosly", email = "roni.kobrosly@gmail.com" }]
-requires-python = ">=3.12"
-dependencies    = []                    # stdlib-only, by design
-keywords        = ["agents", "llm", "evaluation", "claude", "data-science"]
-classifiers = [
-  "Development Status :: 3 - Alpha",
-  "Intended Audience :: Developers",
-  "Intended Audience :: Science/Research",
-  "Operating System :: OS Independent",
-  "Programming Language :: Python :: 3",
-  "Programming Language :: Python :: 3.12",
-  "Programming Language :: Python :: 3.13",
-  "Programming Language :: Python :: 3.14",
-  "Topic :: Software Development :: Code Generators",
-  "Typing :: Typed",
-]
-
-[project.urls]
-Homepage  = "https://github.com/ronikobrosly/RigorLoop"
-Issues    = "https://github.com/ronikobrosly/RigorLoop/issues"
-Changelog = "https://github.com/ronikobrosly/RigorLoop/blob/main/CHANGELOG.md"
-
-[project.scripts]
-rigorloop = "rigorloop.shell.cli:main"
-
-[tool.hatch.version]
-source = "vcs"                          # version = latest vX.Y.Z git tag
-
-[tool.hatch.build.hooks.vcs]
-version-file = "src/rigorloop/_version.py"   # generated; git-ignored
-
-# ---- development dependencies (PEP 735) ----
-[dependency-groups]
-dev = [
-  "pytest>=8",
-  "pytest-cov>=5",
-  "hypothesis>=6",        # property-style tests for splitting/scoring math
-  "mypy>=1.16",           # the exhaustive-match error code (§4) needs ≥1.16
-  "ruff>=0.8",
-  "pre-commit>=4",
-]
-```
-
-Runtime version lookup uses `importlib.metadata.version("rigorloop")` (for
-`rigorloop --version`), with the generated `_version.py` as the in-repo
-fallback for editable installs.
-
-## 3. Environments and "requirements files"
-
-There are deliberately **no `requirements*.txt` files**. The complete story:
-
-- **Runtime:** `dependencies = []`. Nothing to pin.
-- **Development:** the `dev` dependency group above, resolved and locked in a
-  committed **`uv.lock`**. Setup is one command: `uv sync` (creates `.venv`,
-  installs the package editable plus the dev group, honoring the lock).
-- **Without uv:** `pip install -e . --group dev` (pip ≥ 25.1) — same group,
-  unlocked resolution. Documented in `CONTRIBUTING.md` as the fallback path.
-- **CI:** uses `uv sync --locked`, which fails if `uv.lock` is stale — so CI
-  always runs the exact locked toolchain and the lock can't silently drift.
-
-Tool-version pinning for reproducibility lives in exactly two places:
-`uv.lock` (Python tools) and `.pre-commit-config.yaml` (hook revisions).
-Dependabot keeps both fresh (§6).
-
-## 4. Development tools and their configuration
-
-All tool config in `pyproject.toml`:
-
-| Tool | Role | Key configuration |
-|---|---|---|
-| **ruff** (lint) | Correctness + style lints, import sorting | `target-version = "py312"`; rule sets: `E,W,F` (pycodestyle/pyflakes), `I` (isort), `UP` (pyupgrade), `B` (bugbear), `C4` (comprehensions), `SIM` (simplify), `RUF`. Notably `T20` (no stray `print`) with a per-file ignore for `src/rigorloop/shell/**` — printing is a shell effect, forbidden only in the core. |
-| **ruff format** | Formatting (Black-compatible) | Defaults; line length 100. |
-| **mypy** | Strict static typing — load-bearing per `PLAN.md` (exhaustive `match`, split-type leakage guards) | `strict = true`, `warn_unreachable = true`, `enable_error_code = ["exhaustive-match", "ignore-without-code"]`; zero untyped code, including tests. |
-| **pytest** | Test runner | `testpaths = ["tests"]`; markers `unit`, `integration`, `e2e`; `addopts = "--strict-markers"`. |
-| **pytest-cov / coverage** | Coverage measurement | `source = ["rigorloop"]`, branch coverage on; **fail-under gates: 95% for `src/rigorloop/core/`** (pure functions have no excuse), 80% overall. Reported in CI job summary; no external coverage service initially. |
-| **hypothesis** | Property-based tests | For `dataset_calcs` invariants (disjoint/exhaustive/deterministic splits) and `scoring_calcs` math (CI bounds, aggregation identities). |
-| **pre-commit** | Local fast gate mirroring CI | Hooks: `ruff check --fix`, `ruff format`, `mypy` (via local hook running `uv run mypy`), plus hygiene hooks (end-of-file, trailing whitespace, TOML/YAML syntax). |
-
-A `justfile` (or `Makefile` — pick one, `justfile` preferred) gives memorable
-entry points that match CI exactly: `just lint`, `just typecheck`, `just test`,
-`just check` (all three), `just build`.
-
-## 5. Repository additions for open source
-
-```
-.
-├── pyproject.toml
-├── uv.lock
-├── justfile
-├── .pre-commit-config.yaml
-├── CHANGELOG.md                  # Keep a Changelog format; updated per release
-├── CONTRIBUTING.md               # setup (uv + pip paths), quality gates,
-│                                 # pointer to CODING_STYLE.md as hard rules
-├── SECURITY.md                   # supported versions + private reporting via
-│                                 # GitHub security advisories; note that
-│                                 # RigorLoop executes generated code (PLAN §7)
-├── .github/
-│   ├── workflows/
-│   │   ├── ci.yml
-│   │   └── release.yml
-│   ├── dependabot.yml
-│   ├── ISSUE_TEMPLATE/{bug_report.md, feature_request.md}
-│   └── PULL_REQUEST_TEMPLATE.md
-└── src/rigorloop/
-    ├── py.typed                  # ships in the wheel
-    └── _version.py               # generated by hatch-vcs; in .gitignore
-```
-
-README gets the standard badge row (CI status, PyPI version, Python versions,
-license) once the first release is out.
-
-## 6. CI workflow — `.github/workflows/ci.yml`
-
-Triggers: every PR, and pushes to `main`. Concurrency group cancels superseded
-runs on the same ref. All jobs pin actions to major versions and set
-`permissions: contents: read` at the workflow level.
-
-Jobs:
-
-1. **lint** (ubuntu, Python 3.12): `uv sync --locked` →
-   `ruff check .` → `ruff format --check .`.
-2. **typecheck** (ubuntu, 3.12): `mypy src tests` (strict).
-3. **test** — matrix: `python: [3.12, 3.13, 3.14]` × `os: [ubuntu-latest,
-   macos-latest]` (the shell drives subprocesses, so POSIX coverage on both;
-   Windows is explicitly unsupported in v1 — `claude` CLI and sandboxing
-   assumptions are POSIX). Runs `pytest --cov` with coverage gates; uploads
-   the coverage summary to the job summary. Uses `astral-sh/setup-uv` with
-   built-in cache.
-4. **build** (ubuntu, 3.12): `uv build` (sdist + wheel) → `twine check
-   --strict dist/*` → install the built wheel into a fresh venv and run
-   `rigorloop --version` and `python -c "import rigorloop"` — catches
-   packaging mistakes (missing `py.typed`, broken entry point, bad metadata)
-   on every PR, not at release time. Uploads `dist/` as a workflow artifact.
-5. **ci-ok** — a single no-op job that `needs:` all of the above; the one
-   required status check for branch protection (matrix-proof).
-
-Notes:
-
-- `actions/checkout` with `fetch-depth: 0` everywhere — `hatch-vcs` needs tag
-  history to compute the version.
-- No `claude` CLI in CI: unit/integration/E2E tests use the stub CLI and fake
-  agent function from `PLAN.md` §11. Live smoke tests against real `claude`
-  stay local/manual.
-
-## 7. Release workflow — `.github/workflows/release.yml`
-
-Trigger: push of a tag matching `v[0-9]+.[0-9]+.[0-9]+*` (e.g. `v0.3.0`).
-
-```
-jobs:
-  build:            # rebuild from the tag; never reuse CI artifacts
-    - uv build
-    - twine check --strict dist/*
-    - upload dist/ as artifact
-
-  publish-pypi:
-    needs: build
-    environment: pypi                # GitHub environment, protection rules on
-    permissions:
-      id-token: write                # OIDC for Trusted Publishing — no token
-    steps:
-      - download dist/ artifact
-      - uses: pypa/gh-action-pypi-publish@release/v1
-
-  github-release:
-    needs: publish-pypi
-    permissions:
-      contents: write
-    steps:
-      - create GitHub Release from the tag with auto-generated notes
-        plus the matching CHANGELOG.md section; attach dist/*
-```
-
-Security posture:
-
-- **Trusted Publishing** (OIDC): on PyPI, register the publisher as repo
-  `ronikobrosly/RigorLoop`, workflow `release.yml`, environment `pypi`. No
-  `PYPI_API_TOKEN` secret ever exists.
-- The `pypi` GitHub environment carries protection rules (required reviewer:
-  the maintainer) so a pushed tag alone can't publish without a human click —
-  cheap insurance for a solo-maintainer project.
-- Publish job contains *only* download-and-publish; it never runs the test
-  suite or arbitrary project code with the OIDC token available.
-- Tags are pushed by the maintainer; branch protection on `main` requires the
-  `ci-ok` check, so anything tagged has passed CI.
-
-Optional (recommended) pre-flight: a `publish-testpypi` job that runs on
-prerelease tags (`v*rc*`) against `test.pypi.org` with its own trusted
-publisher + `testpypi` environment, verifying the full pipeline before the
-first real release.
-
-## 8. Release process (maintainer runbook)
-
-1. Ensure `main` is green and `CHANGELOG.md` has an entry for the new version
-   (a small `ci.yml` step warns on PRs that touch `src/` without touching
-   `CHANGELOG.md` — advisory, not blocking).
-2. `git tag v0.3.0 && git push origin v0.3.0`.
-3. Approve the `pypi` environment deployment when prompted.
-4. Workflow publishes to PyPI and cuts the GitHub Release.
-5. Verify: `pip install rigorloop==0.3.0` in a scratch venv; `rigorloop
-   --version`.
-
-Version semantics while `0.x`: minor bump = features or breaking changes
-(config format, CLI, artifact layout — call breakage out in the changelog);
-patch bump = fixes only. `1.0.0` when `rigorloop.toml`, the CLI surface, and
-the run-directory format are declared stable.
-
-## 9. Rollout milestones
-
-Ordered; the first three can happen immediately after `PLAN.md` Phase 1
-(scaffolding) lands, and are prerequisites for everything after.
-
-1. **Packaging skeleton** — `pyproject.toml` as specified, `py.typed`,
-   `uv.lock`, `justfile`, `.pre-commit-config.yaml`, tool configs.
-   *Accept:* `uv sync && just check && uv build && twine check dist/*` all
-   pass locally; wheel installs and `rigorloop --version` works.
-2. **CI live** — `ci.yml` + Dependabot config; branch protection on `main`
-   requiring `ci-ok`.
-   *Accept:* a deliberately-failing PR is blocked; a clean PR is green across
-   the full matrix.
-3. **Name claim** — configure the PyPI trusted publisher, then tag `v0.0.1`
-   (packaging skeleton, honest "pre-alpha" description) to claim `rigorloop`
-   and prove the release pipeline end-to-end.
-   *Accept:* `pip install rigorloop` works from PyPI.
-4. **OSS hygiene** — `CONTRIBUTING.md`, `SECURITY.md`, `CHANGELOG.md`, issue/PR
-   templates, README badges.
-5. **First real release** — tag `v0.1.0` once `PLAN.md` Phase 7 (full loop
-   with validation/finalization) is done.
-6. **Post-1.0 niceties** (deferred): docs site (mkdocs-material) on GitHub
-   Pages, Sigstore attestations for releases, Windows support decision,
-   conda-forge feedstock if demand appears.
-
-## 10. Risks and mitigations
-
-| Risk | Mitigation |
-|---|---|
-| Name squatting before first release | Milestone 3 claims `rigorloop` with a `v0.0.1` release as soon as CI is green. |
-| Tag-based publishing fires accidentally | `pypi` environment requires manual approval; strict tag pattern; publishing rights bound to one workflow via OIDC. |
-| `hatch-vcs` version wrong in CI/builds | `fetch-depth: 0` on all checkouts; the build job's wheel-install smoke test asserts `rigorloop --version` matches the expected pattern. |
-| Lock/tool drift breaking new-contributor setup | `uv sync --locked` in CI fails on stale lock; Dependabot PRs keep `uv.lock`, GitHub Actions, and pre-commit hooks current (weekly, grouped). |
-| Matrix cost creep | 3 Python × 2 OS test matrix only; lint/typecheck/build run once on 3.12; concurrency cancellation on superseded pushes. |
-| Supply-chain exposure | Zero runtime deps; dev deps locked; actions pinned; `permissions` minimal per job; no secrets anywhere in the pipeline (OIDC only). |

rigorloop-0.1.0/PLAN.md

@@ -1,535 +0,0 @@
-# RigorLoop Implementation Plan
-
-This plan describes how to build RigorLoop: a statistically-sound agentic build
-framework that iteratively produces a coding solution (script, agent skill, or
-guidance markdown) from user-provided gold-standard input/output examples, using
-a strict dev / validation / final-test split to avoid overfitting.
-
-It is written against the requirements in `CLAUDE.md` and the architectural
-constraints in `CODING_STYLE.md` (functional core / imperative shell).
-
----
-
-## 1. Goals and non-goals
-
-**Goals**
-
-- Accept a user-provided task description, a set of gold-standard
-  `(input, expected_output)` examples, and a set of verification checks.
-- Split examples into disjoint **dev**, **validation**, and **test** sets, and
-  enforce that split throughout the run with type-level and process-level
-  leakage guards.
-- Run an iterative loop: a **strategy agent** (with persistent private memory)
-  reviews dev-set results and directs a pool of **executor agents** (stateless,
-  concurrent, seeing only the current directive) that each produce a candidate
-  solution.
-- Score candidates against the dev set with the user's checks; periodically
-  evaluate the current best candidate on the validation set; evaluate the final
-  chosen solution **exactly once** on the test set.
-- Emit a portable final artifact: the winning solution plus a statistical
-  report (scores with confidence intervals on all three sets).
-
-**Non-goals**
-
-- Building deterministic scripts to pass simple unit tests (explicitly out of
-  scope per `CLAUDE.md`).
-- Training or fine-tuning models. All "learning" happens in the strategy
-  agent's log and directives.
-- Giving agents tools. Every agent call is `claude -p --tools ""` — headless,
-  no tools; the RigorLoop shell performs all execution and I/O itself.
-
-## 2. Key decisions (assumptions to confirm)
-
-| Decision | Choice | Rationale |
-|---|---|---|
-| Language | Python ≥ 3.12 | `CODING_STYLE.md` examples are Python; `.gitignore` is Python. 3.12 gives `match` statements, `typing.assert_never`, and stdlib `tomllib`. |
-| Dependencies | Stdlib-only core; minimal shell deps | Purity rules make heavy frameworks unhelpful. No FP library — a small handwritten `Result`/`Option` in `core/types.py`. |
-| Example format | JSONL: one `{"input": ..., "expected_output": ...}` per line | Simple, streamable, natural for text-in/structure-out tasks. |
-| Config format | `rigorloop.toml` | Stdlib `tomllib`; human-editable. |
-| Agent invocation | `claude -p --tools "" --output-format json --model <m>` via subprocess | Mandated by `CLAUDE.md`; `--output-format json` gives parseable envelopes (verified against the installed CLI). |
-| Concurrency | `concurrent.futures.ThreadPoolExecutor` around subprocess calls | Executor agents are subprocess-bound; threads are sufficient and simple. |
-| Packaging | `pyproject.toml`, `src/` layout, console script `rigorloop` | Standard modern Python packaging. |
-
-## 3. Architecture: functional core / imperative shell
-
-Everything that *decides* is a pure function; everything that *acts* is a thin
-shell that executes the decisions. The core never touches I/O, time,
-randomness, the network, or subprocesses — those arrive as plain-data inputs.
-
-```
-src/rigorloop/
-├── core/                     # 100% pure; testable with zero mocks
-│   ├── types.py              # Result/Option, domain ADTs, effect descriptions
-│   ├── dataset_calcs.py      # parsing examples, splitting, split manifests
-│   ├── scoring_calcs.py      # check evaluation, aggregation, statistics
-│   ├── strategy_calcs.py     # context assembly, decision & response parsing,
-│   │                         # cadence rules, stopping rules, leaderboard
-│   └── prompt_calcs.py       # pure prompt-string builders in two channels:
-│                             # agent-context (strategy, executor) and
-│                             # evaluation (solution-under-test, judge)
-└── shell/
-    ├── agent_calls.py        # claude subprocess wrapper, concurrency, retries
-    ├── io_actions.py         # run directory, file read/write, sandboxed
-    │                         # execution of candidate scripts & custom checks
-    └── cli.py                # argparse entry point; the orchestration loop
-```
-
-Notes:
-
-- `prompt_calcs.py` and `types.py` extend the layout sketched in
-  `CODING_STYLE.md`; the three prescribed core modules and two shell modules
-  are kept as named (snake_case per Python convention: `io_actions`).
-- The orchestration loop lives in the shell (`cli.py`), but it is a *dumb
-  driver*: at each step it hands current state to a core function and receives
-  back a value describing what to do next (an effect description), then
-  performs it. The core sequences nothing; it returns plans.
-- Dependencies are injected: the RNG seed, the clock, the run-id generator,
-  and the "run an agent" function are parameters, never imported globals.
-  Integration tests swap in a fake agent function.
-
-## 4. Domain model (core/types.py)
-
-All data is frozen dataclasses (products) and tagged unions typed as
-`Union[...]` matched exhaustively with `match`/`assert_never` (sums). No
-`None`-as-maybe: `Option[T]`. No exceptions for expected failure: `Result[T, E]`
-with meaningful error sum types.
-
-Key types (illustrative, not exhaustive):
-
-```python
-# Foundations
-Option[T]  = Some(value) | Nothing
-Result[T, E] = Ok(value) | Err(error)
-
-# Examples — the split is encoded in the TYPE, not a field.
-Example        = (example_id: str, input_text: str, expected_output: str)
-DevExample     = wraps Example      # only type agent-context builders accept
-ValExample     = wraps Example      # never enters any agent-context prompt
-TestExample    = wraps Example      # never enters any agent-context prompt
-SplitManifest  = (seed, ratios, per-split example-id hashes)
-
-# What we are building
-SolutionKind   = ScriptSolution | SkillSolution | GuidanceSolution
-Candidate      = (candidate_id, loop_index, kind, content, directive_id)
-
-# Verification checks (user-configured)
-Check          = ExactMatch | NormalizedMatch(rules) | JsonEquality
-               | RegexMatch(pattern) | NumericTolerance(atol, rtol)
-               | CustomPython(script_path)          # run by the shell
-               | LlmJudge(rubric, n_samples, pass_threshold)
-CheckOutcome   = Passed | Failed(reason) | Errored(detail)
-
-# Scores
-ExampleResult  = (example_id, raw_output, outcomes per check)
-CandidateScore = (pass_rate, ci_low, ci_high, per_check_breakdown, n)
-
-# Strategy
-StrategyLogEntry  = (loop_index, observations, hypotheses, directives_issued,
-                     dev_summary, Option[val_summary])
-ChampionArtifact  = (candidate_id, kind, content)    # solution content ONLY —
-                                                     # no scores, no failures
-Directive         = (directive_id, approach_summary, instructions,
-                     base: Option[ChampionArtifact]) # refine champion / explore
-StrategyDecision  = Continue(directives, validate: Option[candidate_id])
-                  | Stop(StopReason)                 # validation is a field of
-                                                     # Continue, not a variant:
-                                                     # a peek never stalls a loop
-StopReason        = BudgetExhausted | ValidationPlateau | TargetReached
-                  | StrategyRequestedStop(reason) | StrategyUnresponsive
-
-# Prompt channels — distinct types; the §6 leakage guards apply to the first
-AgentContextPrompt = prompt for the strategy or executor agents; builders
-                     accept Dev-typed examples and aggregate scores only
-EvalPrompt         = prompt running a solution-under-test or an LLM judge on
-                     ONE example (any split); its output returns to the
-                     harness as data, never into an AgentContextPrompt
-
-# Effects — descriptions the core returns and the shell executes
-AgentRequest   = (role, prompt, model, timeout_s)
-RunScript      = (script_path, stdin_text, timeout_s)
-EffectPlan     = list of the above + persistence descriptions
-```
-
-**"Parse, don't validate":** the shell reads raw JSONL/TOML and immediately
-calls core parsers that return `Result[TypedThing, ParseError]`. Past that
-boundary the core never re-checks validity.
-
-## 5. The run protocol
-
-### Phase A — Intake and split (once per run)
-
-1. Shell reads `rigorloop.toml` + examples JSONL; core parses both into typed
-   config and `list[Example]` (`Result`-returning).
-2. Core detects **exact duplicates** (identical input text) before splitting:
-   duplicates are collapsed to one example and a warning with counts is
-   surfaced. A duplicate straddling dev and test would silently corrupt the
-   holdout. Near-duplicate detection is out of scope for v1 and called out as
-   a caveat in the README.
-3. `dataset_calcs.split(examples, ratios, seed)` deterministically partitions
-   into dev/val/test (default **60/20/20**, configurable). Pure: the seed is a
-   parameter. Guarantees: disjoint, exhaustive, stable for a given seed.
-4. Core emits a `SplitManifest` with content hashes; shell persists it. On
-   resume, the manifest is re-verified so a re-run can never reshuffle examples
-   across splits (a silent-leakage guard). The manifest also pins the
-   configured agent model and `claude` CLI version, because skill/guidance
-   scores are conditional on the evaluating model (§8).
-5. Core computes power warnings (see §8) — e.g. "validation set of 12 examples
-   can only distinguish pass-rate differences of ~±25 points" — and the shell
-   surfaces them before spending tokens.
-
-### Phase B — The loop (repeated up to `max_loops`)
-
-Each iteration:
-
-1. **Strategy turn.** Core assembles the strategy context — full detail for
-   the most recent `strategy_full_detail_loops` loops (default 4) with compact
-   per-loop summaries beyond that (so the context can't grow without bound),
-   the dev leaderboard with confidence intervals, aggregated dev failure
-   patterns for recent candidates, the current champion's full solution
-   content, and (if a validation was run) the aggregate validation score.
-   `prompt_calcs.build_strategy_prompt` renders it. Shell runs the agent; core
-   parses the JSON reply into a `StrategyDecision` (`Result`; one
-   reformat-retry on parse failure). If the retry also fails, the harness
-   substitutes a fallback decision — `Continue` with a single
-   refine-the-champion directive — and logs the substitution; two consecutive
-   fallbacks end the run with `Stop(StrategyUnresponsive)`.
-2. **Executor fan-out.** For `Continue(directives, _)`, core builds one
-   executor prompt per directive: task description, the directive (which may
-   embed the champion artifact as a refinement starting point — see §6), the
-   output contract, the check descriptions, and a sampled subset of dev
-   examples (default `min(30, all)`). The subset is **resampled every loop**,
-   deterministically from the injected seed and loop index; the strategy
-   prompt states this explicitly so loop-to-loop score movement isn't
-   over-attributed to directives when it is partly sample luck. Shell runs the
-   K agents concurrently (default 4).
-3. **Materialize & execute.** Core parses each reply's single fenced
-   `solution` block into a `Candidate` (malformed → one retry, then recorded
-   as a failed candidate, never crashing the loop). Shell materializes and
-   evaluates it against every dev example, per solution kind (§7), collecting
-   raw outputs as plain data. Evaluation of a candidate **short-circuits**
-   after `max_consecutive_eval_failures` consecutive errors/timeouts (default
-   5) — a hanging script must not burn dev-set-size × timeout — and the
-   candidate is recorded with an aborted evaluation.
-4. **Score.** `scoring_calcs` evaluates deterministic checks purely; for
-   `LlmJudge` checks the shell first collects judge verdicts
-   (n-sample majority vote via `claude -p --tools ""`), then hands the verdict
-   data to the core for aggregation. Output: `CandidateScore` with a Wilson
-   interval, plus per-example results.
-5. **Bookkeeping.** Core folds the new scores into the leaderboard and
-   produces the next `StrategyLogEntry`; shell persists loop artifacts.
-
-### Phase C — Validation checkpoints
-
-- Cadence decided by a pure rule in `strategy_calcs`: validate the current
-  dev-best candidate every `val_every` loops (default 3), or when a new
-  candidate beats the previous dev-best **beyond the paired-test noise band**
-  (§8 — raw margins chase noise), or when the strategy agent sets the
-  `validate` field of `Continue`. Triggered peeks respect a minimum gap of
-  `min_loops_between_peeks` loops (default 2), so the easy improvements of
-  early loops can't front-load the peek budget.
-- Hard cap on total validation evaluations (default 10). Every peek at
-  validation weakens it as an unbiased signal, so peeks are budgeted, counted,
-  and reported.
-- Only the **aggregate** validation score (and dev–val gap) is ever fed back
-  to the strategy agent — never raw validation examples or per-example
-  failures.
-
-### Phase D — Finalization (once)
-
-1. Stopping rule fires (budget exhausted, validation plateau over `patience`
-   checkpoints — where "improvement" means exceeding the CI band, not any raw
-   uptick — target score reached, or strategy stop).
-2. Winner = the reigning validation champion under **noise-aware selection**:
-   a challenger displaces the champion only when its validation score exceeds
-   the champion's beyond the paired-test noise band (McNemar / paired
-   bootstrap, §8); dev score breaks within-band ties. Selecting on validation
-   rather than dev is the core anti-overfitting mechanism; requiring the
-   noise band blunts the winner's curse of taking a max over noisy peeks.
-3. Shell evaluates the winner on the test set — the only time test examples
-   are ever read after splitting. No agent sees test data or test results;
-   this is purely a harness computation, run exactly once.
-4. Shell writes `final/`: the solution artifact (directly usable outside the
-   framework), plus `report.md` with dev/val/test scores + CIs, the dev–val–test
-   gaps, loop history, validation-peek count, per-check breakdowns, an
-   explicit note that the winner's validation score is selection-biased (the
-   test score is the honest number), and — for skill/guidance kinds — the
-   pinned eval model version the scores are conditional on.
-
-## 6. Agent roles and leakage controls
-
-| | Strategy agent | Executor agents | LLM judge |
-|---|---|---|---|
-| Cardinality | 1, logically persistent across loops (via its log; each call is still stateless `claude -p`) | K per loop, concurrent | n samples per (example, judge check) |
-| Sees | Own prior log (recent loops in full, older compacted), dev leaderboard + CIs, dev failure patterns, the champion's solution content, **aggregate** val scores | Task, current directive (optionally embedding the champion artifact), check descriptions, dev-example sample. **Nothing else about prior loops.** | Rubric, one candidate output, one expected output |
-| Never sees | Raw val/test examples, per-example val results | Other executors' work, strategy log, any val/test data, prior mistakes or per-example failures | Anything else |
-| Produces | JSON `StrategyDecision` | One fenced `solution` block | JSON verdict |
-
-Prompts are split into **two typed channels**, and the leakage guarantee is
-scoped to the first:
-
-- **Agent-context prompts** (`AgentContextPrompt`: strategy and executor).
-  The guarantee applies absolutely: no val/test example content, ever.
-- **Evaluation prompts** (`EvalPrompt`: running a skill/guidance candidate on
-  an example, or an LLM judge over one output). These *necessarily* embed the
-  example under evaluation — including val/test examples at checkpoint and
-  finalization time. That is sanctioned: an `EvalPrompt` is built by a
-  separate builder, executed in isolation, and its result returns to the
-  harness as plain data. Nothing from the evaluation channel ever reaches an
-  agent-context prompt except as aggregate scores.
-
-Leakage is then enforced twice:
-
-- **By type:** agent-context prompt builders accept only `DevExample` values
-  (and aggregate score types). Passing a `ValExample` or `TestExample` is a
-  type error; `AgentContextPrompt` and `EvalPrompt` are distinct types, so an
-  evaluation prompt cannot flow into an agent call site.
-- **By test:** dedicated tests assert that no *agent-context* prompt produced
-  during a simulated full run contains any val/test example content
-  (substring scan over every strategy/executor prompt built with a fake agent
-  runner).
-
-The requirement that "each execution agent only sees the current loop's
-strategy" holds by construction: executor prompts are built from the
-directive alone, and the strategy log is a distinct type that the executor
-prompt builder cannot accept. The **one sanctioned carry-forward channel** is
-the champion artifact: the strategy agent may embed the current best
-solution's *content* in a directive as a refinement starting point, which is
-what makes incremental convergence possible at all. Solution content only —
-never scores, never prior mistakes, never per-example failures. The
-`ChampionArtifact` type carries nothing else, and a dedicated test asserts
-that rendered directives contain no score or failure text.
-
-## 7. Solution kinds and how each is evaluated
-
-| Kind | Artifact | Evaluation of one example |
-|---|---|---|
-| `ScriptSolution` | Executable Python script | Shell runs it in a subprocess (`RunScript` effect): example input on stdin, structured output expected on stdout, timeout + output-size cap. Non-zero exit / timeout → `Errored`. |
-| `SkillSolution` | Skill markdown (e.g. Claude Skill `SKILL.md`) | Shell runs `claude -p --tools ""` with the skill content injected via `--append-system-prompt` and the example input as the prompt; the reply is the raw output to score. |
-| `GuidanceSolution` | Guidance markdown (AGENTS.md / CLAUDE.md style) | Same harness as skills: guidance prepended as system prompt, input as prompt. |
-
-Skill/guidance evaluation and judge calls run in the **evaluation prompt
-channel** (§6) — sanctioned to embed the example under evaluation, whatever
-its split. Two documented caveats for these kinds: (a) evaluation is
-tool-less by design while guidance files in the wild steer tool-using agents,
-so measured transfer to real usage is weaker — noted in the report; (b) a
-candidate's score is a draw from a *stochastic* evaluator, handled
-statistically in §8 and by pinning the eval model version in the manifest.
-
-The executor **output contract** is strict and stated in the prompt: exactly
-one fenced block tagged `solution`, nothing executable outside it. The core
-parser returns `Result[Candidate, MalformedReply]`; the shell grants one
-reformat retry.
-
-Sandboxing: generated scripts and `CustomPython` checks are untrusted code.
-V1 mitigations: subprocess with hard timeout, no stdin inheritance, output
-caps, and a scratch working directory. Documented loudly as *not* a security
-boundary; OS-level sandboxing is a listed future hardening item (§12).
-
-## 8. Statistical methodology
-
-This is the "rigor" in RigorLoop; all of it lives in `scoring_calcs.py` as
-pure, individually testable functions.
-
-- **Headline pass rate:** an example passes iff **all** configured checks
-  pass on it (conjunctive); per-check pass rates are always reported
-  alongside so a single strict check can't hide behind the aggregate.
-- **Pass-rate uncertainty:** Wilson score intervals (95%) on every reported
-  pass rate — honest at the small n this framework will often see.
-- **Continuous scores** (e.g. judge scores averaged): bootstrap percentile
-  CIs; the resample indices derive from an injected seed so results are
-  reproducible.
-- **Comparing candidates on the same set:** paired analysis — McNemar's test
-  for pass/fail checks, paired bootstrap for continuous scores. The
-  leaderboard marks differences that are within noise, and the strategy prompt
-  states them as "not statistically distinguishable" so the strategy agent
-  doesn't chase noise. With dozens of candidates per run, pairwise 95% flags
-  will include some false positives; there is no formal multiple-comparison
-  correction in v1, and the strategy-prompt language hedges accordingly.
-- **CI-band-gated improvement:** everywhere the protocol asks "did it
-  improve?" — champion switching (§5 D), triggered validation peeks (§5 C),
-  and the plateau stopping rule — improvement means exceeding the paired-test
-  noise band, never a raw score uptick.
-- **Stochastic evaluators:** for skill/guidance kinds (and judge checks),
-  each per-example outcome carries model-sampling variance on top of
-  example-sampling variance. v1 evaluates one sample per example, so CIs for
-  these kinds are flagged in the leaderboard and report as conditional on the
-  pinned eval model and as understating total uncertainty; the manifest pins
-  the model and CLI version so every number is attributable.
-- **Judge self-preference:** the same model family builds and judges
-  solutions — a documented upward bias that deterministic checks don't share.
-  Per-role models (a different judge model) are the post-v1 answer.
-- **Overfitting signal:** dev–val gap tracked per checkpoint and plotted in
-  the final report; a widening gap triggers a warning in the strategy context.
-- **Selection & reporting discipline:** noise-aware selection on validation,
-  never dev; test evaluated once; the winner's validation score carries an
-  explicit selection-bias caveat in the report (test is the honest number);
-  validation-peek count reported; power warnings at intake when splits are
-  too small to support the configured target margins.
-
-## 9. Configuration and CLI
-
-```toml
-# rigorloop.toml
-[task]
-description_file = "task.md"
-solution_kind    = "script"          # script | skill | guidance
-examples_file    = "examples.jsonl"
-
-[split]
-ratios = [0.6, 0.2, 0.2]
-seed   = 17
-
-[loop]
-max_loops              = 12
-executors_per_loop     = 4
-dev_examples_in_prompt = 30          # resampled every loop (seed + loop index)
-max_consecutive_eval_failures = 5    # short-circuit a hanging candidate's eval
-strategy_full_detail_loops    = 4    # older loops appear as compact summaries
-
-[validation]
-val_every  = 3
-max_peeks  = 10
-min_loops_between_peeks = 2          # damps triggered-peek front-loading
-patience   = 2                       # checkpoints without CI-band improvement → stop
-target_pass_rate = 0.95              # optional early-success stop
-
-[agents]
-model      = "claude-sonnet-5"
-timeout_s  = 300
-
-[[checks]]
-type = "json_equality"
-
-[[checks]]
-type = "llm_judge"
-rubric = "Output captures every entity mentioned in the input..."
-n_samples = 3
-pass_threshold = 0.67
-```
-
-CLI (argparse, in `shell/cli.py`):
-
-- `rigorloop init` — scaffold `rigorloop.toml`, `task.md`, an example JSONL.
-- `rigorloop check` — parse config/examples, print split sizes + power
-  warnings, estimate the agent-call budget **kind-aware**: for skill/guidance
-  kinds, candidate evaluation itself costs one `claude` call per dev example
-  per candidate — the dominant cost, ahead of judge calls. No tokens spent.
-- `rigorloop run [--resume RUN_ID]` — execute the protocol.
-- `rigorloop report RUN_ID` — re-render the report from persisted artifacts.
-
-## 10. Run directory (persistence & resumability)
-
-```
-runs/<run_id>/
-├── manifest.json            # config snapshot + SplitManifest (hashes)
-│                            # + pinned eval model & claude CLI version
-├── splits/{dev,val,test}.jsonl
-├── strategy_log.jsonl       # append-only; the strategy agent's memory
-├── leaderboard.json
-├── loops/<n>/
-│   ├── strategy_{prompt,reply}.md
-│   └── candidates/<id>/{solution.*, outputs.jsonl, scores.json}
-└── final/
-    ├── solution.*           # the portable deliverable
-    ├── report.md
-    └── test_results.json
-```
-
-Every artifact is plain JSON/JSONL/markdown written by `io_actions.py`
-(append-only where possible). Resume = shell reloads state files, core
-re-derives the in-memory state, loop continues. The split manifest hash check
-on resume prevents dataset drift mid-run.
-
-## 11. Testing strategy
-
-- **Core, zero mocks (the bulk):** splitting determinism/disjointness/ratios;
-  every check evaluator; Wilson/bootstrap/McNemar math against known values;
-  strategy & executor reply parsers (valid, malformed, adversarial);
-  cadence/stopping rules; prompt builders (golden-file tests).
-- **Leakage tests (first-class):** the §6 substring-scan test over all
-  *agent-context* prompts from a simulated run (evaluation-channel prompts
-  are exempt by design); a test that rendered directives carry champion
-  solution content only — no score or failure text; a test asserting the test
-  set is read at most once; type-level guards for both prompt channels
-  exercised.
-- **Shell integration (thin):** `agent_calls` against a stub executable that
-  mimics `claude -p --output-format json` (success, timeout, garbage output);
-  script sandbox timeout/output-cap behavior; run-dir round-trip + resume.
-- **End-to-end with a fake agent function:** a scripted fake plays strategy +
-  executors, letting a full multi-loop run (with validation checkpoints and
-  finalization) execute in milliseconds with deterministic assertions on the
-  final report.
-- Tooling: `pytest`, `ruff`, `mypy --strict` (strict typing is load-bearing —
-  it enforces the exhaustive-match and split-type guarantees).
-
-## 12. Implementation milestones
-
-Each phase ends green: tests pass, `mypy --strict` clean.
-
-1. **Scaffolding** — `pyproject.toml`, package layout, CI-ready test harness;
-   `core/types.py` (`Result`, `Option`, first domain ADTs).
-   *Accept:* `pytest` and `mypy --strict` run clean on the skeleton.
-2. **Dataset core** — example parsing, exact-duplicate detection, splitting,
-   manifests, power warnings.
-   *Accept:* property-style tests for determinism/disjointness pass.
-3. **Scoring core** — deterministic check evaluators, aggregation, Wilson +
-   bootstrap + McNemar.
-   *Accept:* statistics validated against hand-computed known values.
-4. **Shell foundations** — `agent_calls.py` (subprocess wrapper, JSON envelope
-   parsing → `Result`, retries, concurrency), `io_actions.py` (run dir,
-   script sandbox), `cli.py` with `init`/`check`.
-   *Accept:* stub-CLI integration tests pass; `rigorloop check` works on a
-   sample project.
-5. **Single-loop end-to-end (script kind)** — executor prompt building, reply
-   parsing, candidate materialization/execution, dev scoring; one loop with a
-   hard-coded strategy directive.
-   *Accept:* fake-agent E2E produces a scored leaderboard.
-6. **Strategy loop** — strategy prompts/parsing, strategy log with windowed
-   compaction (full detail last N loops), champion carry-forward in
-   directives, the strategy fallback path, multi-loop orchestration, leakage
-   tests for both prompt channels.
-   *Accept:* multi-loop fake-agent E2E; leakage scan test green; a refine
-   directive demonstrably embeds the champion artifact.
-7. **Validation & stopping** — checkpoints, peek budget + gap damping,
-   CI-band plateau/target stopping, noise-aware winner selection,
-   finalization with one-shot test evaluation and `report.md` (including the
-   selection-bias caveat).
-   *Accept:* E2E run yields full report; test set touched exactly once.
-8. **Remaining kinds & judge checks** — `LlmJudge` (n-sample majority),
-   `SkillSolution`/`GuidanceSolution` evaluation harness, `CustomPython`
-   checks.
-   *Accept:* per-kind E2E with fake agents.
-9. **Hardening & docs** — resume, cost/budget accounting surfaced in `check`
-   and the report, README + worked example project (the README must carry the
-   cross-run test-reuse warning: re-running after seeing a test score burns
-   the holdout), live smoke test against the real `claude` CLI on a toy task.
-
-Future (post-v1) items: OS-level sandboxing for generated code, stratified
-splitting, adaptive validation cadence, HTML report.
-
-## 13. Risks and mitigations
-
-| Risk | Mitigation |
-|---|---|
-| Malformed agent replies break loops | Strict output contracts; `Result`-returning parsers; one reformat retry; failed candidates recorded, never fatal. |
-| Overfitting to dev despite the design | Noise-aware selection on validation; capped + gap-damped peeks; dev–val gap surfaced to strategy agent and report; selection-bias caveat in report; test untouched until the end. |
-| Test set burned by repeated runs | Per-run guarantee only; README warns loudly that iterating after seeing a test score requires fresh holdout examples. |
-| Loops plateau because executors regenerate from scratch | Champion artifact carried forward: strategy agent may embed the best solution's content in a refine directive (content only — never mistakes or per-example failures). |
-| Small datasets → meaningless stats | Power warnings at `check` time; CIs on every number; leaderboard marks statistically indistinguishable differences. |
-| Token/cost blowout | Kind-aware `rigorloop check` pre-run budget estimate; per-candidate eval short-circuiting; per-run call ceiling; loop and executor caps in config. |
-| Untrusted generated code | Timeouts, output caps, scratch dirs; documented as not a security boundary; OS sandboxing on the roadmap. |
-| LLM nondeterminism muddies comparisons | n-sample judge voting; paired statistical tests; reproducible seeds for everything the harness controls. |
-| `claude` CLI flag drift | All flags isolated in one shell function in `agent_calls.py`; stub-CLI tests define the expected envelope. |
-
-## 14. Open questions (defaults chosen; flag if wrong)
-
-1. **Split ratios** default to 60/20/20 — acceptable default?
-2. **Structured inputs**: v1 treats `input`/`expected_output` as strings
-   (JSON-encoded when structured). Native multi-field examples later?
-3. **Model choice**: single configured model for all roles in v1; per-role
-   models (cheaper executors, stronger strategist) is an easy follow-on.
-4. **Judge budget**: `LlmJudge` multiplies per-example evaluation cost by
-   `n_samples` (and for skill/guidance kinds it stacks on top of the already
-   call-per-example evaluation) — is an n-sample majority of 3 per example
-   acceptable, or should judges score only failures of deterministic checks
-   first (tiered checking)?