touchstone-bench 0.1.0__tar.gz

touchstone_bench-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,13 @@
+name: ci
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv python install 3.13
+      - run: uv sync --group dev
+      - run: uv run ruff check .
+      - run: uv run complexipy --max-complexity-allowed 8 touchstone
+      - run: uv run pytest -q

touchstone_bench-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,19 @@
+name: release
+on:
+  push:
+    tags: ["v*"]
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # PyPI trusted publishing, no token to store
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv build
+      - run: uv publish
+      - uses: softprops/action-gh-release@v2
+        with:
+          files: dist/*

touchstone_bench-0.1.0/.gitignore

@@ -0,0 +1,17 @@
+.venv/
+__pycache__/
+*.pyc
+.touchstone/
+*.db
+*.db-wal
+*.db-shm
+.pytest_cache/
+.ruff_cache/
+dist/
+harbor-tasks/
+*.egg-info/
+.DS_Store
+touchstone.toml
+
+# Harbor job directories belong under the dataset root, never in the repo
+jobs/

touchstone_bench-0.1.0/.python-version

@@ -0,0 +1 @@
+3.13

touchstone_bench-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Pebble ML
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

touchstone_bench-0.1.0/PKG-INFO

@@ -0,0 +1,170 @@
+Metadata-Version: 2.5
+Name: touchstone-bench
+Version: 0.1.0
+Summary: Turn a company's running AI agent into a Harbor benchmark of its own work — capture, survey, review, train.
+Author-email: Sam Larson <sam@pebbleml.com>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: fastapi>=0.110
+Requires-Dist: httpx>=0.27
+Requires-Dist: jsonschema>=4.21
+Requires-Dist: python-multipart>=0.0.9
+Requires-Dist: tomli-w>=1.2.0
+Requires-Dist: typer>=0.12
+Requires-Dist: uvicorn>=0.29
+Requires-Dist: websockets>=12
+Provides-Extra: otel
+Requires-Dist: opentelemetry-sdk>=1.20; extra == 'otel'
+Provides-Extra: whisper
+Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
+Description-Content-Type: text/markdown
+
+# Touchstone
+
+A touchstone is the black stone assayers rubbed gold against to prove its purity before anyone paid
+for it. Touchstone does that for AI agents: it turns a company's running agent into a
+[Harbor](https://github.com/harbor-framework/harbor) benchmark of its own work, lets the people who
+know the product review that benchmark by voice, and turns the results into training data.
+
+Harbor is the middle of the product, unchanged. Touchstone is four extensions around it:
+
+1. **Capture** — one line of code records what the production agent does.
+2. **Survey** — a read-only agent reads the recordings and the code, copies the system into a Harbor
+   environment, and writes Harbor tasks against it.
+3. **Review** — a voice/text room where product people look at finished trials and correct the
+   verifiers; Harbor regrades after every correction.
+4. **Train** — turns finished jobs into distillation and RL datasets.
+
+Everything Touchstone produces is a plain Harbor dataset the customer owns and can run without us.
+Local-first: it runs on the customer's machine with their own Claude or Codex login — no Touchstone
+account, no Touchstone servers. Open source. By Pebble ML.
+
+Horizontal by design: nothing in `touchstone/harbor/` knows about any one customer, product, or
+domain.
+
+## The five commands
+
+```
+touchstone init                          # write touchstone.toml + the .touchstone/ db
+touchstone demo                          # run the built-in agent and capture episodes (zero-key)
+touchstone survey <repo>                 # read-only: map the code, simulate its services, score fidelity
+touchstone bench -m <provider/model>     # run a model over the dataset; print pass rate per task
+touchstone jobs                          # list Harbor job dirs with their pass rate per task
+touchstone serve  /  touchstone review     # open the review room (voice or text)
+touchstone train                         # turn finished jobs into distill + RL datasets
+```
+
+`touchstone survey` reads the recordings and the code with a read-only coding agent
+(`[survey] provider`, default `claude-cli`; the customer's login pays), maps the tools and their
+network boundaries, generates a SQLite-backed simulator per service, and replays every recorded call
+through the real tools to score each simulator's fidelity. It is idempotent — outputs are reused
+unless `--force` — and touches nothing outside `touchstone/` in the target repo.
+
+`touchstone bench` also takes `--agent packaged|replica`, `--dataset <dir>`, and
+`--against <job_dir>` (a per-task comparison against a previous run). `touchstone doctor` reports the
+environment. Capture is one line in your own app:
+
+```python
+import touchstone
+touchstone.trace()   # records every model + tool call to .touchstone/touchstone.db
+```
+
+Run your real agent on a different model with one env var: set `TOUCHSTONE_MODEL` and capture rewrites
+the `model=` keyword on every openai/anthropic/litellm call before it goes through — same SDK, same
+code path. (A model passed positionally is left untouched.) This is how the packaged agent under test
+is benched against a candidate model without editing the customer's code.
+
+## The dataset
+
+A survey produces a `touchstone/` directory in the customer's repo — a plain Harbor dataset they own
+and version:
+
+```
+touchstone/
+  dataset.toml      the Harbor dataset manifest (metadata; tasks run as the implicit tasks/ dataset)
+  tasks/<name>/     one Harbor task per directory (instruction.md, task.toml, environment/,
+                    solution/, tests/)
+  environment/      the customer's system, copied to run in a sandbox
+  agent/            the agent under test as a Harbor custom agent (agent.toml, tools.py, entry.py)
+  simulators/       a small local service per network boundary
+  baseline.json     what the current setup passes today (the first-five-minutes sentence)
+  report.md         what was mapped, simulated, and left open
+```
+
+The agent under test runs in one of two modes, chosen automatically and recorded in `agent/agent.toml`:
+
+- **packaged** (highest fidelity): the survey generates `agent/entry.py` — a `run(user_message)`
+  that drives the customer's *real* agent loop for one message, importing their own modules. The
+  Harbor agent runs `bash /app/agent/run.sh` in the sandbox with the model as a setting
+  (`TOUCHSTONE_MODEL`) and imports the trajectory the customer's own capture wrote. An adapter check
+  proves entry.py runs and calls a tool before this mode is chosen.
+- **replica** (fallback when the code will not run): `touchstone.harbor.agent:TouchstoneAgent` runs
+  an OpenAI-compatible tool-calling loop whose system prompt comes from `agent/agent.toml` and whose
+  tool schemas + dispatch come from `agent/tools.py`.
+
+Either way it records an ATIF `trajectory.json`, and `touchstone bench -m <candidate>` re-runs it on
+any model without touching the customer's code.
+
+## The review room
+
+`touchstone serve` then `touchstone review` opens a room (voice or text) where a product person walks
+the finished trials with an AI. It opens from the product's goal (the job labels and the baseline
+pass count), then picks trials in order — verifier unsure, models disagree, never reviewed, then
+gate failures — and for each reads the instruction, the trajectory in plain words, and every
+criterion's score, and asks "do you agree it passed?". Agreement records trust (the share of
+reviewed trials where the human agreed with the verifier, shown live). On a disagreement the agent
+drafts a criterion change — edit/add/remove a rewardkit check, a dimension weight, a judge line, or
+the instruction wording — reads it back, and on "yes" writes the `tests/` file and runs
+`harbor job regrade` (the tasks are authored with a separate verifier so grading reruns from the
+recorded artifacts, no agent), then reads out the new reward and any other trials that moved.
+"Always" applies the same criterion to every task with the same job. Everything the room decides is
+a row in `reviews` and a file change under `touchstone/` — nothing else.
+
+## Training data
+
+`touchstone train` reads the Harbor job directories (`trajectory.json` + `reward.json` per trial)
+and writes, under `touchstone/train/`, `distill.jsonl` (full trajectories from trials that scored at
+or above the threshold, for a student to copy), `rl_tasks.toml` (tasks in the learnability band —
+pass rate strictly between 0 and 1 — with the verifier as the reward), and `manifest.json` (per-task
+pass rate per model and where each task went). `touchstone train --teacher <spec>` runs the teacher
+job first. Training itself stops at the exact GPU command; Touchstone produces the data, not the run.
+
+## Storage
+
+Only what is captured lives in SQLite (`.touchstone/touchstone.db`): `episodes`, `spans`, `rooms`,
+`room_messages`, and `reviews` (what a review room decided about a task's trial). Harbor's own
+`jobs/` directories are the run record — Touchstone reads them, it has no runs table.
+
+## Running Harbor (remote Docker note)
+
+Harbor bind-mounts local directories, so it must run where a Docker daemon lives. A laptop without a
+daemon can offload to a host: set
+
+```toml
+[harbor]
+host = "harbor-host.example"           # an SSH host with Docker + harbor
+remote_root = "/srv/touchstone"
+```
+
+and `touchstone bench` (via `touchstone/harbor/run.py`) rsyncs the dataset to that host, runs Harbor
+there over SSH, and rsyncs the job directory back. `scripts/harbor-mini.sh` is a thin wrapper for the
+same path. With a local Docker daemon, everything runs locally and these settings are ignored.
+
+## Install
+
+```
+uv tool install --from git+ssh://git@github.com/saml212/touchstone touchstone-bench
+```
+
+or, in a checkout, `uv run touchstone …`. Requires [Harbor](https://docs.harborframework.com)
+(`uv tool install harbor`) and a Docker daemon to run benchmarks.
+
+## Develop
+
+```
+uv run pytest -q
+uv run ruff check .
+```
+
+Commits are small and sequential; see `docs/DESIGN.md` for the full v3 design.

touchstone_bench-0.1.0/README.md

@@ -0,0 +1,148 @@
+# Touchstone
+
+A touchstone is the black stone assayers rubbed gold against to prove its purity before anyone paid
+for it. Touchstone does that for AI agents: it turns a company's running agent into a
+[Harbor](https://github.com/harbor-framework/harbor) benchmark of its own work, lets the people who
+know the product review that benchmark by voice, and turns the results into training data.
+
+Harbor is the middle of the product, unchanged. Touchstone is four extensions around it:
+
+1. **Capture** — one line of code records what the production agent does.
+2. **Survey** — a read-only agent reads the recordings and the code, copies the system into a Harbor
+   environment, and writes Harbor tasks against it.
+3. **Review** — a voice/text room where product people look at finished trials and correct the
+   verifiers; Harbor regrades after every correction.
+4. **Train** — turns finished jobs into distillation and RL datasets.
+
+Everything Touchstone produces is a plain Harbor dataset the customer owns and can run without us.
+Local-first: it runs on the customer's machine with their own Claude or Codex login — no Touchstone
+account, no Touchstone servers. Open source. By Pebble ML.
+
+Horizontal by design: nothing in `touchstone/harbor/` knows about any one customer, product, or
+domain.
+
+## The five commands
+
+```
+touchstone init                          # write touchstone.toml + the .touchstone/ db
+touchstone demo                          # run the built-in agent and capture episodes (zero-key)
+touchstone survey <repo>                 # read-only: map the code, simulate its services, score fidelity
+touchstone bench -m <provider/model>     # run a model over the dataset; print pass rate per task
+touchstone jobs                          # list Harbor job dirs with their pass rate per task
+touchstone serve  /  touchstone review     # open the review room (voice or text)
+touchstone train                         # turn finished jobs into distill + RL datasets
+```
+
+`touchstone survey` reads the recordings and the code with a read-only coding agent
+(`[survey] provider`, default `claude-cli`; the customer's login pays), maps the tools and their
+network boundaries, generates a SQLite-backed simulator per service, and replays every recorded call
+through the real tools to score each simulator's fidelity. It is idempotent — outputs are reused
+unless `--force` — and touches nothing outside `touchstone/` in the target repo.
+
+`touchstone bench` also takes `--agent packaged|replica`, `--dataset <dir>`, and
+`--against <job_dir>` (a per-task comparison against a previous run). `touchstone doctor` reports the
+environment. Capture is one line in your own app:
+
+```python
+import touchstone
+touchstone.trace()   # records every model + tool call to .touchstone/touchstone.db
+```
+
+Run your real agent on a different model with one env var: set `TOUCHSTONE_MODEL` and capture rewrites
+the `model=` keyword on every openai/anthropic/litellm call before it goes through — same SDK, same
+code path. (A model passed positionally is left untouched.) This is how the packaged agent under test
+is benched against a candidate model without editing the customer's code.
+
+## The dataset
+
+A survey produces a `touchstone/` directory in the customer's repo — a plain Harbor dataset they own
+and version:
+
+```
+touchstone/
+  dataset.toml      the Harbor dataset manifest (metadata; tasks run as the implicit tasks/ dataset)
+  tasks/<name>/     one Harbor task per directory (instruction.md, task.toml, environment/,
+                    solution/, tests/)
+  environment/      the customer's system, copied to run in a sandbox
+  agent/            the agent under test as a Harbor custom agent (agent.toml, tools.py, entry.py)
+  simulators/       a small local service per network boundary
+  baseline.json     what the current setup passes today (the first-five-minutes sentence)
+  report.md         what was mapped, simulated, and left open
+```
+
+The agent under test runs in one of two modes, chosen automatically and recorded in `agent/agent.toml`:
+
+- **packaged** (highest fidelity): the survey generates `agent/entry.py` — a `run(user_message)`
+  that drives the customer's *real* agent loop for one message, importing their own modules. The
+  Harbor agent runs `bash /app/agent/run.sh` in the sandbox with the model as a setting
+  (`TOUCHSTONE_MODEL`) and imports the trajectory the customer's own capture wrote. An adapter check
+  proves entry.py runs and calls a tool before this mode is chosen.
+- **replica** (fallback when the code will not run): `touchstone.harbor.agent:TouchstoneAgent` runs
+  an OpenAI-compatible tool-calling loop whose system prompt comes from `agent/agent.toml` and whose
+  tool schemas + dispatch come from `agent/tools.py`.
+
+Either way it records an ATIF `trajectory.json`, and `touchstone bench -m <candidate>` re-runs it on
+any model without touching the customer's code.
+
+## The review room
+
+`touchstone serve` then `touchstone review` opens a room (voice or text) where a product person walks
+the finished trials with an AI. It opens from the product's goal (the job labels and the baseline
+pass count), then picks trials in order — verifier unsure, models disagree, never reviewed, then
+gate failures — and for each reads the instruction, the trajectory in plain words, and every
+criterion's score, and asks "do you agree it passed?". Agreement records trust (the share of
+reviewed trials where the human agreed with the verifier, shown live). On a disagreement the agent
+drafts a criterion change — edit/add/remove a rewardkit check, a dimension weight, a judge line, or
+the instruction wording — reads it back, and on "yes" writes the `tests/` file and runs
+`harbor job regrade` (the tasks are authored with a separate verifier so grading reruns from the
+recorded artifacts, no agent), then reads out the new reward and any other trials that moved.
+"Always" applies the same criterion to every task with the same job. Everything the room decides is
+a row in `reviews` and a file change under `touchstone/` — nothing else.
+
+## Training data
+
+`touchstone train` reads the Harbor job directories (`trajectory.json` + `reward.json` per trial)
+and writes, under `touchstone/train/`, `distill.jsonl` (full trajectories from trials that scored at
+or above the threshold, for a student to copy), `rl_tasks.toml` (tasks in the learnability band —
+pass rate strictly between 0 and 1 — with the verifier as the reward), and `manifest.json` (per-task
+pass rate per model and where each task went). `touchstone train --teacher <spec>` runs the teacher
+job first. Training itself stops at the exact GPU command; Touchstone produces the data, not the run.
+
+## Storage
+
+Only what is captured lives in SQLite (`.touchstone/touchstone.db`): `episodes`, `spans`, `rooms`,
+`room_messages`, and `reviews` (what a review room decided about a task's trial). Harbor's own
+`jobs/` directories are the run record — Touchstone reads them, it has no runs table.
+
+## Running Harbor (remote Docker note)
+
+Harbor bind-mounts local directories, so it must run where a Docker daemon lives. A laptop without a
+daemon can offload to a host: set
+
+```toml
+[harbor]
+host = "harbor-host.example"           # an SSH host with Docker + harbor
+remote_root = "/srv/touchstone"
+```
+
+and `touchstone bench` (via `touchstone/harbor/run.py`) rsyncs the dataset to that host, runs Harbor
+there over SSH, and rsyncs the job directory back. `scripts/harbor-mini.sh` is a thin wrapper for the
+same path. With a local Docker daemon, everything runs locally and these settings are ignored.
+
+## Install
+
+```
+uv tool install --from git+ssh://git@github.com/saml212/touchstone touchstone-bench
+```
+
+or, in a checkout, `uv run touchstone …`. Requires [Harbor](https://docs.harborframework.com)
+(`uv tool install harbor`) and a Docker daemon to run benchmarks.
+
+## Develop
+
+```
+uv run pytest -q
+uv run ruff check .
+```
+
+Commits are small and sequential; see `docs/DESIGN.md` for the full v3 design.

touchstone_bench-0.1.0/benchmarks/survey/2026-09-25.json

@@ -0,0 +1,44 @@
+{
+  "generated_at": "2026-09-25T18:08:01.407612+00:00",
+  "targets": [
+    {
+      "name": "touchstone-example",
+      "repo": "/Users/samuellarson/Pebble/Github/touchstone-example",
+      "run": "python demo-style capture already shipped with the example (httpx, ORDERS_URL env base)",
+      "services": {
+        "orders_service": 1.0
+      },
+      "fidelity_min": 1.0,
+      "written": 7,
+      "gated": 7,
+      "baseline_pass": 1.0,
+      "agreement": 1.0
+    },
+    {
+      "name": "weather-gpt-assistant",
+      "repo": "/Users/samuellarson/Pebble/Github/survey-targets/weather-gpt-assistant",
+      "run": "python scripts/record.py against a local WeatherAPI fake (requests, hardcoded host via net shim)",
+      "services": {
+        "WeatherAPI": 1.0
+      },
+      "fidelity_min": 1.0,
+      "written": 6,
+      "gated": 6,
+      "baseline_pass": 1.0,
+      "agreement": 1.0
+    },
+    {
+      "name": "book-flight-llm",
+      "repo": "/Users/samuellarson/Pebble/Github/survey-targets/book-flight-llm",
+      "run": "python scripts/record.py against a local flight-backend fake (requests, FLIGHT_API_BASE_URL env)",
+      "services": {
+        "flight-booking-api": 1.0
+      },
+      "fidelity_min": 1.0,
+      "written": 4,
+      "gated": 4,
+      "baseline_pass": 1.0,
+      "agreement": 1.0
+    }
+  ]
+}

touchstone_bench-0.1.0/benchmarks/survey/2026-09-25.md

@@ -0,0 +1,9 @@
+# Survey benchmark
+
+Generated 2026-09-25T18:08:01.407612+00:00. Each target was recorded, then surveyed; this scores the outputs.
+
+| target | services (fidelity) | tasks written/gated | baseline pass | agreement |
+|---|---|---|---|---|
+| touchstone-example | orders_service 100% | 7/7 | 100% | 100% |
+| weather-gpt-assistant | WeatherAPI 100% | 6/6 | 100% | 100% |
+| book-flight-llm | flight-booking-api 100% | 4/4 | 100% | 100% |

touchstone_bench-0.1.0/benchmarks/survey/targets.toml

@@ -0,0 +1,21 @@
+# The survey agent's benchmark targets. Each was recorded against a local fake of its external
+# service (with openai/gpt-4o-mini) so .touchstone/ holds real conversations, then surveyed with
+# `touchstone survey <repo>`. `touchstone survey-bench <this file>` scores the resulting outputs.
+# `run` documents how each target's conversations were recorded (not executed by survey-bench).
+# The two open-source clones live under ~/Pebble/Github/survey-targets/ as gitignored scratch and
+# are never redistributed; book-flight-llm ships no license, so it is a private test target only.
+
+[[target]]
+name = "touchstone-example"
+repo = "~/Pebble/Github/touchstone-example"
+run = "python demo-style capture already shipped with the example (httpx, ORDERS_URL env base)"
+
+[[target]]
+name = "weather-gpt-assistant"
+repo = "~/Pebble/Github/survey-targets/weather-gpt-assistant"
+run = "python scripts/record.py against a local WeatherAPI fake (requests, hardcoded host via net shim)"
+
+[[target]]
+name = "book-flight-llm"
+repo = "~/Pebble/Github/survey-targets/book-flight-llm"
+run = "python scripts/record.py against a local flight-backend fake (requests, FLIGHT_API_BASE_URL env)"