touchstone-eval 0.1.0__tar.gz

touchstone_eval-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,59 @@
+name: Publish to PyPI
+
+# Publishes touchstone-eval to PyPI when a version tag is pushed, e.g.:
+#   git tag v0.1.0 && git push origin v0.1.0
+#
+# Uses PyPI Trusted Publishing (OIDC) — no API token / password stored as a secret.
+# One-time setup on PyPI: project Settings -> Publishing -> add a GitHub publisher with
+#   owner: krimvp   repo: touchstone   workflow: publish.yml   environment: pypi
+# (Create the "pypi" environment under the repo's Settings -> Environments first, or use
+#  PyPI's "pending publisher" flow to register it before the project's first release.)
+
+on:
+  push:
+    tags:
+      - "v*"
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    name: Build sdist + wheel
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Build distributions
+        run: uv build
+
+      - name: Check metadata
+        run: uvx twine check dist/*
+
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    name: Publish to PyPI
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/touchstone-eval
+    permissions:
+      id-token: write # required for Trusted Publishing (OIDC)
+    steps:
+      - name: Download built distributions
+        uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@release/v1

touchstone_eval-0.1.0/.gitignore

@@ -0,0 +1,29 @@
+# Run outputs (intermediate + final results) — large and machine-specific.
+runs/
+
+# Transient run logs
+*.log
+
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+.pytest_cache/
+.ruff_cache/
+
+# Env / secrets
+.env
+
+# Private held-out eval set — your OWN usecases, never committed (contamination-proof
+# tiebreaker; Lessons 3 & 6). Run with:  touchstone --evals-dir evals-private run
+# The README + example template stay tracked; real private cases are ignored.
+evals-private/*
+!evals-private/.gitignore
+!evals-private/README.md
+!evals-private/example-private-case
+!evals-private/example-private-case/**

touchstone_eval-0.1.0/CONTEXT.md

@@ -0,0 +1,184 @@
+# touchstone
+
+A personal eval benchmark for deciding which model works best for the user's own
+usecases. This glossary fixes the language the framework and its docs use.
+
+## Language
+
+### Benchmark structure
+
+**Case**:
+One eval — a task plus the input, artifacts, graders, and expectations needed to judge it.
+_Avoid_: test, suite, scenario.
+
+**Run**:
+A single execution of the benchmark that expands a matrix and produces a report.
+_Avoid_: session, job.
+
+**Cell**:
+The atomic unit of work and persistence: one `(Case × Harness × Model × Trial)`.
+_Avoid_: task-run, instance.
+
+**Trial**:
+One repeated attempt of the same Cell coordinates, used for consistency / pass@k.
+_Avoid_: attempt, sample.
+
+**Grader**:
+A component that turns a Harness's result into a Score.
+_Avoid_: judge (reserve "judge" for the model-as-judge grader specifically), scorer, checker.
+
+**Harness**:
+The swappable thing that turns a Case's task into an output. Every Harness is an adapter
+behind one interface.
+_Avoid_: runner, agent, driver.
+
+### Observation & interaction
+
+**Trace**:
+The normalized, vendor-neutral event stream captured from a Harness run (messages, tool
+calls, token usage, etc.). The framework's own schema — never an external protocol's types.
+_Avoid_: log (reserve for raw stdout/transcript), transcript (that is the raw capture).
+
+**Trace Event**:
+One item in a Trace (e.g. a tool call, a token-usage update, a permission request).
+_Avoid_: span (reserve "span" for the future LangFuse mapping), record.
+
+**Tracing** (a Harness capability):
+A Harness's ability to emit a Trace. Opt-in; Harnesses that lack it degrade to output-only.
+_Avoid_: instrumentation, observability (use as adjectives, not as the capability name).
+
+**Interaction** (a Harness capability):
+A Harness's ability to let the framework answer the agent's mid-run requests (tool
+permission / approval / input). Strictly richer than Tracing.
+_Avoid_: feedback, callback, HITL.
+
+**Output-only**:
+A Harness that exposes neither Tracing nor Interaction — only its final output. Today's
+default and the universal fallback.
+_Avoid_: black-box, basic.
+
+**Tool Kind**:
+The portable, canonical category of a tool call (`read | write | execute | search |
+fetch | other`) — the one tool axis that means the same across agents. Distinct from a
+tool's `raw_name` (verbatim from the agent) and `name` (the Adapter's normalized name).
+_Avoid_: tool type, category.
+
+**Interaction Policy**:
+The per-Case rule that answers *agent-initiated* mid-run requests. One of `auto-approve`,
+`auto-deny`, `scripted`, `llm-based`, or `manual`.
+_Avoid_: handler, strategy.
+
+**Turn**:
+One *eval-initiated* prompt sent to the agent within a Cell. Distinct from an
+agent-initiated request (which the Interaction Policy answers) and from a Trial (a repeat
+of the whole Cell). The first Turn is the Case's task; later Turns are scripted follow-ups.
+_Avoid_: round, step, message.
+
+**Conversation**:
+The ordered Turns of a Case, sent one after another, each dispatched once the agent's
+previous Turn reaches a `stop`. A single-prompt Case is a one-Turn Conversation.
+_Avoid_: dialogue, thread, chat.
+
+**Responder**:
+The fixed auxiliary LLM that answers the agent's mid-run requests under Case guidelines
+when the Interaction Policy is `llm-based`. A control variable, held constant across the
+matrix like the Judge, so the agent stays the only thing being compared.
+_Avoid_: helper, user-sim, proxy.
+
+**Judge**:
+The fixed auxiliary LLM used by the model-as-judge Grader to score output. Like the
+Responder, a control variable held constant across the matrix.
+_Avoid_: grader (the Judge is used *by* a Grader, not a synonym for one).
+
+**Adapter**:
+A concrete Harness implementation. The **ACP adapter** is the single rich adapter — one
+implementation driving every Agent-Client-Protocol agent (Claude via `claude-agent-acp`,
+Codex, Gemini, droid, devin-cli, …) through one event-translation path. The **CLI adapter**
+is the generic output-only fallback. (A native **Claude Agent SDK adapter** is an optional
+future no-Node alternative, not part of the core.)
+_Avoid_: backend, plugin, connector.
+
+### Execution & isolation
+
+**Sandbox**:
+The isolated working directory a Cell's Harness operates in, prepared fresh from the Case
+source. Self-contained: own directory, own subprocess env, never shared between Cells.
+_Avoid_: workspace, workdir (informal synonyms only).
+
+**Isolation Mode**:
+How a Sandbox is created from the source — `copy` (copy a folder), `clone` (git clone at a
+commit), or `worktree` (git worktree at a commit). Inferred from source type and
+overridable; `worktree` is opt-in, not the default.
+_Avoid_: sandbox type, strategy.
+
+**Environment**:
+A Cell's own throwaway dependency setup (the *broader Sandbox*), provisioned when a Case
+declares an `environment` block. A **Provisioner** (selected by `kind`) prepares it:
+`pip-venv` (default) / `uv` build a per-Cell virtualenv and install declared dependencies
+(and optionally the Sandbox repo via `install: editable`); `command` runs declared shell
+commands for ecosystems with project-local deps (`npm ci`, `cargo fetch`). Every subprocess
+the Cell spawns (Harness, setup, `command`/`tests`/`pytest` Graders) runs under it via an
+explicit env, never a shared global interpreter. Absent the block, the Cell uses the host.
+_Avoid_: virtualenv (one Provisioner's mechanism, not the concept), image, container (no
+OS-level isolation *yet* — see Executor).
+
+**Provisioner**:
+The strategy that prepares a Cell's Environment, selected by `environment.kind` (`pip-venv`,
+`uv`, `command`). Mirrors Isolation Mode for the Sandbox: one declarative knob, multiple
+backends, one contract (return the subprocess `env`, or `None` for host).
+_Avoid_: installer, builder.
+
+**Executor**:
+Where a Cell's non-Harness commands run, behind one `run(argv, cwd, env)` + `create_venv`
+interface. `LocalExecutor` runs host subprocesses; `ContainerExecutor` runs them via
+`docker exec` in a container with the cell bind-mounted (selected by a Case's `container`
+block), bringing OS-level isolation and OS packages. Provisioning, `setup.run`, and the
+`command`/`tests`/`pytest` Graders run through the Cell's Executor; the Harness still runs on
+the host (ADR 0005). The provisioner recipes are written once and run under either backend.
+_Avoid_: runner (that is the orchestration loop), shell, backend (use as adjective only).
+
+**Reachability / Availability**:
+Whether this host can reach a Case's external git repos (its remote `source` and/or
+`fixtures`). A preflight probes each (access-level `git ls-remote`, cached per URL) before the
+Run does work, then applies the **availability policy**: `fail` (default — a single unreachable
+*required* Case aborts the Run) or `skip` (degrade unreachable Cases to the `skipped` status and
+continue). A Case marked `availability: optional` always degrades. `skipped` is terminal and
+excluded from every aggregate — distinct from `failed`, which is a defect. See ADR 0008.
+_Avoid_: offline (a probe failure may be auth, not network), error (a skip is not a failure).
+
+## Relationships
+
+- A **Run** expands into many **Cells**; each **Cell** has one **Harness**, one model, one **Trial** index.
+- A **Harness** is realized by exactly one **Adapter**; an **Adapter** declares its **Tracing** and **Interaction** capabilities.
+- A **Tracing**-capable **Harness** produces a **Trace** (a sequence of **Trace Events**) per **Cell**, alongside the raw transcript.
+- **Interaction** implies **Tracing** (you cannot answer requests you cannot observe), not vice-versa.
+- **Graders** may read the final output, the **Trace**, or both.
+- A **Trace** is the source mapped to LangFuse spans later — graders and LangFuse both depend on the **Trace**, never on ACP or the Claude SDK directly.
+- A **Case** opts into observation via an `observe` block (Tracing and/or Interaction); absent it, the **Cell** is **Output-only**. A Run-level flag can override.
+- When a **Case** requests more than its **Adapter** supports, the **Cell** soft-degrades (empty **Trace**, warning recorded) — unless a **Grader** needs the **Trace**, which is a hard failure for that **Cell**.
+- An **Interaction Policy** of `llm-based` uses a **Responder**; deterministic policies (`auto-approve`/`auto-deny`/`scripted`) use none. `manual` is non-reproducible and excluded from aggregation; `llm-based` is included but flagged responder-mediated.
+- Every mid-run request and its answer is recorded in the **Trace** (`permission_request` / `permission_response`), whatever the **Interaction Policy**.
+- A **Case** is a **Conversation** of one or more **Turns**; **Turns** are eval-initiated, while requests answered by the **Interaction Policy** are agent-initiated. Both happen within one **Cell**.
+- Each **Cell** gets its own **Sandbox** via an **Isolation Mode**; all modes yield a fully isolated, parallel-safe directory. Commit-pinned modes (`clone`/`worktree`) give reproducibility.
+- A **Cell** that declares an **Environment** also gets its own venv beside the **Sandbox**; the venv (and its installed dependencies) is the *broader Sandbox* that keeps dependency-bearing Cells reproducible and parallel-safe — provisioned before the agent runs, torn down with the **Sandbox**.
+- A **Cell**'s outcome lives in its own `result.json` (the source of truth); the run manifest is a derived index merged from those, so parallel **Cells** never write the same file.
+- Before a **Run** does work it checks **Reachability** of every **Case**'s external repos and applies the availability policy; unreachable **Cases** either abort the Run (`fail`) or become `skipped` **Cells** (`skip`/`optional`), which are excluded from every aggregate — a missing private repo can never silently shrink the benchmark.
+- Model selection is **Adapter-specific**: a CLI flag (CLI adapter) or applied **via the ACP instruction** — launch arg and/or `session/new` / `session/setConfigOption` (ACP adapter). A model string is opaque to the framework and may be a custom alias (e.g. `glm-5.1:cloud`).
+- A model is meaningful only relative to a **Harness**, so the matrix **pairs models per-Harness** (entries of `{harness, models}`) rather than taking a blind cross-product.
+
+## Example dialogue
+
+> **Dev:** "Does the ACP **Adapter** give us **Interaction**?"
+> **User:** "Yes — ACP's `request_permission` lets us answer the agent, so that **Adapter** is **Interaction**-capable. The generic **CLI Adapter** is **Output-only**."
+> **Dev:** "And aider today?"
+> **User:** "**Output-only** — no ACP, no SDK. We still observe its final output; the **Trace** is just empty of tool events."
+
+## Flagged ambiguities
+
+- "ACP" was used as if it were the abstraction. Resolved: ACP is one **Adapter**'s transport;
+  the abstraction is the **Trace**. The Claude Agent SDK populates the same **Trace** without ACP.
+- "wrap the calls" meant two distinct capabilities — **Tracing** (observe) and **Interaction**
+  (respond). Resolved: they are separate, with **Interaction** implying **Tracing**.
+- "tool name" was treated as one thing. Resolved into three: `raw_name` (verbatim, never
+  lost), `name` (Adapter-normalized), and **Tool Kind** (portable enum). Cross-model grading
+  prefers **Tool Kind**; within-agent grading may use `raw_name`.

touchstone_eval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 krimvp
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

touchstone_eval-0.1.0/PKG-INFO

@@ -0,0 +1,343 @@
+Metadata-Version: 2.4
+Name: touchstone-eval
+Version: 0.1.0
+Summary: Personal eval benchmark: compare model outcomes across swappable CLI-agent harnesses on custom tasks.
+Project-URL: Homepage, https://github.com/krimvp/touchstone
+Project-URL: Repository, https://github.com/krimvp/touchstone
+Project-URL: Issues, https://github.com/krimvp/touchstone/issues
+Author-email: krimvp <anton.balboa@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: acp,agent,benchmark,claude-code,cli,eval,evaluation,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: judge
+Requires-Dist: anthropic>=0.40; extra == 'judge'
+Provides-Extra: langfuse
+Requires-Dist: langfuse>=2.0; extra == 'langfuse'
+Description-Content-Type: text/markdown
+
+# touchstone
+
+> A *touchstone* is the dark stone jewelers rub gold against to read its purity from the
+> streak it leaves — telling true gold from convincing fakes. That is this benchmark's whole
+> job: telling apart models that look identical on paper, by the marks they leave on real work.
+
+A personal eval benchmark for answering one question: **for my usecases, which model
+works best?**
+
+Each eval (a *case*) bundles its own task, its own input source files, its own AI
+artifacts (skills / commands / plugins / MCP), and its own definition of a correct
+outcome. A *run* executes a **matrix** of cells — one cell per
+`(case × harness × model × trial)` — fully isolated and persisted independently, then
+aggregates everything into a single report.
+
+## Core model
+
+```
+Case (one eval)            Matrix axes            Cell (unit of work + persistence)
+  task / prompt      ×   harnesses[]        =     sandbox + transcript + output
+  source/ files          models[]                 + grader scores + metrics + status
+  artifacts/             trials (k)
+  graders[]
+```
+
+- **Harness** — the swappable thing that turns a task into an output, behind one interface
+  (`harness/base.py`). `echo` (fake) and `claude-code` are output-only. For *rich* runs
+  (a Trace of tool calls / tokens / cost) there are two paths: **`claude-code-stream`** drives
+  Claude natively over `--output-format stream-json` (no ACP, no Node; Tracing-only,
+  autonomous via skip-permissions — see `docs/adr/0006`), and the **ACP adapter** drives any
+  Agent Client Protocol agent (droid, gemini, codex, claude-acp, devin-cli) with full
+  observation **and** bidirectional interaction. ACP is one rich path, not the only one — the
+  Trace is the contract.
+- **Graders** — `command` (run tests/build), `files` (expected files / grep patterns),
+  `model_judge` (LLM-as-judge), and `trace` (assert over observed tool usage / token &
+  cost budgets). All run; combined per the case's `expect.pass_threshold`.
+- **Observation & interaction** (opt-in per case via `observe:`) — capture a normalized
+  **Trace** (tool calls, tokens, cost, permission events) and answer the agent's mid-run
+  requests with an **Interaction Policy** (`auto-approve`/`auto-deny`/`scripted`/
+  `llm-based`/`manual`). See `CONTEXT.md` + `docs/adr/`.
+- **Resumability & parallelism** — each cell's `result.json` is the source of truth (the
+  manifest is a derived index), so cells run in parallel (`--workers`) without contention
+  and `run --resume <id>` continues after a crash.
+
+## Install
+
+The published package is `touchstone-eval`; the command it installs is `touchstone`
+(the bare `touchstone` name on PyPI belongs to an unrelated, abandoned project).
+
+```bash
+uvx touchstone-eval --help          # run without installing (recommended)
+pipx install touchstone-eval        # or install as an isolated tool
+pip install touchstone-eval         # or into the current environment
+```
+
+Add the optional extras when you need them: `touchstone-eval[judge]` (Anthropic SDK for
+`model_judge`), `[langfuse]` (export), `[dev]` (pytest).
+
+For local development from a checkout:
+
+```bash
+pip install -e ".[judge,dev]"   # judge = Anthropic SDK for model_judge; dev = pytest
+```
+
+## Usage
+
+```bash
+touchstone validate                 # schema-check every evals/<case>/case.yaml
+touchstone list                     # list cases and past runs
+touchstone run                      # run the whole evals/ suite
+touchstone run --eval example-case --harness echo --trials 2
+touchstone run --harness droid --with-model A --with-model B  # compare models, same harness
+touchstone run --workers 4          # run cells in parallel
+touchstone run --resume <run_id>    # continue an interrupted run
+touchstone report <run_id>          # (re)generate runs/<run_id>/report.md
+touchstone export <run_id> [--push] # write runs/<id>/langfuse.json (and optionally push)
+```
+
+### Comparing models on the same harness
+
+The matrix is what answers "which model for my usecases?" — distinct models become
+distinct cells, and the report ranks them in a per-case matrix + a leaderboard (score,
+cost, time, tools, tokens). A case can declare the models inline
+(`matrix.models` / `matrix.entries[].models`), or you can hold a harness fixed and push
+models through it at run time without editing the case:
+
+```bash
+# Run these models on droid even if the cases declared only one — they replace the
+# case's models for that harness. Each becomes its own row in the comparison.
+touchstone run --harness droid \
+  --with-model custom:glm-5.1:cloud-0 --with-model custom:glm-4.6:cloud
+```
+
+`--with-model` *replaces* the declared models (so you can introduce new ones); `--model`
+only *filters* the models a case already declares. Models are agent-specific opaque
+strings, so prefix `HARNESS=` (`--with-model droid=A`) to scope an override to one harness
+when a run spans several.
+
+ACP agents are configured in `acp_agents.yaml` (see `acp_agents.yaml.example`); the
+built-in profiles (`droid`, `gemini`, `codex`, `claude-acp`, `devin-cli`) work out of the
+box once the agent's CLI is on `PATH`. `evals/observed-droid/` is a worked example of a
+fully observed, interactive, multi-turn case.
+
+Real harnesses (e.g. `claude-code`) cost money and require their CLI on `PATH`.
+The built-in `echo` harness runs the full loop with no network/API spend — use it for
+testing the framework itself.
+
+## Defining a case
+
+See `evals/example-case/case.yaml` for a worked example. Schema:
+
+```yaml
+id: my-case
+description: ...
+task:
+  prompt: |
+    What the model/agent must accomplish.
+source:                  # optional; copied fresh into every cell sandbox
+  path: ./source         # ...or  {repo: owner/name, commit: <sha>}  (pinned clone)
+  # repo form may add `subdir: <dir>` to use just one sub-directory of the clone as the
+  # sandbox — lets one fixtures repo hold many cases (see "Source fixtures repo" below).
+artifacts:               # optional AI artifacts injected into the harness
+  skills:   [./artifacts/skills/foo]
+  commands: [./artifacts/commands/bar.md]
+  mcp:      ./artifacts/.mcp.json
+environment:             # optional per-cell dependency setup (the "broader sandbox")
+  kind: pip-venv               # pip-venv (default) | uv | command  — how deps are provisioned
+  requirements: [markupsafe]   # (pip-venv/uv) installed into an isolated venv per cell
+  install: editable            # (pip-venv/uv) `pip install -e .` (src-layout pkg + its deps)
+  # kind: command → run shell installs for project-local ecosystems, e.g.
+  #   commands: ["npm ci"]      # node_modules / target/ etc. live in the sandbox
+setup:                   # optional; introduce the task state after clone, before the agent
+  stub: [{file: pkg/mod.py, function: target}]   # blank a fn body -> NotImplementedError
+  run:  ["rm -rf .git"]                           # shell commands in the sandbox
+matrix:
+  harnesses: [claude-code]
+  models:    [opus, sonnet, haiku]
+  trials:    3
+graders:
+  - {type: command, cmd: "pytest -q", weight: 1.0}
+  - {type: files, patterns: ["retry", "backoff"]}
+  - {type: model_judge, rubric: ./graders/rubric.md, model: opus, pass_threshold: 0.8}
+expect:
+  pass_threshold: 1.0
+```
+
+### Source fixtures repo
+
+A case's bulky, hand-written assets — synthetic codebases to debug **and** the `hidden/`
+oracle test suites — live **out of this repo**, in a separate fixtures repo
+([`krimvp/touchstone-eval-fixtures`](https://github.com/krimvp/touchstone-eval-fixtures)), so they
+don't pollute the runner/eval tree. The eval repo keeps only the *contract* (task, graders,
+expectations); the fixtures repo holds the code. Each case has one directory there, split by
+**visibility**:
+
+```
+<case-id>/
+  source/   # agent-VISIBLE input  → promoted to the sandbox before the agent runs
+  hidden/   # grader ORACLE         → injected at grade time only; the agent never sees it
+```
+
+A case wires the two halves with two independent pins (both default-pinned by commit):
+
+```yaml
+source:    {repo: krimvp/touchstone-eval-fixtures, commit: <sha>, subdir: <case-id>/source}
+fixtures:  {repo: krimvp/touchstone-eval-fixtures, commit: <sha>}   # subdir defaults to <case-id>
+graders:
+  - {type: pytest, inject: ["./hidden/test_x.py"]}   # resolved under <case-id>/hidden/
+```
+
+- **`source`** clones the repo, checks out the commit, and promotes `<case-id>/source/` into
+  the sandbox (no `.git`, like `copy`). SWE-bench-style cases point `source` at the *real
+  upstream repo* instead, so they have only a `hidden/` in the fixtures repo (no `source/`).
+- **`fixtures`** names the repo that graders resolve `inject:` paths against — `Case.asset()`
+  pulls each hidden file from a host-cached clone (`src/touchstone/fixtures.py`), at grade
+  time, *after* the agent has stopped. Because `source/` and `hidden/` are sibling directories
+  and only `source/` is promoted, the oracle can never leak into the agent's sandbox.
+
+Keep the fixtures repo **private** for the anti-memorization cases. `evals/example-case/`
+stays local (`source: path`) as the offline worked example / integration fixture.
+
+### Real-repo (SWE-bench-style) cases
+
+A case can pin a real GitHub repo at a commit (`source: {repo, commit}`), `setup.stub` a
+function to blank its body, and inject **hidden tests** (oracle = the real function) only
+at grade time — so the agent reimplements real library code and the `pytest` grader scores
+the fraction of FAIL→PASS tests. See `evals/repo-*-droid/`.
+
+When a repo needs third-party dependencies or isn't importable from its root (a `src/`
+layout), declare an **`environment`**: each cell gets its own throwaway virtualenv, into
+which `requirements` are pip-installed and — with `install: editable` — the repo itself
+(`pip install -e .`, which resolves a src-layout package and pulls its deps). Every
+subprocess the cell spawns (harness, setup, and the `command`/`pytest` graders) runs under
+that venv via an explicit env, so dependency-bearing cases stay reproducible and
+parallel-safe (no shared site-packages). Worked examples: `repo-smarttruncate-droid`
+(a `requirements` dep) and `repo-securefilename-droid` (`install: editable`, src-layout).
+
+### Non-Python projects
+
+Cases aren't Python-specific. The `command`, `files`, `model_judge`, and `trace` graders
+are language-agnostic, and the **`tests`** grader gives the same partial-credit scoring as
+`pytest` for any runner whose results it can read. Two substrates, **XML primary with a
+console fallback**:
+
+- **JUnit XML** (`junit_xml: <glob>`) — the universal report format every framework/build
+  tool can emit (Maven Surefire, Gradle, pytest `--junitxml`, vitest/jest/mocha reporters,
+  `go-junit-report`, `cargo2junit`). Deterministic, exact per-test counts, framework-agnostic.
+- **Console summary** (`_parse_counts`) — scraped when no XML report is produced: pytest/
+  unittest, `node --test`/TAP, Maven Surefire, **`go test -v`** (`--- PASS:`/`--- FAIL:`), and
+  **`cargo test`** (`test result: … N passed; M failed`).
+
+A `tests` grader with `gate: true` is a validity **gate** (never adds credit; disqualifies the
+cell to 0 on failure) — use it to mirror SWE-bench's PASS_TO_PASS regression gate in any
+language. `inject` takes either a bare filename (dropped at the sandbox root) or `{src, dest}`
+to place a hidden test at a runner-specific path (e.g. Maven's `src/test/java/...`). Use
+`setup.run` to blank the function (the AST-based `setup.stub` is Python-only); the
+`implemented` gate works on any language when pointed at explicit `files`. Worked examples:
+`repo-js-wordwrap-droid` (CommonJS, `node --test`), `repo-java-camelcase-droid` (Maven,
+Surefire), and the `repo-swebench-*` battery — real recent GitHub issues across Python, Go
+(`go test`), Java (Surefire + JUnit XML), JS/TS (mocha/ava/TAP), and Rust (`cargo test`).
+
+**Dependencies aren't special — how they're *isolated* is.** Real projects have
+dependencies; the question is only whether installing them safely needs the `environment`
+venv. It depends on where the ecosystem puts deps:
+
+| Ecosystem | Where deps go | Isolation | How to declare |
+| --- | --- | --- | --- |
+| Python | shared `site-packages` (mutable) | needs the per-cell venv | `environment:` `kind: pip-venv` (or `uv`) + `requirements` / `install: editable` |
+| Node / Rust / Go | project-local (`node_modules`, `target/`, build cache) | per-cell for free | `environment:` `kind: command` + `commands: ["npm ci"]` etc. |
+| Java / Maven | shared `~/.m2` (versioned, immutable artifacts) | safe to share across cells | resolved by the build (`mvn test`) |
+
+The `environment.kind` is the one declarative knob (mirroring the Sandbox's Isolation Mode):
+`pip-venv` and `uv` build an isolated venv and install into it; `command` runs your install
+commands for ecosystems whose deps are project-local.
+
+### OS-level isolation + OS packages (containers)
+
+For cases that need OS packages or a pinned, reproducible build/grade environment, declare a
+**`container`**: provisioning, `setup.run`, and the `command`/`tests`/`pytest` graders then run
+inside it (via `docker exec`), with the cell bind-mounted at its same path.
+
+```yaml
+container:
+  image: python:3.12-slim          # pin by digest (…@sha256:…) for full reproducibility
+  setup: ["apt-get update -qq", "apt-get install -y -qq libxml2"]   # OS packages, once at start
+  caches: [".cache/pip"]           # share the host's cache so cells don't re-download deps
+environment:
+  kind: pip-venv                   # the venv is now built *inside* the container
+  requirements: [lxml, pytest]
+graders:
+  - {type: pytest, inject: ["./hidden/test_x.py"], weight: 4.0}     # runs in the container
+```
+
+`caches` mounts a home-relative dir (e.g. `.cache/pip`, `.m2`) shared with the host and
+across cells, so a fresh container per cell reuses already-downloaded dependencies instead
+of re-fetching them — the same shared-cache benefit the host's `~/.m2` gives today. The
+suite uses this on its dependency-bearing cases: `repo-js-wordwrap` (`node:20-slim`,
+zero-dep), `repo-smarttruncate` / `repo-securefilename` (`python:3.12-slim` + pip cache),
+and `repo-java-camelcase` (`maven:3.9-eclipse-temurin-21` + shared `~/.m2`).
+
+Every provisioner and grader runs through the Cell's **Executor** — `LocalExecutor` (host
+subprocess) by default, `ContainerExecutor` when a `container` is declared — so the same
+recipe runs under either backend (needs the docker daemon running). The Harness (the agent
+under test) still runs on the host against the bind-mounted Sandbox; running the agent
+itself in-container is future work. See `docs/adr/0005`.
+
+So the earlier zero-dep examples were picked to keep the *demo* offline, not because deps
+are rare. `repo-java-camelcase-droid` is a genuinely dependency-bearing non-Python case:
+commons-text's source needs `commons-lang3`, which Maven resolves from Maven Central.
+
+## Bring your own private repos (reachability & fallback)
+
+`touchstone` is an **engine + a public sample battery**. The verdict you can actually trust for
+"which model is best **for me**" comes from *your own* tasks, so the design is built to pull
+case material from external git repos you own — both the agent-visible `source: {repo, commit}`
+and the hidden oracle in `fixtures: {repo, commit}` — some of them private. Auth is just your
+normal git credentials (SSH agent / `gh` / a credential helper); nothing extra to configure.
+
+Because a given host may not have access to every referenced repo (a teammate's private
+fixtures, a CI box without keys, an offline laptop), a run **probes each case's external repos
+before doing any work** (`git ls-remote`, cached per URL) and applies a policy:
+
+```bash
+touchstone run                                  # default: FAIL FAST if any required repo is unreachable
+touchstone run --on-unavailable skip            # degrade: skip unreachable cases, run the rest
+touchstone validate --check-access              # preflight only: report what a run would skip/fail on
+```
+
+- **Fail by default.** A missing repo on a host you expected to be complete is a *loud, early*
+  error — never a silently smaller benchmark (which would corrupt cross-model comparisons).
+- **`--on-unavailable skip`** degrades the unreachable cases to a `skipped` status: excluded
+  from every score and the leaderboard, surfaced in a "Skipped (unavailable)" report section,
+  and **not** counted as failures. Resume re-probes, so a transient outage is retried.
+- **Per-case `availability: optional`** marks a case that may reference a repo you might not
+  have — it degrades to `skipped` even under the default fail mode.
+- Only *access* failures (no auth / no network / not found) are degradable; a bad commit or
+  schema error is a defect and still fails loudly.
+
+A fork can repoint the default hidden-fixtures repo to its own private one without editing every
+case by setting `TOUCHSTONE_FIXTURES_REPO=owner/my-fixtures`. Your fully-private held-out suite
+lives in `evals-private/` (gitignored) and runs with `--evals-dir evals-private` — see its
+README. Design: `docs/adr/0008-reachability-and-availability-policy.md`.
+
+## Layout
+
+```
+evals/<case>/        the benchmark suite (one dir per case)
+src/touchstone/     the framework (config, harness/, grader/, runner, report, cli)
+runs/<run_id>/       results (gitignored): manifest.json + cells/ + report.md
+```