woods 1.6.1 → 2.0.0.beta2

data/docs/DOCKER_SETUP.md

@@ -0,0 +1,454 @@
+# Docker Setup Guide
+
+This guide covers running Woods in a Dockerized Rails application, extraction, MCP server configuration, and troubleshooting.
+
+## Architecture Overview
+
+Woods extraction requires a booted Rails environment, so it runs inside the application container. The simplest MCP setup runs the Index Server from that same application bundle through Docker stdio. A host-side Index Server is an optional optimization when the host also has a compatible Ruby bundle and can read the published index.
+
+```
+HOST                                    APPLICATION CONTAINER
+─────────────────────────────           ───────────────────────────
+MCP client                              Rails App + Woods bundle
+  docker compose exec -T ────────────▶    woods-mcp /app/tmp/woods
+                                          29 schemas; 14 registered
+                                          reads the published index
+
+Extraction commands ─────────────────▶   bundle exec rake woods:extract
+                                          writes /app/tmp/woods/
+
+Console Server, two launch paths:
+
+  Embedded (9 tools)                    rake woods:console
+  MCP client spawns via                   boots Rails, runs MCP in-process
+  docker compose exec -T ────────────▶    Tier 1 read-only tools only
+
+  Configured launcher (same tools)
+  woods-console-mcp on host          rake woods:console
+  execs docker exec -i ──────────────▶  same embedded server
+```
+
+The Index Server reads static files and does not boot Rails, even when its process runs in the application container. The Console Server queries live application state and does boot Rails. Treat Console access as a separate security decision.
+
+## Installation
+
+### 1. Add the gem
+
+```ruby
+# Gemfile
+group :development do
+  gem 'woods', '~> 2.0'
+end
+```
+
+```bash
+docker compose exec app bundle install
+```
+
+### 2. Run the install generator
+
+```bash
+docker compose exec app bundle exec rails generate woods:install
+```
+
+This creates `config/initializers/woods.rb` with default configuration.
+
+### 3. Decide whether to keep the legacy application migration
+
+The generator emits `db/migrate/*_create_woods_tables.rb`, but Woods 2's shipped structural index and storage backends do not use those application tables. For a new default installation, remove that generated migration before the next Rails boot. Keep and run it only when deliberately preserving an older/custom integration that uses `woods_units`, `woods_edges`, and `woods_embeddings`, after normal schema-change review.
+
+### 4. Configure
+
+Edit `config/initializers/woods.rb` inside the container (or on the host if the app directory is volume-mounted):
+
+```ruby
+Woods.configure do |config|
+  config.output_dir = Rails.root.join('tmp/woods')
+end
+```
+
+## Extraction
+
+Run extraction inside the container:
+
+```bash
+# Full extraction
+docker compose exec app bundle exec rake woods:extract
+
+# Incremental (changed files only)
+docker compose exec app bundle exec rake woods:incremental
+
+# Automatic structural updates while the container is running
+docker compose exec app bundle exec rake woods:watch
+
+# Framework/gem sources only
+docker compose exec app bundle exec rake woods:extract_framework
+```
+
+Run the watcher as its own development service or process-manager entry, not as a one-off terminal command. Docker Desktop bind mounts may not deliver reliable native filesystem events; set `WOODS_WATCH_POLL=1` for polling when needed. The watcher updates structural generations automatically, while semantic vectors still require `woods:embed_incremental`.
+
+### Index persistence
+
+Persist `tmp/woods/` if the index should survive container replacement. A bind mount also makes it available to optional host-side tools:
+
+```yaml
+services:
+  app:
+    volumes:
+      - .:/app                    # Full app mount, output lands at ./tmp/woods/
+      # OR mount just the output:
+      # - ./tmp/woods:/app/tmp/woods
+```
+
+### Verify the published index
+
+Use Woods' generation-aware checks inside the same environment that performed extraction:
+
+```bash
+docker compose exec app bundle exec rake woods:validate
+docker compose exec app bundle exec rake woods:stats
+```
+
+Woods 2 publishes `generation.json`, which points at the active payload manifest. A root-level `manifest.json` is valid only for a legacy layout; do not use its presence as the v2 health check.
+
+### Path Translation
+
+Use the path visible to the process that actually runs the command:
+
+| Context | Path | Example |
+|---------|------|---------|
+| Rake tasks (inside container) | Container path | `/app/tmp/woods` |
+| Index Server through Docker | Container path | `/app/tmp/woods` |
+| Optional host-side Index Server | Host path | `./tmp/woods` or `/home/dev/my-app/tmp/woods` |
+
+## Index Server setup
+
+### Default: run it through the application container
+
+This path needs no Ruby, Bundler, Woods executable, or host-visible index on the host. Compose must be able to resolve the project, so set `cwd` to the host application root. `-T` disables Compose's pseudo-TTY while keeping stdin attached for MCP:
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "docker",
+      "args": [
+        "compose", "exec", "-T", "app",
+        "bundle", "exec", "woods-mcp", "/app/tmp/woods"
+      ],
+      "cwd": "/absolute/host/path/to/app"
+    }
+  }
+}
+```
+
+Replace `app` and `/app` with the Compose service and container path used by the project. The service must already be running. For plain Docker, use the exact container name and interactive stdin:
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "docker",
+      "args": ["exec", "-i", "my_app_web_1", "bundle", "exec", "woods-mcp", "/app/tmp/woods"]
+    }
+  }
+}
+```
+
+Two persistence gotchas:
+
+1. **tmpfs wipes the index on container restart.** If `/app/tmp` is tmpfs, re-run `rake woods:extract` after each restart, or persist the index by mounting a named volume at the index path (e.g. `- woods-data:/app/tmp/woods`) or writing the index outside the tmpfs mount.
+2. **Use the container path.** The Docker-launched server needs `/app/tmp/woods`, not the host's `./tmp/woods`.
+
+### Optional: run the Index Server on the host
+
+Use this only when the application bundle, a supported Ruby, and the Woods executable are installed on the host **and** the index directory is host-visible:
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "bundle",
+      "args": ["exec", "woods-mcp-start", "./tmp/woods"],
+      "cwd": "/absolute/host/path/to/app"
+    }
+  }
+}
+```
+
+`woods-mcp-start` validates the directory and active published generation, then replaces itself with `woods-mcp`. It does not install dependencies or restart a failed process.
+
+| | Container process (default) | Host process (optional) |
+|---|---|---|
+| **Index location** | Anywhere in the container | Host-visible bind mount |
+| **Path in `.mcp.json`** | Container path (`/app/tmp/woods`) | Host path (`./tmp/woods`) |
+| **Survives container replacement** | Only with a bind/named volume | Yes, on host disk |
+| **Needs Ruby/Woods bundle on host** | No | Yes |
+
+## Console Server Setup
+
+The Console Server queries live Rails state. There are two launch paths for the same embedded server.
+
+Before either path can start, deliberately enable live-data access in the Rails initializer:
+
+```ruby
+Woods.configure do |config|
+  config.console_mcp_enabled = true
+  config.console_mcp_token = ENV["WOODS_CONSOLE_MCP_TOKEN"]
+end
+```
+
+The process exits with status 1 while this master switch is false. Review [Console MCP setup and security](CONSOLE_MCP_SETUP.md) before enabling it.
+
+Stdio does not send the bearer token, but production Rails boot still requires `WOODS_CONSOLE_MCP_TOKEN` to contain at least 32 characters whenever Console is enabled. Provide it to the container through the application's normal secret mechanism. Outside production, omitting it warns and leaves the Console HTTP endpoint guarded with 401.
+
+### Comparison
+
+| | Direct Docker command | Configured launcher |
+|---|---|---|
+| **Where it runs** | Inside container via Docker stdio | `woods-console-mcp` execs `docker exec -i` |
+| **Config needed** | Rails master switch + `.mcp.json` | Rails master switch + `console.yml` + `.mcp.json` |
+| **Tools available** | 9 by default; 11 with read tools enabled | Same 9 or 11 |
+| **Setup complexity** | Minimal | Moderate |
+| **Best for** | Quick setup | Reusable launcher config |
+
+### Option 1: Embedded (9 Tier 1 tools)
+
+The MCP client spawns a Docker stdio process directly. The container boots Rails and runs the MCP server in-process. Only Tier 1 read-only tools are available (count, sample, find, pluck, aggregate, association_count, schema, recent, status).
+
+```json
+{
+  "mcpServers": {
+    "woods-console": {
+      "command": "docker",
+      "args": [
+        "compose", "exec", "-T", "app",
+        "bundle", "exec", "rake", "woods:console"
+      ],
+      "cwd": "/absolute/host/path/to/app"
+    }
+  }
+}
+```
+
+> **Compose uses `-T`; plain Docker uses `-i`.** Compose attaches stdin by default, and `-T` prevents a pseudo-TTY from corrupting MCP framing. Plain `docker exec` needs `-i` to keep stdin attached.
+
+If you use `docker exec` (not `docker compose exec`), provide the exact container name:
+
+```json
+{
+  "mcpServers": {
+    "woods-console": {
+      "command": "docker",
+      "args": [
+        "exec", "-i", "my_app_web_1",
+        "bundle", "exec", "rake", "woods:console"
+      ]
+    }
+  }
+}
+```
+
+### Option 2: Configured launcher
+
+The `woods-console-mcp` binary runs on the host and replaces itself with
+`docker exec -i ... bundle exec rake woods:console`. It exposes the same
+embedded tool surface as Option 1.
+
+This optional path requires Woods and the `woods-console-mcp` executable on
+the host. If Woods is installed only in the application container, use the
+direct Docker command in Option 1. If the host executable comes from the
+application bundle, configure the client with `bundle exec` and an absolute
+host `cwd`, as shown in the [launcher guide](CONSOLE_MCP_SETUP.md#option-d-launcher-wrapper).
+
+**Step 1: Create `console.yml`**
+
+```yaml
+# ~/.woods/console.yml
+mode: docker
+container: my_app_web_1
+```
+
+Find your container name with:
+
+```bash
+docker ps --format '{{.Names}}'
+```
+
+For Docker Compose, names follow the pattern `<project>-<service>-<number>` (e.g., `my_app-app-1`).
+
+**Step 2: Configure `.mcp.json`**
+
+```json
+{
+  "mcpServers": {
+    "woods-console": {
+      "command": "woods-console-mcp"
+    }
+  }
+}
+```
+
+The launcher reads `~/.woods/console.yml` by default. To use a different path:
+
+```json
+{
+  "mcpServers": {
+    "woods-console": {
+      "command": "woods-console-mcp",
+      "env": {
+        "WOODS_CONSOLE_CONFIG": "/path/to/console.yml"
+      }
+    }
+  }
+}
+```
+
+## Complete `.mcp.json` Example
+
+Both servers configured together for a Docker environment, after `config.console_mcp_enabled = true` has been deliberately set:
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "docker",
+      "args": ["compose", "exec", "-T", "app", "bundle", "exec", "woods-mcp", "/app/tmp/woods"],
+      "cwd": "/absolute/host/path/to/app"
+    },
+    "woods-console": {
+      "command": "docker",
+      "args": [
+        "compose", "exec", "-T", "app",
+        "bundle", "exec", "rake", "woods:console"
+      ],
+      "cwd": "/absolute/host/path/to/app"
+    }
+  }
+}
+```
+
+This uses the embedded console directly. To use the equivalent configured
+launcher, replace the `woods-console` entry:
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "docker",
+      "args": ["compose", "exec", "-T", "app", "bundle", "exec", "woods-mcp", "/app/tmp/woods"],
+      "cwd": "/absolute/host/path/to/app"
+    },
+    "woods-console": {
+      "command": "woods-console-mcp"
+    }
+  }
+}
+```
+
+## Task Reference
+
+Which tasks need Docker and which don't:
+
+| Task | Needs Rails? | Run via |
+|------|---|---|
+| `woods:extract` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:incremental` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:watch` | Yes | Dedicated container/process-manager entry |
+| `woods:extract_framework` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:embed` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:embed_incremental` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:console` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:flow[EntryPoint]` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:notion_sync` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:validate` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:stats` | Yes | `docker compose exec app bundle exec rake ...` |
+| `woods:clean` | Yes | `docker compose exec app bundle exec rake ...` |
+
+## Container Name Discovery
+
+Docker Compose generates container names using the pattern `<project>-<service>-<number>`:
+
+```bash
+# List all running containers
+docker ps --format '{{.Names}}'
+
+# Filter for your app service
+docker ps --format '{{.Names}}' | grep app
+```
+
+The project name defaults to the directory name of the `docker-compose.yml` file. Override it with `COMPOSE_PROJECT_NAME` or the `name:` key in `docker-compose.yml`.
+
+## Troubleshooting
+
+### Extraction output is not persisted
+
+**Symptom:** The index disappears when the application container is replaced.
+
+**Fix:** Volume-mount the app directory or the Woods output directory:
+
+```yaml
+volumes:
+  - .:/app
+```
+
+Then re-run extraction and verify it inside the container with `woods:validate` and `woods:stats`.
+
+### MCP client shows "connection refused" or no tools
+
+**Symptom:** The Index or Console server doesn't respond.
+
+**Check:**
+1. Container is running: `docker ps`
+2. For Compose stdio, `-T` is present; for plain `docker exec`, `-i` is present
+3. The server path is visible in its execution context: container path for Docker launch, host path for host launch
+
+### Incorrect stdio flags
+
+**Symptom:** Console server starts but immediately exits, or MCP client reports "broken pipe."
+
+**Fix:** Disable Compose's pseudo-TTY, or keep stdin open for plain Docker:
+
+```text
+Compose: "args": ["compose", "exec", "-T", "app", ...]
+Docker:  "args": ["exec", "-i", "container-name", ...]
+```
+
+### Wrong container name
+
+**Symptom:** `Error response from daemon: No such container: ...`
+
+**Fix:** Check the actual name with `docker ps --format '{{.Names}}'` and update your `.mcp.json` or `console.yml`.
+
+### Path confusion between host and container
+
+**Symptom:** Index Server reports "No manifest.json" even though extraction succeeded.
+
+**Fix:** Match the path to the server process:
+
+```
+Container launch: /app/tmp/woods
+Host launch:      /absolute/host/project/tmp/woods
+```
+
+### Rails boot noise breaks MCP protocol
+
+**Symptom:** MCP client shows JSON parse errors.
+
+**Fix:** The `woods:console` rake task redirects stdout to stderr before Rails boots. If you still see issues, check for `puts` or `print` calls in your initializers that run before the task captures stdout.
+
+### A tool from the 31-schema inventory is not listed
+
+**Expected behavior.** Supported servers register 9 Tier 1 tools by default.
+Tier 2, Tier 3, and `console_eval` are inventory only.
+
+To register `console_sql` and `console_query`, enable
+`console_embedded_read_tools` in Woods configuration or pass
+`embedded_read_tools: true` to the Rack middleware. See
+[CONSOLE_MCP_SETUP.md](CONSOLE_MCP_SETUP.md) for details.
+
+### Woods MCP tools not available in a git worktree
+
+When working in a git worktree, subagents may not find the woods MCP servers because `.mcp.json` discovery is path-based and the worktree has a different root directory. See [MCP_WORKTREE_SETUP.md](MCP_WORKTREE_SETUP.md) for the fix and verification steps.
+
+See [CONSOLE_MCP_SETUP.md](CONSOLE_MCP_SETUP.md) for detailed console server documentation.

data/docs/EMBEDDING_MODELS.md

@@ -0,0 +1,136 @@
+# Embedding Models
+
+Woods supports OpenAI's embedding API and any model served by a local Ollama
+instance. This doc covers the Ollama side, which model to pick, why the
+default is conservative, and how to add a new one.
+
+## TL;DR
+
+| Model | Native context | Dimensions | Size on disk | Recommended for |
+|---|---|---|---|---|
+| `nomic-embed-text` (default) | 2048 | 768 | 274 MB | General use, small footprint |
+| `bge-m3` | **8192** | 1024 | 1.2 GB | Large Rails units, fewer chunks |
+| `snowflake-arctic-embed2` | 8192 | 1024 | 1.2 GB | Multilingual projects |
+| `mxbai-embed-large` | 512 | 1024 | 670 MB | Short text (tweets, commit msgs) |
+| `all-minilm` | 512 | 384 | 46 MB | Tight-memory environments |
+
+The default is `nomic-embed-text` because it's small, fast, and widely available
+through Ollama. Pull it before the first embed with `ollama pull nomic-embed-text`.
+If you're indexing a large Rails codebase and don't mind
+pulling a bigger model, switching to `bge-m3` usually gives you:
+
+- **Fewer chunks per unit**: 4× the context means most concern-inlined models
+  and service objects fit in a single embedding, which keeps retrieval scores
+  cleaner and the vector index smaller.
+- **Stronger retrieval**: `bge-m3` benchmarks meaningfully better than
+  `nomic-embed-text` on code-search MTEB tasks.
+
+The tradeoff is download size (1.2 GB vs 274 MB), RAM usage during inference,
+and a dimension change (768 → 1024) that requires re-indexing when you switch.
+
+## Switching models
+
+```ruby
+Woods.configure do |config|
+  config.embedding_provider = :ollama
+  config.embedding_options = {
+    model: 'bge-m3',
+    host: 'http://localhost:11434'
+  }
+  # Everything else, chunker sizing, token counting, num_ctx, is picked
+  # up automatically from the model name.
+end
+```
+
+Pull the model first:
+
+```bash
+ollama pull bge-m3
+```
+
+If you switch models on an existing install, drop your vector index before
+re-indexing, the embedding dimension change is incompatible with existing
+vectors. Woods raises `Woods::MCP::DimensionMismatch` if you miss it:
+`rake woods:embed` refuses up front on pgvector/Qdrant, and the MCP server
+refuses at boot when a dump's recorded dimension disagrees with the provider.
+
+## Why `num_ctx` isn't enough
+
+Ollama has a long-running regression (upstream issue
+[ollama/ollama#14186](https://github.com/ollama/ollama/issues/14186)) where
+`options.num_ctx` does **not** lift the effective context ceiling on
+`/api/embed` for models whose native context is smaller than the requested
+override. For `nomic-embed-text` (native 2048) the server rejects inputs above
+that with a `400 "the input length exceeds the context length"` regardless of
+`num_ctx`.
+
+Woods works around this two ways:
+
+1. **Advertise the native ceiling, not the override.** `Provider::Ollama` keeps
+   a model → native-context registry (`MODEL_CONTEXT_LENGTHS`) so the chunker
+   sizes inputs to what Ollama will actually accept. `num_ctx` is still passed
+   through the request body in case the regression is fixed upstream, but
+   nothing relies on it.
+2. **Verify client-side with the real tokenizer.** When the
+   [`tokenizers`](https://github.com/ankane/tokenizers-ruby) gem is installed,
+   `Embedding::TokenCounter` loads the `bert-base-uncased` WordPiece tokenizer
+   (the one every BERT-family embedding model is built on) and the chunker
+   re-verifies every slice. WordPiece fragments CamelCase and `::` separators
+   differently than character-based estimation suggests, this verification is
+   what catches the 10–20 % gap between our estimate and Ollama's internal
+   count.
+
+## Adding a new model to the registry
+
+If you want Woods to auto-pick `num_ctx` for a model we don't ship support for:
+
+1. Pull the model: `ollama pull your-model`.
+2. Read its native context:
+   ```bash
+   curl -s http://localhost:11434/api/show -d '{"model":"your-model"}' \
+     | jq '.model_info | to_entries[] | select(.key | contains("context_length"))'
+   ```
+3. Verify the server actually honours it (some models report a larger context
+   than the server enforces, see the nomic-embed-text case above). Send a
+   request with a known-large input and check for 400s:
+   ```bash
+   curl -s http://localhost:11434/api/embed \
+     -d "{\"model\":\"your-model\",\"input\":\"$(ruby -e 'print "word " * 8500')\"}" \
+     | jq '.error // "no error, context held"'
+   ```
+4. Add it to `MODEL_CONTEXT_LENGTHS` in `lib/woods/embedding/provider.rb`
+   (keep the value at the **enforced** ceiling, not the advertised one).
+
+## When to stay on `nomic-embed-text`
+
+- Fresh installs, CI, evaluation environments, no extra pull step.
+- Small/toy codebases where the 2048-token ceiling isn't a real constraint.
+- Memory-constrained Ollama hosts (the 1.2 GB `bge-m3` weights vs 274 MB for
+  `nomic-embed-text` can matter on a shared laptop).
+
+## When to switch to `bge-m3` or `snowflake-arctic-embed2`
+
+- Indexing a real-world Rails app where concern-inlined models or long service
+  objects routinely exceed 2048 tokens.
+- You already pay the `tokenizers` gem install cost and want the tighter
+  coupling between client-side verification and server-side enforcement.
+- Multilingual content (Arctic Embed 2 is BGE M3 with a stronger non-English
+  story).
+
+## Deterministic fake provider (no model at all)
+
+For CI, sandboxes, and offline hosts where neither OpenAI nor Ollama is
+reachable, `config.embedding_provider = :fake` wires
+`Woods::Embedding::Provider::Fake`, deterministic bag-of-words hashing with
+L2 normalization, no network endpoint, configurable dimension
+(`embedding_options = { dims: 128 }`). Cosine similarity stays mechanically
+meaningful (shared vocabulary ranks closer), but the vectors are **not
+semantically meaningful embeddings**: use it to smoke-test the
+embed → store → retrieve pipeline, never for production retrieval quality.
+See [CONFIGURATION_REFERENCE.md](./CONFIGURATION_REFERENCE.md#fake-embeddings-ci--sandboxes--offline-hosts).
+
+## Related
+
+- [CONFIGURATION_REFERENCE.md](./CONFIGURATION_REFERENCE.md): full config surface
+- [TOKEN_BENCHMARK.md](./TOKEN_BENCHMARK.md): where our chars/token numbers come from
+- [BACKEND_MATRIX.md](./BACKEND_MATRIX.md): picking a vector store that matches

data/docs/EVALUATION.md

@@ -0,0 +1,91 @@
+# Evaluating Woods
+
+Three harnesses, three questions.
+
+| Task | Question it answers | Command |
+|---|---|---|
+| Retrieval evaluation | Does `codebase_retrieve` return the right units for a query? | `bin/rails woods:evaluate` |
+| Baseline comparison | How does a naive strategy score on the same queries? | `bin/rails "woods:evaluate:baseline[grep]"` |
+| Agent-level ablation | Does an agent resolve real tasks better, or cheaper, with the index on? | `bin/rails "woods:evaluate:ablation[config/eval_ablation.json]"` |
+
+## Retrieval evaluation
+
+`woods:evaluate` runs a ground-truth query set (`config/eval_queries.json`, `EVAL_QUERY_SET`) through the retrieval pipeline and reports precision at 5 and 10, recall, MRR, context completeness, and token efficiency. `EVAL_BASELINE_FILE` (a versioned thresholds file, see `Woods::Evaluation::Baseline`) turns it into a gate; `EVAL_MIN_PRECISION`, `EVAL_MIN_RECALL`, and `EVAL_MIN_MRR` override single thresholds. It needs an embedded index; set `embedding_provider = :fake` for an offline run. Output: `tmp/eval_report.json` (`EVAL_OUTPUT`).
+
+`woods:evaluate:baseline[grep|random|file_level]` scores a naive strategy on the same query set for comparison.
+
+## Agent-level ablation
+
+Retrieval scores say whether the right units come back. They do not say whether an agent finishes the job. The ablation runs the same task set twice per task, once with the Woods MCP server available and once without, and compares resolution rate, tokens, cost, and turns.
+
+**This is a harness for collecting paired on/off runs, not a source of causal evidence.** Sample size, task selection, and agent nondeterminism all bear on what a result means. Treat it as one input, alongside the retrieval evaluation and manual review, not as a controlled experiment on its own. It is the on/off comparison "Code Isn't Memory" (arXiv 2606.22417, June 2026) reported, where gains concentrated in multi-file changes.
+
+### Disposable checkouts
+
+Every trial (one task, one condition) runs in its own disposable `git worktree` checkout, created from a single baseline SHA resolved once at the start of the run (the current `HEAD` of the application repository), never in the caller's own working directory:
+
+1. `git worktree add --detach <tmp-checkout> <baseline-sha>`
+2. the task set's optional `reset` command, inside the checkout
+3. the agent command, inside the checkout
+4. the task's `check` command, inside the checkout
+5. `git worktree remove --force <tmp-checkout>`
+
+Both conditions for a task start from the identical commit, so a difference between them is not confounded by one trial running against a dirtier tree than the other. A `reset` failure aborts the trial and is counted as an error; it is never silently ignored. The checkout is removed whether the trial succeeded, failed, or timed out.
+
+### Provenance
+
+Every result carries the agent command that ran, the model the agent's JSON payload reported (when present), the MCP wiring in effect (`--mcp-config <path>` or `--strict-mcp-config`), the Woods generation number found in the checkout's `tmp/woods/generation.json` (when present), and the baseline SHA the checkout came from. Compare results only across runs sharing the same baseline SHA and Woods generation.
+
+### Woods availability preflight
+
+Before a trial runs, the harness checks that Woods is actually available or actually absent, distinguishing "MCP enabled" (the agent command is wired to reach the Woods MCP server) from "index present" (an index exists on disk in the checkout):
+
+- `on` requires both: an `--mcp-config` reference in the agent command, and a materialized index in the checkout.
+- `off` requires confirmation that MCP is truly unreachable: `--strict-mcp-config` present, and no `--mcp-config` reference at all. Both flags together still fail, since `--mcp-config` wires the agent to Woods regardless of strict mode.
+
+A failed check aborts the trial and is counted as an error, the same as a reset failure. The default probe is injectable (`woods_probe:` on `AblationRunner.new`) for a host app with a different wiring convention.
+
+### Task set
+
+```json
+{
+  "schema_version": 1,
+  "agent_on":  "claude -p {prompt} --output-format json --mcp-config .mcp.json",
+  "agent_off": "claude -p {prompt} --output-format json --strict-mcp-config",
+  "reset": "git checkout -- . && git clean -fdq",
+  "tasks": [
+    { "id": "comment-count", "prompt": "Add a comments_count counter cache to Post.", "check": "bin/rspec spec/models/post_spec.rb" }
+  ]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `agent_on`, `agent_off` | Shell command templates; `{prompt}` is replaced with the shell-escaped prompt. The agent must print one JSON object on stdout in the `claude -p --output-format json` shape (`usage`, `total_cost_usd`, `num_turns`, `duration_ms`, optionally `model`). `EVAL_AGENT_ON` and `EVAL_AGENT_OFF` override them. |
+| `reset` | Optional command run inside each trial's disposable checkout before the agent runs, so both conditions start from the same tree |
+| `tasks[].check` | Command whose exit status decides resolution (a spec file, a script, `bin/rails test`) |
+| `tasks[].workdir` | Relative to the checkout root; default `.` |
+
+`spec/fixtures/evaluation_ablation_tasks.example.json` is a format fixture only.
+
+### Running it
+
+```bash
+bin/rails woods:extract                                     # index the tree under test
+bin/rails "woods:evaluate:ablation[config/eval_ablation.json]"
+# EVAL_ABLATION_OUTPUT=tmp/eval_ablation.json
+```
+
+`woods:evaluate:ablation` never boots Rails: it shells out to the agent command and the task set's own `check` commands, and never touches `Rails.application`.
+
+The report holds every attempt (`task_id`, `condition`, `resolved`, `total_tokens`, `cost_usd`, `turns`, `duration_ms`, `error`, `provenance`) and a summary per condition (`resolution_rate`, `mean_tokens`, `mean_cost_usd`, `mean_turns`, `tasks`, `errors`) plus a `delta` (on minus off, present only when both conditions ran). Tokens are the sum of input, output, cache-creation, and cache-read tokens. `mean_tokens` is `null` when every trial in that condition errored before the agent produced JSON.
+
+A timeout (`AblationRunner.new(..., timeout: seconds)`, default 600) bounds every command a trial runs, applied independently to each one rather than as a single budget shared across the trial: worktree add/remove, the optional `reset`, the agent invocation, and the check. A timed-out `reset` or `check` counts as an error the same way a timed-out agent invocation does. When the default subprocess executor is in use, a timed-out command is terminated (`TERM`, then `KILL` if still alive after a short grace period) so it never outlives the trial that started it.
+
+### Where to get tasks
+
+The Rails Foundation's "Agents on Rails" benchmark (announced 2026-08-13, built by Evil Martians) is a ready-made task set once its tasks and checks are checked into a host app. Any set of tasks with a deterministic `check` works.
+
+### Reading the numbers
+
+A negative `delta.mean_tokens` with an equal or higher `delta.resolution_rate` is the result that sells the index, on the tasks actually run. It does not, by itself, establish that the index causes the difference: run at least ten tasks, two conditions on five tasks is noise, and remember that agent runs are not perfectly reproducible even at a fixed baseline SHA. Keep the baseline SHA and Woods generation fixed across a comparison (both are recorded per result), and do not let the `on` agent run `woods:extract` mid-task.